Oct 30 08:51:25.895337 ip-10-0-143-244 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 30 08:51:25.895349 ip-10-0-143-244 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 30 08:51:25.895358 ip-10-0-143-244 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 30 08:51:25.895664 ip-10-0-143-244 systemd[1]: Failed to start Kubernetes Kubelet. Oct 30 08:51:36.006134 ip-10-0-143-244 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 30 08:51:36.006160 ip-10-0-143-244 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 8f978fbde73a4901aa4f630411ad5fa4 -- Oct 30 08:53:18.774405 ip-10-0-143-244 systemd[1]: Starting Kubernetes Kubelet... Oct 30 08:53:19.235203 ip-10-0-143-244 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 08:53:19.235203 ip-10-0-143-244 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 08:53:19.235203 ip-10-0-143-244 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 08:53:19.235203 ip-10-0-143-244 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 08:53:19.242276 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242184 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 08:53:19.242276 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242273 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 08:53:19.242276 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242277 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 08:53:19.242276 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242281 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 08:53:19.242395 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242284 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 08:53:19.242395 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242288 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 08:53:19.242395 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242290 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 08:53:19.242395 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242301 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 08:53:19.242395 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242304 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 08:53:19.242395 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242307 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 08:53:19.242395 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242310 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 08:53:19.242395 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242312 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 08:53:19.242395 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242315 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 08:53:19.242395 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242318 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 08:53:19.242395 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242320 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 08:53:19.242395 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242323 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 08:53:19.242395 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242326 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 08:53:19.242395 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242329 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 30 08:53:19.242395 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242331 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 08:53:19.242395 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242334 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 08:53:19.242395 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242337 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 08:53:19.242395 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242339 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 08:53:19.242395 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242342 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 08:53:19.242395 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242345 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 08:53:19.242871 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242347 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 08:53:19.242871 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242350 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 08:53:19.242871 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242353 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 08:53:19.242871 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242355 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 08:53:19.242871 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242358 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 08:53:19.242871 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242360 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 08:53:19.242871 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242363 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 08:53:19.242871 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242365 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 08:53:19.242871 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242368 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 08:53:19.242871 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242371 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 08:53:19.242871 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242373 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 08:53:19.242871 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242375 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 08:53:19.242871 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242379 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 08:53:19.242871 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242382 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 08:53:19.242871 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242385 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 08:53:19.242871 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242388 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 08:53:19.242871 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242391 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 08:53:19.242871 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242393 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 08:53:19.242871 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242396 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 08:53:19.242871 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242399 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 08:53:19.243352 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242402 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 08:53:19.243352 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242404 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 08:53:19.243352 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242408 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 08:53:19.243352 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242410 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 08:53:19.243352 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242413 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 08:53:19.243352 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242415 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 08:53:19.243352 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242418 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 08:53:19.243352 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242421 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 08:53:19.243352 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242424 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 08:53:19.243352 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242426 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 08:53:19.243352 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242429 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 08:53:19.243352 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242431 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 08:53:19.243352 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242434 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 08:53:19.243352 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242437 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 08:53:19.243352 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242440 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 08:53:19.243352 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242443 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 08:53:19.243352 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242445 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 08:53:19.243352 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242448 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 08:53:19.243352 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242450 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 08:53:19.243801 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242453 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 08:53:19.243801 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242456 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 08:53:19.243801 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242459 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 08:53:19.243801 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242461 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 08:53:19.243801 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242464 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 08:53:19.243801 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242466 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 08:53:19.243801 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242470 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 08:53:19.243801 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242480 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 08:53:19.243801 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242484 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 08:53:19.243801 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242487 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 08:53:19.243801 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242490 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 08:53:19.243801 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242493 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 08:53:19.243801 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242495 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 08:53:19.243801 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242498 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 08:53:19.243801 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242501 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 08:53:19.243801 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242503 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 08:53:19.243801 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242507 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 08:53:19.243801 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242509 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 08:53:19.243801 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242513 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 08:53:19.244301 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242518 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 08:53:19.244301 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242522 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 08:53:19.244301 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242525 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 08:53:19.244301 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242528 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 08:53:19.244301 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242533 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 08:53:19.244301 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242537 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 08:53:19.244301 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242540 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 08:53:19.244301 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242543 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 08:53:19.244301 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242545 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 08:53:19.244301 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242549 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 08:53:19.244301 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242552 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 08:53:19.244301 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242554 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 08:53:19.244301 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242557 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 08:53:19.244301 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242560 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 08:53:19.244301 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242562 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 08:53:19.244301 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242565 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 08:53:19.244301 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242568 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 08:53:19.244301 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242570 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 08:53:19.244301 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242573 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 08:53:19.244741 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242576 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 08:53:19.244741 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242579 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 08:53:19.244741 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242582 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 08:53:19.244741 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242585 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 08:53:19.244741 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.242588 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 08:53:19.244741 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244306 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 08:53:19.244741 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244314 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 08:53:19.244741 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244319 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 08:53:19.244741 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244322 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 08:53:19.244741 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244324 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 08:53:19.244741 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244327 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 08:53:19.244741 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244330 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 08:53:19.244741 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244332 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 08:53:19.244741 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244335 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 08:53:19.244741 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244338 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 08:53:19.244741 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244341 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 08:53:19.244741 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244343 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 08:53:19.244741 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244346 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 08:53:19.244741 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244349 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 08:53:19.244741 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244351 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 08:53:19.245491 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244354 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 08:53:19.245491 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244356 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 08:53:19.245491 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244359 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 08:53:19.245491 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244363 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 08:53:19.245491 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244366 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 08:53:19.245491 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244368 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 08:53:19.245491 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244371 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 08:53:19.245491 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244374 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 08:53:19.245491 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244376 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 08:53:19.245491 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244379 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 08:53:19.245491 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244382 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 08:53:19.245491 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244384 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 08:53:19.245491 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244388 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 08:53:19.245491 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244391 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 08:53:19.245491 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244394 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 08:53:19.245491 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244396 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 08:53:19.245491 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244399 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 08:53:19.245491 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244402 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 08:53:19.245491 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244405 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 08:53:19.246034 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244408 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 08:53:19.246034 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244410 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 08:53:19.246034 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244413 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 08:53:19.246034 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244415 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 08:53:19.246034 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244418 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 08:53:19.246034 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244420 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 08:53:19.246034 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244423 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 08:53:19.246034 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244425 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 08:53:19.246034 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244428 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 08:53:19.246034 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244430 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 08:53:19.246034 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244433 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 08:53:19.246034 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244436 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 08:53:19.246034 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244438 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 08:53:19.246034 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244441 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 08:53:19.246034 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244444 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 08:53:19.246034 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244446 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 08:53:19.246034 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244449 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 08:53:19.246034 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244452 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 08:53:19.246034 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244455 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 08:53:19.246034 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244457 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 08:53:19.246542 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244460 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 08:53:19.246542 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244463 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 08:53:19.246542 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244466 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 08:53:19.246542 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244469 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 30 08:53:19.246542 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244472 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 08:53:19.246542 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244474 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 08:53:19.246542 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244477 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 08:53:19.246542 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244480 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 08:53:19.246542 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244482 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 08:53:19.246542 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244485 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 08:53:19.246542 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244488 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 08:53:19.246542 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244491 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 08:53:19.246542 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244493 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 08:53:19.246542 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244496 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 08:53:19.246542 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244498 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 08:53:19.246542 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244502 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 08:53:19.246542 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244504 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 08:53:19.246542 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244507 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 08:53:19.246542 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244509 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 08:53:19.246542 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244512 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 08:53:19.247020 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244514 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 08:53:19.247020 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244517 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 08:53:19.247020 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244520 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 08:53:19.247020 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244522 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 08:53:19.247020 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244525 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 08:53:19.247020 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244527 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 08:53:19.247020 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244530 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 08:53:19.247020 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244532 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 08:53:19.247020 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244536 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 08:53:19.247020 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244538 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 08:53:19.247020 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244541 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 08:53:19.247020 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244544 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 08:53:19.247020 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244547 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 08:53:19.247020 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244549 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 08:53:19.247020 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244552 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 08:53:19.247020 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244554 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 08:53:19.247020 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244557 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 08:53:19.247020 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244559 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 08:53:19.247020 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244562 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 08:53:19.247516 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244564 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 08:53:19.247516 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244567 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 08:53:19.247516 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244570 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 08:53:19.247516 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244573 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 08:53:19.247516 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244576 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 08:53:19.247516 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244580 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 08:53:19.247516 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244583 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 08:53:19.247516 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244586 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 08:53:19.247516 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244589 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 08:53:19.247516 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244591 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 08:53:19.247516 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244594 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 08:53:19.247516 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244596 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 08:53:19.247516 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.244601 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 08:53:19.247516 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246311 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 08:53:19.247516 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246321 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 08:53:19.247516 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246325 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 08:53:19.247516 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246328 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 08:53:19.247516 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246330 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 08:53:19.247516 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246333 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 08:53:19.248057 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246336 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 08:53:19.248057 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246340 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 08:53:19.248057 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246343 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 08:53:19.248057 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246345 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 08:53:19.248057 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246348 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 08:53:19.248057 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246351 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 08:53:19.248057 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246354 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 08:53:19.248057 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246357 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 08:53:19.248057 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246359 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 08:53:19.248057 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246362 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 08:53:19.248057 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246365 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 08:53:19.248057 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246368 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 08:53:19.248057 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246371 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 08:53:19.248057 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246373 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 08:53:19.248057 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246376 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 08:53:19.248057 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246378 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 08:53:19.248057 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246381 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 08:53:19.248057 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246383 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 08:53:19.248057 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246386 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 08:53:19.248057 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246388 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 08:53:19.248545 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246391 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 08:53:19.248545 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246393 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 08:53:19.248545 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246396 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 08:53:19.248545 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246398 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 08:53:19.248545 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246401 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 08:53:19.248545 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246407 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 08:53:19.248545 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246410 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 08:53:19.248545 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246412 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 08:53:19.248545 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246415 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 08:53:19.248545 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246418 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 08:53:19.248545 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246421 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 08:53:19.248545 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246423 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 08:53:19.248545 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246426 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 08:53:19.248545 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246428 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 08:53:19.248545 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246432 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 08:53:19.248545 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246434 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 08:53:19.248545 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246437 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 08:53:19.248545 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246439 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 08:53:19.248545 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246442 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 08:53:19.248991 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246444 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 08:53:19.248991 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246447 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 08:53:19.248991 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246449 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 08:53:19.248991 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246452 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 08:53:19.248991 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246454 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 08:53:19.248991 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246457 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 08:53:19.248991 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246459 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 08:53:19.248991 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246462 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 08:53:19.248991 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246465 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 08:53:19.248991 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246469 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 08:53:19.248991 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246479 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 08:53:19.248991 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246483 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 08:53:19.248991 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246486 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 08:53:19.248991 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246489 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 08:53:19.248991 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246492 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 08:53:19.248991 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246494 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 08:53:19.248991 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246497 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 08:53:19.248991 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246500 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 08:53:19.248991 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246503 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 08:53:19.249447 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246505 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 08:53:19.249447 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246508 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 08:53:19.249447 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246511 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 08:53:19.249447 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246513 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 08:53:19.249447 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246516 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 08:53:19.249447 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246519 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 08:53:19.249447 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246522 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 08:53:19.249447 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246524 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 08:53:19.249447 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246527 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 08:53:19.249447 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246529 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 08:53:19.249447 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246532 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 08:53:19.249447 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246535 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 08:53:19.249447 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246537 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 08:53:19.249447 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246540 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 08:53:19.249447 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246542 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 08:53:19.249447 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246545 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 08:53:19.249447 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246548 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 08:53:19.249447 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246550 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 30 08:53:19.249447 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246554 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 08:53:19.249447 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246557 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 08:53:19.249917 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246560 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 08:53:19.249917 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246563 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 08:53:19.249917 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246565 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 08:53:19.249917 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246568 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 08:53:19.249917 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246570 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 08:53:19.249917 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246573 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 08:53:19.249917 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246575 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 08:53:19.249917 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246578 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 08:53:19.249917 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246582 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 08:53:19.249917 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246586 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 08:53:19.249917 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246589 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 08:53:19.249917 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246592 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 08:53:19.249917 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246595 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 08:53:19.249917 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246598 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 08:53:19.249917 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246601 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 08:53:19.249917 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246604 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 08:53:19.249917 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246610 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 08:53:19.249917 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246613 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 08:53:19.249917 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246615 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 08:53:19.250371 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246618 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 08:53:19.250371 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246621 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 08:53:19.250371 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.246623 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 08:53:19.250371 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247318 2575 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 08:53:19.250371 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247328 2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 08:53:19.250371 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247334 2575 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 08:53:19.250371 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247339 2575 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 08:53:19.250371 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247344 2575 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 08:53:19.250371 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247347 2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 08:53:19.250371 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247355 2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 08:53:19.250371 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247359 2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 08:53:19.250371 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247363 2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 08:53:19.250371 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247367 2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 08:53:19.250371 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247371 2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 08:53:19.250371 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247374 2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 08:53:19.250371 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247377 2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 08:53:19.250371 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247381 2575 flags.go:64] FLAG: --cgroup-root="" Oct 30 08:53:19.250371 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247384 2575 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 08:53:19.250371 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247387 2575 flags.go:64] FLAG: --client-ca-file="" Oct 30 08:53:19.250371 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247390 2575 flags.go:64] FLAG: --cloud-provider="external" Oct 30 08:53:19.250371 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247393 2575 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 08:53:19.250371 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247398 2575 flags.go:64] FLAG: --cluster-domain="" Oct 30 08:53:19.250371 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247401 2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 08:53:19.250371 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247404 2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 08:53:19.250937 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247407 2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 08:53:19.250937 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247411 2575 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 08:53:19.250937 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247415 2575 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 08:53:19.250937 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247420 2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 08:53:19.250937 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247423 2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 08:53:19.250937 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247427 2575 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 08:53:19.250937 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247430 2575 flags.go:64] FLAG: --contention-profiling="false" Oct 30 08:53:19.250937 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247433 2575 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 08:53:19.250937 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247436 2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 08:53:19.250937 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247439 2575 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 08:53:19.250937 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247442 2575 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 08:53:19.250937 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247446 2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 08:53:19.250937 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247449 2575 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 08:53:19.250937 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247452 2575 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 08:53:19.250937 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247455 2575 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 08:53:19.250937 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247459 2575 flags.go:64] FLAG: --enable-server="true" Oct 30 08:53:19.250937 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247462 2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 08:53:19.250937 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247467 2575 flags.go:64] FLAG: --event-burst="100" Oct 30 08:53:19.250937 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247470 2575 flags.go:64] FLAG: --event-qps="50" Oct 30 08:53:19.250937 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247473 2575 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 08:53:19.250937 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247476 2575 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 08:53:19.250937 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247480 2575 flags.go:64] FLAG: --eviction-hard="" Oct 30 08:53:19.250937 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247484 2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 08:53:19.250937 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247487 2575 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 08:53:19.251595 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247490 2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 08:53:19.251595 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247494 2575 flags.go:64] FLAG: --eviction-soft="" Oct 30 08:53:19.251595 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247497 2575 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 08:53:19.251595 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247500 2575 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 08:53:19.251595 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247503 2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 08:53:19.251595 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247506 2575 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 08:53:19.251595 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247509 2575 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 08:53:19.251595 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247512 2575 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 08:53:19.251595 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247515 2575 flags.go:64] FLAG: --feature-gates="" Oct 30 08:53:19.251595 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247519 2575 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 08:53:19.251595 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247523 2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 08:53:19.251595 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247528 2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 08:53:19.251595 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247532 2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 08:53:19.251595 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247535 2575 flags.go:64] FLAG: --healthz-port="10248" Oct 30 08:53:19.251595 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247538 2575 flags.go:64] FLAG: --help="false" Oct 30 08:53:19.251595 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247542 2575 flags.go:64] FLAG: --hostname-override="ip-10-0-143-244.ec2.internal" Oct 30 08:53:19.251595 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247545 2575 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 08:53:19.251595 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247548 2575 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 08:53:19.251595 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247551 2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 08:53:19.251595 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247554 2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 08:53:19.251595 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247558 2575 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 08:53:19.251595 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247560 2575 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 08:53:19.251595 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247564 2575 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 08:53:19.252172 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247567 2575 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 08:53:19.252172 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247570 2575 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 08:53:19.252172 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247573 2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 08:53:19.252172 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247576 2575 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 08:53:19.252172 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247579 2575 flags.go:64] FLAG: --kube-reserved="" Oct 30 08:53:19.252172 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247583 2575 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 08:53:19.252172 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247585 2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 08:53:19.252172 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247589 2575 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 08:53:19.252172 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247592 2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 08:53:19.252172 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247595 2575 flags.go:64] FLAG: --lock-file="" Oct 30 08:53:19.252172 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247598 2575 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 08:53:19.252172 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247601 2575 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 08:53:19.252172 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247604 2575 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 08:53:19.252172 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247614 2575 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 08:53:19.252172 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247617 2575 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 08:53:19.252172 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247621 2575 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 08:53:19.252172 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247623 2575 flags.go:64] FLAG: --logging-format="text" Oct 30 08:53:19.252172 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247626 2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 08:53:19.252172 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247630 2575 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 08:53:19.252172 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247633 2575 flags.go:64] FLAG: --manifest-url="" Oct 30 08:53:19.252172 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247638 2575 flags.go:64] FLAG: --manifest-url-header="" Oct 30 08:53:19.252172 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247642 2575 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 08:53:19.252172 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247645 2575 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 08:53:19.252172 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247650 2575 flags.go:64] FLAG: --max-pods="110" Oct 30 08:53:19.252172 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247653 2575 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 08:53:19.252762 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247656 2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 08:53:19.252762 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247659 2575 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 08:53:19.252762 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247662 2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 08:53:19.252762 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247665 2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 08:53:19.252762 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247668 2575 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 08:53:19.252762 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247671 2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 08:53:19.252762 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247679 2575 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 08:53:19.252762 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247682 2575 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 08:53:19.252762 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247685 2575 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 08:53:19.252762 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247688 2575 flags.go:64] FLAG: --pod-cidr="" Oct 30 08:53:19.252762 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247691 2575 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 08:53:19.252762 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247695 2575 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 08:53:19.252762 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247698 2575 flags.go:64] FLAG: --pods-per-core="0" Oct 30 08:53:19.252762 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247701 2575 flags.go:64] FLAG: --port="10250" Oct 30 08:53:19.252762 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247704 2575 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 08:53:19.252762 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247707 2575 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f7ef334f56ea5bee" Oct 30 08:53:19.252762 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247710 2575 flags.go:64] FLAG: --qos-reserved="" Oct 30 08:53:19.252762 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247713 2575 flags.go:64] FLAG: --read-only-port="10255" Oct 30 08:53:19.252762 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247716 2575 flags.go:64] FLAG: --register-node="true" Oct 30 08:53:19.252762 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247719 2575 flags.go:64] FLAG: --register-with-taints="" Oct 30 08:53:19.252762 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247723 2575 flags.go:64] FLAG: --registry-burst="10" Oct 30 08:53:19.252762 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247726 2575 flags.go:64] FLAG: --registry-qps="5" Oct 30 08:53:19.252762 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247729 2575 flags.go:64] FLAG: --reserved-cpus="" Oct 30 08:53:19.252762 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247732 2575 flags.go:64] FLAG: --reserved-memory="" Oct 30 08:53:19.252762 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247735 2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 08:53:19.252762 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247739 2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 08:53:19.253405 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247742 2575 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 08:53:19.253405 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247746 2575 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 08:53:19.253405 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247749 2575 flags.go:64] FLAG: --runonce="false" Oct 30 08:53:19.253405 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247752 2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 08:53:19.253405 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247755 2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 08:53:19.253405 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247758 2575 flags.go:64] FLAG: --seccomp-default="false" Oct 30 08:53:19.253405 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247761 2575 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 08:53:19.253405 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247764 2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 08:53:19.253405 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247767 2575 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 08:53:19.253405 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247770 2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 08:53:19.253405 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247773 2575 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 08:53:19.253405 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247776 2575 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 08:53:19.253405 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247779 2575 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 08:53:19.253405 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247782 2575 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 08:53:19.253405 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247784 2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 08:53:19.253405 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247787 2575 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 08:53:19.253405 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247790 2575 flags.go:64] FLAG: --system-cgroups="" Oct 30 08:53:19.253405 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247793 2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 08:53:19.253405 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247799 2575 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 08:53:19.253405 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247802 2575 flags.go:64] FLAG: --tls-cert-file="" Oct 30 08:53:19.253405 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247805 2575 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 08:53:19.253405 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247830 2575 flags.go:64] FLAG: --tls-min-version="" Oct 30 08:53:19.253405 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247834 2575 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 08:53:19.253405 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247839 2575 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 08:53:19.253405 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247842 2575 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 08:53:19.254015 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247845 2575 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 08:53:19.254015 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247848 2575 flags.go:64] FLAG: --v="2" Oct 30 08:53:19.254015 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247853 2575 flags.go:64] FLAG: --version="false" Oct 30 08:53:19.254015 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247857 2575 flags.go:64] FLAG: --vmodule="" Oct 30 08:53:19.254015 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247861 2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 08:53:19.254015 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.247864 2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 08:53:19.254015 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248004 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 08:53:19.254015 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248008 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 08:53:19.254015 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248012 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 08:53:19.254015 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248015 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 08:53:19.254015 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248018 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 08:53:19.254015 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248021 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 08:53:19.254015 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248024 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 08:53:19.254015 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248026 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 08:53:19.254015 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248029 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 08:53:19.254015 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248031 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 08:53:19.254015 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248034 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 08:53:19.254015 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248037 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 08:53:19.254015 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248040 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 08:53:19.254015 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248042 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 08:53:19.254015 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248045 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 08:53:19.254548 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248047 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 08:53:19.254548 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248050 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 08:53:19.254548 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248052 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 08:53:19.254548 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248055 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 08:53:19.254548 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248057 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 08:53:19.254548 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248060 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 08:53:19.254548 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248063 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 08:53:19.254548 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248066 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 08:53:19.254548 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248069 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 08:53:19.254548 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248073 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 08:53:19.254548 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248076 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 30 08:53:19.254548 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248079 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 08:53:19.254548 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248095 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 08:53:19.254548 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248098 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 08:53:19.254548 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248111 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 08:53:19.254548 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248114 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 08:53:19.254548 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248117 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 08:53:19.254548 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248120 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 08:53:19.254548 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248122 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 08:53:19.254548 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248125 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 08:53:19.255033 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248129 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 08:53:19.255033 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248132 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 08:53:19.255033 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248134 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 08:53:19.255033 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248137 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 08:53:19.255033 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248140 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 08:53:19.255033 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248144 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 08:53:19.255033 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248148 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 08:53:19.255033 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248152 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 08:53:19.255033 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248155 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 08:53:19.255033 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248157 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 08:53:19.255033 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248160 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 08:53:19.255033 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248163 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 08:53:19.255033 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248165 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 08:53:19.255033 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248168 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 08:53:19.255033 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248170 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 08:53:19.255033 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248173 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 08:53:19.255033 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248176 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 08:53:19.255033 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248178 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 08:53:19.255033 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248181 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 08:53:19.255507 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248184 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 08:53:19.255507 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248186 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 08:53:19.255507 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248190 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 08:53:19.255507 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248194 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 08:53:19.255507 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248197 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 08:53:19.255507 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248199 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 08:53:19.255507 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248202 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 08:53:19.255507 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248204 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 08:53:19.255507 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248207 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 08:53:19.255507 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248210 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 08:53:19.255507 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248212 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 08:53:19.255507 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248215 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 08:53:19.255507 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248219 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 08:53:19.255507 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248224 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 08:53:19.255507 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248227 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 08:53:19.255507 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248230 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 08:53:19.255507 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248233 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 08:53:19.255507 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248236 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 08:53:19.255507 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248238 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 08:53:19.255953 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248241 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 08:53:19.255953 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248244 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 08:53:19.255953 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248246 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 08:53:19.255953 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248249 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 08:53:19.255953 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248251 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 08:53:19.255953 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248254 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 08:53:19.255953 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248256 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 08:53:19.255953 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248259 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 08:53:19.255953 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248262 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 08:53:19.255953 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248264 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 08:53:19.255953 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248267 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 08:53:19.255953 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248269 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 08:53:19.255953 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248272 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 08:53:19.255953 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248275 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 08:53:19.255953 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248277 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 08:53:19.255953 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248281 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 08:53:19.255953 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248284 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 08:53:19.255953 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248287 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 08:53:19.255953 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248290 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 08:53:19.255953 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248292 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 08:53:19.256481 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248295 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 08:53:19.256481 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248298 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 08:53:19.256481 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248301 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 08:53:19.256481 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248303 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 08:53:19.256481 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248306 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 08:53:19.256481 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248309 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 08:53:19.256481 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248312 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 08:53:19.256481 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248315 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 08:53:19.256481 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248317 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 08:53:19.256481 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248320 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 08:53:19.256481 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248323 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 08:53:19.256481 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248325 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 08:53:19.256481 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248328 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 08:53:19.256481 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.248356 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 08:53:19.256481 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248397 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 08:53:19.256481 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248402 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 08:53:19.256862 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248405 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 08:53:19.256862 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248409 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 08:53:19.256862 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248412 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 08:53:19.256862 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248415 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 08:53:19.256862 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248418 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 08:53:19.256862 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248421 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 08:53:19.256862 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248423 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 08:53:19.256862 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248426 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 08:53:19.256862 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248429 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 08:53:19.256862 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248431 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 08:53:19.256862 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248434 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 08:53:19.256862 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248436 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 08:53:19.256862 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248441 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 08:53:19.256862 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248444 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 08:53:19.256862 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248447 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 08:53:19.256862 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248450 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 08:53:19.256862 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248453 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 08:53:19.256862 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248455 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 08:53:19.256862 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248458 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 08:53:19.257370 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248461 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 08:53:19.257370 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248465 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 08:53:19.257370 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248469 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 08:53:19.257370 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248473 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 08:53:19.257370 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248475 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 08:53:19.257370 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248478 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 08:53:19.257370 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248480 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 08:53:19.257370 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248483 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 08:53:19.257370 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248486 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 08:53:19.257370 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248488 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 08:53:19.257370 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248491 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 08:53:19.257370 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248493 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 08:53:19.257370 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248496 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 08:53:19.257370 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248499 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 08:53:19.257370 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248501 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 08:53:19.257370 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248503 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 08:53:19.257370 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248506 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 08:53:19.257370 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248509 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 08:53:19.257370 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248511 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 08:53:19.257829 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248514 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 08:53:19.257829 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248516 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 08:53:19.257829 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248519 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 08:53:19.257829 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248522 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 08:53:19.257829 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248524 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 08:53:19.257829 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248527 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 08:53:19.257829 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248530 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 08:53:19.257829 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248533 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 08:53:19.257829 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248536 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 08:53:19.257829 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248539 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 08:53:19.257829 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248542 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 08:53:19.257829 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248545 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 08:53:19.257829 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248547 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 08:53:19.257829 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248550 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 08:53:19.257829 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248553 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 08:53:19.257829 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248555 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 08:53:19.257829 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248559 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 08:53:19.257829 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248561 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 08:53:19.257829 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248564 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 08:53:19.258295 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248566 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 08:53:19.258295 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248569 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 08:53:19.258295 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248572 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 08:53:19.258295 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248574 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 08:53:19.258295 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248576 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 08:53:19.258295 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248579 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 08:53:19.258295 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248582 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 08:53:19.258295 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248584 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 08:53:19.258295 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248587 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 08:53:19.258295 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248590 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 08:53:19.258295 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248592 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 08:53:19.258295 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248595 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 08:53:19.258295 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248597 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 08:53:19.258295 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248600 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 08:53:19.258295 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248602 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 08:53:19.258295 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248604 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 08:53:19.258295 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248607 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 08:53:19.258295 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248610 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 08:53:19.258295 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248612 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 08:53:19.258743 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248616 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 08:53:19.258743 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248618 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 08:53:19.258743 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248621 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 08:53:19.258743 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248624 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 08:53:19.258743 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248627 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 08:53:19.258743 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248630 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 08:53:19.258743 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248632 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 08:53:19.258743 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248635 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 08:53:19.258743 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248638 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 08:53:19.258743 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248640 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 30 08:53:19.258743 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248643 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 08:53:19.258743 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248645 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 08:53:19.258743 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248648 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 08:53:19.258743 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248650 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 08:53:19.258743 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248653 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 08:53:19.258743 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248656 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 08:53:19.258743 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248658 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 08:53:19.258743 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248660 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 08:53:19.258743 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248663 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 08:53:19.258743 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248665 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 08:53:19.259233 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248668 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 08:53:19.259233 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248671 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 08:53:19.259233 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248673 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 08:53:19.259233 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248676 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 08:53:19.259233 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248680 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 08:53:19.259233 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248684 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 08:53:19.259233 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248688 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 08:53:19.259233 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.248691 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 08:53:19.259233 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.255892 2575 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 08:53:19.259233 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.255907 2575 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 08:53:19.259233 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.255921 2575 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 08:53:19.259233 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.255926 2575 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 08:53:19.259233 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.255983 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 08:53:19.259233 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.255988 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 08:53:19.259233 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.255992 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 08:53:19.259233 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.255996 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 08:53:19.259233 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.255999 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 08:53:19.259233 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256002 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 08:53:19.259233 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256005 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 08:53:19.259691 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256008 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 08:53:19.259691 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256010 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 08:53:19.259691 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256013 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 08:53:19.259691 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256016 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 08:53:19.259691 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256019 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 08:53:19.259691 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256022 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 08:53:19.259691 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256025 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 08:53:19.259691 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256027 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 08:53:19.259691 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256030 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 08:53:19.259691 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256034 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 08:53:19.259691 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256037 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 08:53:19.259691 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256041 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 08:53:19.259691 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256043 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 08:53:19.259691 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256046 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 08:53:19.259691 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256049 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 08:53:19.259691 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256053 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 08:53:19.259691 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256057 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 08:53:19.259691 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256060 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 08:53:19.259691 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256063 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 08:53:19.260148 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256066 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 08:53:19.260148 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256069 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 08:53:19.260148 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256072 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 08:53:19.260148 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256075 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 08:53:19.260148 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256077 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 08:53:19.260148 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256080 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 08:53:19.260148 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256100 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 08:53:19.260148 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256105 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 08:53:19.260148 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256107 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 08:53:19.260148 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256110 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 08:53:19.260148 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256113 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 08:53:19.260148 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256116 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 08:53:19.260148 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256119 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 08:53:19.260148 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256122 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 08:53:19.260148 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256124 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 08:53:19.260148 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256127 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 08:53:19.260148 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256130 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 08:53:19.260148 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256132 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 08:53:19.260148 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256135 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 08:53:19.260148 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256137 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 08:53:19.260622 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256140 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 08:53:19.260622 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256143 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 08:53:19.260622 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256145 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 08:53:19.260622 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256148 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 08:53:19.260622 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256150 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 08:53:19.260622 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256153 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 08:53:19.260622 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256155 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 30 08:53:19.260622 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256159 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 08:53:19.260622 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256161 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 08:53:19.260622 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256164 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 08:53:19.260622 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256167 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 08:53:19.260622 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256170 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 08:53:19.260622 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256173 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 08:53:19.260622 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256176 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 08:53:19.260622 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256179 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 08:53:19.260622 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256182 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 08:53:19.260622 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256185 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 08:53:19.260622 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256187 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 08:53:19.260622 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256190 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 08:53:19.260622 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256193 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 08:53:19.261212 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256196 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 08:53:19.261212 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256198 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 08:53:19.261212 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256201 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 08:53:19.261212 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256204 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 08:53:19.261212 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256207 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 08:53:19.261212 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256210 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 08:53:19.261212 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256212 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 08:53:19.261212 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256215 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 08:53:19.261212 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256217 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 08:53:19.261212 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256220 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 08:53:19.261212 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256222 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 08:53:19.261212 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256225 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 08:53:19.261212 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256227 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 08:53:19.261212 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256230 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 08:53:19.261212 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256232 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 08:53:19.261212 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256235 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 08:53:19.261212 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256237 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 08:53:19.261212 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256240 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 08:53:19.261212 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256242 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 08:53:19.261212 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256245 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 08:53:19.261727 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256248 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 08:53:19.261727 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256250 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 08:53:19.261727 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256253 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 08:53:19.261727 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256256 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 08:53:19.261727 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256259 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 08:53:19.261727 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256262 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 08:53:19.261727 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256264 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 08:53:19.261727 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256267 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 08:53:19.261727 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256269 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 08:53:19.261727 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256272 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 08:53:19.261727 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256275 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 08:53:19.261727 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256277 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 08:53:19.261727 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256280 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 08:53:19.261727 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256282 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 08:53:19.261727 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256285 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 08:53:19.261727 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256287 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 08:53:19.261727 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256290 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 08:53:19.261727 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256293 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 08:53:19.261727 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256296 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 08:53:19.262188 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256298 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 08:53:19.262188 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.256328 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 08:53:19.262188 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256488 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 08:53:19.262188 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256494 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 08:53:19.262188 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256497 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 08:53:19.262188 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256500 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 30 08:53:19.262188 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256503 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 08:53:19.262188 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256506 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 08:53:19.262188 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256508 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 08:53:19.262188 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256511 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 08:53:19.262188 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256513 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 08:53:19.262188 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256516 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 08:53:19.262188 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256519 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 08:53:19.262188 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256521 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 08:53:19.262188 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256525 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 08:53:19.262188 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256529 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 08:53:19.262575 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256532 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 08:53:19.262575 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256535 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 08:53:19.262575 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256538 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 08:53:19.262575 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256541 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 08:53:19.262575 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256543 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 08:53:19.262575 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256546 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 08:53:19.262575 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256549 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 08:53:19.262575 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256551 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 08:53:19.262575 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256554 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 08:53:19.262575 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256556 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 08:53:19.262575 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256559 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 08:53:19.262575 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256561 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 08:53:19.262575 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256564 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 08:53:19.262575 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256566 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 08:53:19.262575 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256569 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 08:53:19.262575 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256572 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 08:53:19.262575 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256574 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 08:53:19.262575 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256577 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 08:53:19.262575 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256579 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 08:53:19.262575 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256582 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 08:53:19.263033 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256584 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 08:53:19.263033 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256587 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 08:53:19.263033 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256590 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 08:53:19.263033 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256592 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 08:53:19.263033 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256595 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 08:53:19.263033 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256597 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 08:53:19.263033 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256600 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 08:53:19.263033 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256602 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 08:53:19.263033 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256605 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 08:53:19.263033 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256607 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 08:53:19.263033 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256609 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 08:53:19.263033 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256612 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 08:53:19.263033 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256615 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 08:53:19.263033 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256618 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 08:53:19.263033 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256620 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 08:53:19.263033 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256623 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 08:53:19.263033 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256625 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 08:53:19.263033 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256628 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 08:53:19.263033 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256631 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 08:53:19.263489 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256633 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 08:53:19.263489 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256636 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 08:53:19.263489 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256638 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 08:53:19.263489 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256641 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 08:53:19.263489 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256643 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 08:53:19.263489 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256646 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 08:53:19.263489 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256648 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 08:53:19.263489 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256651 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 08:53:19.263489 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256653 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 08:53:19.263489 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256656 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 08:53:19.263489 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256659 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 08:53:19.263489 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256661 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 08:53:19.263489 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256664 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 08:53:19.263489 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256666 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 08:53:19.263489 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256669 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 08:53:19.263489 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256671 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 08:53:19.263489 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256673 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 08:53:19.263489 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256676 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 08:53:19.263489 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256678 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 08:53:19.263489 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256681 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 08:53:19.263949 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256684 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 08:53:19.263949 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256686 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 08:53:19.263949 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256689 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 08:53:19.263949 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256691 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 08:53:19.263949 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256694 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 08:53:19.263949 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256696 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 08:53:19.263949 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256699 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 08:53:19.263949 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256701 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 08:53:19.263949 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256704 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 08:53:19.263949 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256707 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 08:53:19.263949 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256710 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 08:53:19.263949 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256714 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 08:53:19.263949 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256718 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 08:53:19.263949 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256721 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 08:53:19.263949 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256724 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 08:53:19.263949 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256727 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 08:53:19.263949 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256730 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 08:53:19.263949 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256732 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 08:53:19.264394 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256735 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 08:53:19.264394 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256737 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 08:53:19.264394 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256740 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 08:53:19.264394 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256742 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 08:53:19.264394 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256745 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 08:53:19.264394 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256748 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 08:53:19.264394 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256750 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 08:53:19.264394 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256753 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 08:53:19.264394 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256755 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 08:53:19.264394 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256758 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 08:53:19.264394 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256760 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 08:53:19.264394 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256762 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 08:53:19.264394 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256765 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 08:53:19.264394 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256768 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 08:53:19.264394 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256770 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 08:53:19.264394 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.256792 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 08:53:19.264769 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256829 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 08:53:19.264769 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256833 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 08:53:19.264769 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256836 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 08:53:19.264769 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256840 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 08:53:19.264769 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256843 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 08:53:19.264769 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256846 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 08:53:19.264769 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256849 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 08:53:19.264769 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256854 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 08:53:19.264769 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256857 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 08:53:19.264769 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256859 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 08:53:19.264769 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256862 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 08:53:19.264769 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256865 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 08:53:19.264769 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256867 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 08:53:19.264769 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256870 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 08:53:19.264769 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256873 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 08:53:19.264769 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256876 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 08:53:19.264769 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256879 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 08:53:19.264769 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256882 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 08:53:19.264769 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256884 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 08:53:19.264769 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256887 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 08:53:19.265250 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256889 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 08:53:19.265250 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256892 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 08:53:19.265250 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256895 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 08:53:19.265250 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256898 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 08:53:19.265250 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256900 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 08:53:19.265250 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256903 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 08:53:19.265250 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256905 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 08:53:19.265250 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256908 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 08:53:19.265250 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256910 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 08:53:19.265250 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256913 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 08:53:19.265250 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256916 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 08:53:19.265250 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256918 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 08:53:19.265250 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256921 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 08:53:19.265250 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256924 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 08:53:19.265250 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256926 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 08:53:19.265250 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256929 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 08:53:19.265250 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256931 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 08:53:19.265250 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256934 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 08:53:19.265250 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256936 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 08:53:19.265683 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256940 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 08:53:19.265683 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256943 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 08:53:19.265683 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256945 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 08:53:19.265683 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256948 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 08:53:19.265683 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256951 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 08:53:19.265683 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256953 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 08:53:19.265683 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256956 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 08:53:19.265683 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256958 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 08:53:19.265683 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256961 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 08:53:19.265683 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256963 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 08:53:19.265683 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256966 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 08:53:19.265683 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256968 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 08:53:19.265683 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256971 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 08:53:19.265683 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256973 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 08:53:19.265683 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256976 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 08:53:19.265683 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256979 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 08:53:19.265683 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256982 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 08:53:19.265683 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256985 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 08:53:19.265683 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256988 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 08:53:19.265683 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256992 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 08:53:19.266157 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256994 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 08:53:19.266157 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256997 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 08:53:19.266157 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.256999 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 08:53:19.266157 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.257002 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 08:53:19.266157 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.257004 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 08:53:19.266157 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.257007 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 08:53:19.266157 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.257010 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 08:53:19.266157 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.257012 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 08:53:19.266157 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.257016 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 08:53:19.266157 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.257020 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 08:53:19.266157 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.257023 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 08:53:19.266157 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.257026 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 08:53:19.266157 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.257030 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 08:53:19.266157 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.257032 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 08:53:19.266157 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.257035 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 08:53:19.266157 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.257037 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 08:53:19.266157 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.257040 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 08:53:19.266157 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.257042 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 08:53:19.266157 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.257045 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 08:53:19.266612 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.257047 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 08:53:19.266612 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.257050 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 08:53:19.266612 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.257052 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 08:53:19.266612 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.257055 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 08:53:19.266612 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.257057 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 08:53:19.266612 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.257060 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 08:53:19.266612 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.257062 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 08:53:19.266612 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.257067 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 08:53:19.266612 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.257069 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 08:53:19.266612 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.257072 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 08:53:19.266612 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.257075 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 08:53:19.266612 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.257078 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 08:53:19.266612 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.257080 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 08:53:19.266612 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.257097 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 08:53:19.266612 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.257100 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 30 08:53:19.266612 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.257103 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 08:53:19.266612 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.257105 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 08:53:19.266612 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.257108 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 08:53:19.266612 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.257111 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 08:53:19.266612 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.257113 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 08:53:19.267077 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.257116 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 08:53:19.267077 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.257119 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 08:53:19.267077 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.257121 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 08:53:19.267077 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.257125 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 08:53:19.267077 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.257128 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 08:53:19.267077 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.257133 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 08:53:19.267077 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.257136 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 08:53:19.267077 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:19.257140 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 08:53:19.267077 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.257195 2575 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 08:53:19.267077 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.259683 2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 08:53:19.267077 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.260683 2575 server.go:1014] "Starting client certificate rotation" Oct 30 08:53:19.267077 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.260781 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 08:53:19.267077 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.260824 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 08:53:19.267077 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.262929 2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 08:53:19.267077 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.264071 2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 08:53:19.277567 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.277548 2575 log.go:25] "Validated CRI v1 runtime API" Oct 30 08:53:19.280596 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.280582 2575 log.go:25] "Validated CRI v1 image API" Oct 30 08:53:19.281464 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.281440 2575 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 08:53:19.285680 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.285659 2575 fs.go:135] Filesystem UUIDs: map[23df4a2e-1e9e-44aa-bb40-ce1370554055:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 c3861628-df76-46ce-9483-cd17d3ddbd4a:/dev/nvme0n1p4] Oct 30 08:53:19.285761 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.285680 2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 08:53:19.290273 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.290132 2575 manager.go:217] Machine: {Timestamp:2026-10-30 08:53:19.289634901 +0000 UTC m=+0.377902682 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099838 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23c0e560d599e449473197f8f2cd77 SystemUUID:ec23c0e5-60d5-99e4-4947-3197f8f2cd77 BootID:8f978fbd-e73a-4901-aa4f-630411ad5fa4 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c0:d4:1d:3f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c0:d4:1d:3f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3e:9e:95:ec:dd:ad Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 08:53:19.290273 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.290269 2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 08:53:19.290411 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.290345 2575 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 08:53:19.294388 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.294367 2575 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 08:53:19.294528 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.294389 2575 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-244.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 08:53:19.294577 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.294530 2575 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 08:53:19.294577 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.294539 2575 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 08:53:19.294577 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.294549 2575 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 08:53:19.294577 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.294562 2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 08:53:19.295631 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.295620 2575 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 08:53:19.299892 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.299879 2575 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 08:53:19.300007 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.299997 2575 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 08:53:19.300819 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.300810 2575 kubelet.go:493] "Attempting to sync node with API server" Oct 30 08:53:19.300853 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.300822 2575 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 08:53:19.300853 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.300832 2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 08:53:19.300853 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.300841 2575 kubelet.go:405] "Adding apiserver pod source" Oct 30 08:53:19.300853 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.300848 2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 08:53:19.301269 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.301254 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 08:53:19.303832 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.303810 2575 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 08:53:19.310503 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.310490 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 08:53:19.311518 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.311505 2575 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 08:53:19.311562 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.311537 2575 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 08:53:19.311724 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.311714 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 08:53:19.311753 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.311731 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 08:53:19.311753 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.311737 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 08:53:19.311753 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.311743 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 08:53:19.311753 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.311750 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 08:53:19.311851 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.311758 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 08:53:19.312496 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.312488 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 08:53:19.312524 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.312498 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 08:53:19.312524 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.312505 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 08:53:19.312524 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.312511 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 08:53:19.312524 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.312520 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 08:53:19.312694 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.312529 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 08:53:19.314424 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.314414 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 08:53:19.314460 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.314426 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 08:53:19.318966 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.318952 2575 server.go:1263] "Started kubelet" Oct 30 08:53:19.319046 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.319024 2575 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 08:53:19.319153 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.319081 2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 08:53:19.319208 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.319177 2575 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 08:53:19.319695 ip-10-0-143-244 systemd[1]: Started Kubernetes Kubelet. Oct 30 08:53:19.320818 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.320721 2575 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 08:53:19.323948 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.323877 2575 server.go:317] "Adding debug handlers to kubelet server" Oct 30 08:53:19.329348 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:19.329330 2575 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 08:53:19.330756 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.330444 2575 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 08:53:19.330756 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:19.330578 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-244.ec2.internal\" not found" Oct 30 08:53:19.331025 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.330942 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 08:53:19.331230 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.331208 2575 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 08:53:19.331230 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.331229 2575 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 08:53:19.331446 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.331431 2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 08:53:19.331540 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.331527 2575 reconstruct.go:97] "Volume reconstruction finished" Oct 30 08:53:19.331589 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.331540 2575 reconciler.go:29] "Reconciler: start to sync state" Oct 30 08:53:19.332932 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.332914 2575 factory.go:153] Registering CRI-O factory Oct 30 08:53:19.332932 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.332933 2575 factory.go:223] Registration of the crio container factory successfully Oct 30 08:53:19.333073 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.333000 2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 08:53:19.333073 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.333008 2575 factory.go:55] Registering systemd factory Oct 30 08:53:19.333073 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.333016 2575 factory.go:223] Registration of the systemd container factory successfully Oct 30 08:53:19.333073 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.333046 2575 factory.go:103] Registering Raw factory Oct 30 08:53:19.333073 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.333059 2575 manager.go:1199] Started watching for new ooms in manager Oct 30 08:53:19.337901 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.334168 2575 manager.go:319] Starting recovery of all containers Oct 30 08:53:19.345249 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:19.345222 2575 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-244.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 08:53:19.345331 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.345254 2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-244.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 08:53:19.346516 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.346500 2575 manager.go:324] Recovery completed Oct 30 08:53:19.351376 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.351363 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 08:53:19.353540 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.353522 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.ec2.internal" event="NodeHasSufficientMemory" Oct 30 08:53:19.353621 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.353557 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 08:53:19.353621 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.353573 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.ec2.internal" event="NodeHasSufficientPID" Oct 30 08:53:19.354176 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.354163 2575 cpu_manager.go:226] "Starting" policy="none" Oct 30 08:53:19.354176 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.354173 2575 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 08:53:19.354337 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.354186 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 08:53:19.354337 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.354208 2575 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 08:53:19.354967 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.354949 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.ec2.internal" event="NodeHasSufficientMemory" Oct 30 08:53:19.355040 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.354977 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 08:53:19.355040 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.354998 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.ec2.internal" event="NodeHasSufficientPID" Oct 30 08:53:19.356353 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.356334 2575 policy_none.go:50] "Start" Oct 30 08:53:19.356353 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.356354 2575 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 08:53:19.356471 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.356386 2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 08:53:19.356471 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.356405 2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 08:53:19.356471 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:19.355317 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-244.ec2.internal.18e342afc5358f6e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-244.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-244.ec2.internal,},FirstTimestamp:2026-10-30 08:53:19.31892107 +0000 UTC m=+0.407188852,LastTimestamp:2026-10-30 08:53:19.31892107 +0000 UTC m=+0.407188852,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-244.ec2.internal,}" Oct 30 08:53:19.356471 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.356455 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wnvwt" Oct 30 08:53:19.359602 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.359587 2575 policy_none.go:44] "Start" Oct 30 08:53:19.364498 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:19.364467 2575 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-244.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 08:53:19.364835 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:19.364468 2575 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 08:53:19.377564 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.377546 2575 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 08:53:19.410314 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.393829 2575 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 08:53:19.410314 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.399097 2575 manager.go:341] "Starting Device Plugin manager" Oct 30 08:53:19.410314 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:19.399760 2575 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 08:53:19.410314 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.399775 2575 server.go:87] "Starting device plugin registration server" Oct 30 08:53:19.410314 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.400006 2575 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 08:53:19.410314 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.400015 2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 08:53:19.410314 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.400142 2575 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 08:53:19.410314 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.400205 2575 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 08:53:19.410314 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.400210 2575 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 08:53:19.410314 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:19.400652 2575 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 08:53:19.410314 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:19.400694 2575 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-244.ec2.internal\" not found" Oct 30 08:53:19.410314 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.402379 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 08:53:19.410314 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.403546 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 08:53:19.410314 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.403570 2575 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 08:53:19.410314 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.403601 2575 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 08:53:19.410314 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:19.403640 2575 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 08:53:19.417469 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.417448 2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 08:53:19.500164 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.500111 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 08:53:19.500987 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.500973 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.ec2.internal" event="NodeHasSufficientMemory" Oct 30 08:53:19.501043 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.501002 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 08:53:19.501043 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.501014 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.ec2.internal" event="NodeHasSufficientPID" Oct 30 08:53:19.501142 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.501051 2575 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-244.ec2.internal" Oct 30 08:53:19.504147 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.504122 2575 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-244.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-244.ec2.internal"] Oct 30 08:53:19.504201 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.504190 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 08:53:19.504906 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.504889 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.ec2.internal" event="NodeHasSufficientMemory" Oct 30 08:53:19.504971 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.504920 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 08:53:19.504971 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.504930 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.ec2.internal" event="NodeHasSufficientPID" Oct 30 08:53:19.507080 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.507068 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 08:53:19.507275 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.507258 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-244.ec2.internal" Oct 30 08:53:19.507323 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.507295 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 08:53:19.507796 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.507781 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.ec2.internal" event="NodeHasSufficientMemory" Oct 30 08:53:19.507862 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.507810 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 08:53:19.507862 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.507820 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.ec2.internal" event="NodeHasSufficientPID" Oct 30 08:53:19.507927 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.507784 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.ec2.internal" event="NodeHasSufficientMemory" Oct 30 08:53:19.507927 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.507891 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 08:53:19.507927 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.507904 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.ec2.internal" event="NodeHasSufficientPID" Oct 30 08:53:19.510548 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.510534 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-244.ec2.internal" Oct 30 08:53:19.510609 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.510559 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 08:53:19.511281 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.511266 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.ec2.internal" event="NodeHasSufficientMemory" Oct 30 08:53:19.511393 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.511294 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 08:53:19.511393 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.511308 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.ec2.internal" event="NodeHasSufficientPID" Oct 30 08:53:19.519910 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.519895 2575 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-244.ec2.internal" Oct 30 08:53:19.520003 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:19.519919 2575 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-244.ec2.internal\": node \"ip-10-0-143-244.ec2.internal\" not found" Oct 30 08:53:19.541684 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:19.541669 2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-244.ec2.internal\" not found" node="ip-10-0-143-244.ec2.internal" Oct 30 08:53:19.544210 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:19.544186 2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-244.ec2.internal\" not found" node="ip-10-0-143-244.ec2.internal" Oct 30 08:53:19.557058 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:19.557043 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-244.ec2.internal\" not found" Oct 30 08:53:19.632943 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.632913 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4bdabdb48827c33749d04f26c951dd86-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-244.ec2.internal\" (UID: \"4bdabdb48827c33749d04f26c951dd86\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-244.ec2.internal" Oct 30 08:53:19.632943 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.632943 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4bdabdb48827c33749d04f26c951dd86-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-244.ec2.internal\" (UID: \"4bdabdb48827c33749d04f26c951dd86\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-244.ec2.internal" Oct 30 08:53:19.633138 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.632961 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e62eebd98344a1611517e7c60f1efe61-config\") pod \"kube-apiserver-proxy-ip-10-0-143-244.ec2.internal\" (UID: \"e62eebd98344a1611517e7c60f1efe61\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-244.ec2.internal" Oct 30 08:53:19.658010 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:19.657980 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-244.ec2.internal\" not found" Oct 30 08:53:19.733209 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.733188 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4bdabdb48827c33749d04f26c951dd86-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-244.ec2.internal\" (UID: \"4bdabdb48827c33749d04f26c951dd86\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-244.ec2.internal" Oct 30 08:53:19.733292 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.733214 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4bdabdb48827c33749d04f26c951dd86-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-244.ec2.internal\" (UID: \"4bdabdb48827c33749d04f26c951dd86\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-244.ec2.internal" Oct 30 08:53:19.733292 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.733231 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e62eebd98344a1611517e7c60f1efe61-config\") pod \"kube-apiserver-proxy-ip-10-0-143-244.ec2.internal\" (UID: \"e62eebd98344a1611517e7c60f1efe61\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-244.ec2.internal" Oct 30 08:53:19.733292 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.733282 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e62eebd98344a1611517e7c60f1efe61-config\") pod \"kube-apiserver-proxy-ip-10-0-143-244.ec2.internal\" (UID: \"e62eebd98344a1611517e7c60f1efe61\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-244.ec2.internal" Oct 30 08:53:19.733384 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.733293 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4bdabdb48827c33749d04f26c951dd86-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-244.ec2.internal\" (UID: \"4bdabdb48827c33749d04f26c951dd86\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-244.ec2.internal" Oct 30 08:53:19.733384 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.733293 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4bdabdb48827c33749d04f26c951dd86-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-244.ec2.internal\" (UID: \"4bdabdb48827c33749d04f26c951dd86\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-244.ec2.internal" Oct 30 08:53:19.758522 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:19.758470 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-244.ec2.internal\" not found" Oct 30 08:53:19.847178 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.847142 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-244.ec2.internal" Oct 30 08:53:19.849323 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:19.849306 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-244.ec2.internal" Oct 30 08:53:19.859144 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:19.859020 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-244.ec2.internal\" not found" Oct 30 08:53:19.959496 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:19.959465 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-244.ec2.internal\" not found" Oct 30 08:53:21.636325 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:21.636263 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-244.ec2.internal\" not found" Oct 30 08:53:21.736859 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:21.736835 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-244.ec2.internal\" not found" Oct 30 08:53:21.817332 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:21.817311 2575 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 08:53:21.836286 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:21.836254 2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 08:53:21.836472 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:21.836446 2575 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 08:53:21.836548 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:21.836450 2575 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 08:53:21.836548 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:21.836491 2575 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 08:53:21.878245 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:21.878218 2575 apiserver.go:52] "Watching apiserver" Oct 30 08:53:21.908453 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:21.908401 2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-244.ec2.internal" Oct 30 08:53:21.908562 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:21.908454 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 08:53:21.938469 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:21.938440 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 08:48:20 +0000 UTC" deadline="2028-04-07 23:22:09.08151529 +0000 UTC" Oct 30 08:53:21.938547 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:21.938471 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12614h28m47.143051414s" Oct 30 08:53:21.947012 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:21.946995 2575 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 08:53:21.947279 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:21.947258 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-s6q7n","kube-system/global-pull-secret-syncer-vb26w","kube-system/konnectivity-agent-h5vr9","openshift-multus/multus-rd9pd","openshift-multus/network-metrics-daemon-z66mv","openshift-network-diagnostics/network-check-target-mz2kl","openshift-network-operator/iptables-alerter-6cncl","openshift-multus/multus-additional-cni-plugins-v8qzv","openshift-dns/node-resolver-lq2p2","openshift-image-registry/node-ca-xqjlj","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6r2gg","openshift-cluster-node-tuning-operator/tuned-pvqcd"] Oct 30 08:53:21.948280 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:21.948252 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bdabdb48827c33749d04f26c951dd86.slice/crio-f1508eeaa257ef4ba2b2e4666b7f69913c95ef0b0b585df918cf1d1557008e0c WatchSource:0}: Error finding container f1508eeaa257ef4ba2b2e4666b7f69913c95ef0b0b585df918cf1d1557008e0c: Status 404 returned error can't find the container with id f1508eeaa257ef4ba2b2e4666b7f69913c95ef0b0b585df918cf1d1557008e0c Oct 30 08:53:21.949217 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:21.949199 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode62eebd98344a1611517e7c60f1efe61.slice/crio-de620c47e9a6a59465581463393d917cb0141b362f379b7199f6dd337451c103 WatchSource:0}: Error finding container de620c47e9a6a59465581463393d917cb0141b362f379b7199f6dd337451c103: Status 404 returned error can't find the container with id de620c47e9a6a59465581463393d917cb0141b362f379b7199f6dd337451c103 Oct 30 08:53:21.951216 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:21.951199 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:53:21.955854 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:21.955827 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vb26w" Oct 30 08:53:21.956217 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:21.955867 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 08:53:21.956217 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:21.956199 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vb26w" podUID="5e591466-fcf6-469d-9c3d-cde0767f4dde" Oct 30 08:53:21.957743 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:21.957724 2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 08:53:21.957816 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:21.957789 2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-244.ec2.internal" Oct 30 08:53:21.958605 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:21.958591 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-h5vr9" Oct 30 08:53:21.961027 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:21.960866 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rd9pd" Oct 30 08:53:21.963165 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:21.963142 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z66mv" Oct 30 08:53:21.963256 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:21.963210 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z66mv" podUID="4c94bc06-63d6-454f-9a56-1253c0315672" Oct 30 08:53:21.965508 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:21.965485 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 08:53:21.965592 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:21.965551 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 08:53:21.965645 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:21.965623 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mz2kl" Oct 30 08:53:21.965698 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:21.965668 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 08:53:21.965698 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:21.965668 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 08:53:21.965698 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:21.965689 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 08:53:21.965698 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:21.965685 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mz2kl" podUID="0278c964-1ae2-42e7-a5b3-1df265e86edc" Oct 30 08:53:21.965844 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:21.965674 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wdt5m\"" Oct 30 08:53:21.965844 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:21.965763 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 08:53:21.967829 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:21.967814 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6cncl" Oct 30 08:53:21.970228 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:21.970207 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-v8qzv" Oct 30 08:53:21.972430 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:21.972413 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lq2p2" Oct 30 08:53:21.974822 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:21.974806 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xqjlj" Oct 30 08:53:21.977027 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:21.977009 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6r2gg" Oct 30 08:53:21.977654 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:21.977640 2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 08:53:21.979267 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:21.979251 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-143-244.ec2.internal"] Oct 30 08:53:21.979321 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:21.979277 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-244.ec2.internal"] Oct 30 08:53:21.979733 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:21.979721 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:53:21.983479 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:21.983366 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-244.ec2.internal" event={"ID":"4bdabdb48827c33749d04f26c951dd86","Type":"ContainerStarted","Data":"f1508eeaa257ef4ba2b2e4666b7f69913c95ef0b0b585df918cf1d1557008e0c"} Oct 30 08:53:21.984399 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:21.984364 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-244.ec2.internal" event={"ID":"e62eebd98344a1611517e7c60f1efe61","Type":"ContainerStarted","Data":"de620c47e9a6a59465581463393d917cb0141b362f379b7199f6dd337451c103"} Oct 30 08:53:22.009226 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.009206 2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 08:53:22.013097 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.013078 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8chm\" (UniqueName: \"kubernetes.io/projected/82a05e91-5a9d-4670-9882-c12ee332ca04-kube-api-access-s8chm\") pod \"aws-ebs-csi-driver-node-6r2gg\" (UID: \"82a05e91-5a9d-4670-9882-c12ee332ca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6r2gg" Oct 30 08:53:22.013191 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.013104 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c67a6b6-a3d8-49bc-a678-b203ef4201a9-tmp-dir\") pod \"node-resolver-lq2p2\" (UID: \"9c67a6b6-a3d8-49bc-a678-b203ef4201a9\") " pod="openshift-dns/node-resolver-lq2p2" Oct 30 08:53:22.013191 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.013120 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2e880cd8-58f9-41e4-8795-4f5e978e18f5-etc-sysconfig\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:53:22.013191 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.013144 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2e880cd8-58f9-41e4-8795-4f5e978e18f5-etc-kubernetes\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:53:22.013320 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.013220 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5e591466-fcf6-469d-9c3d-cde0767f4dde-kubelet-config\") pod \"global-pull-secret-syncer-vb26w\" (UID: \"5e591466-fcf6-469d-9c3d-cde0767f4dde\") " pod="kube-system/global-pull-secret-syncer-vb26w" Oct 30 08:53:22.013320 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.013252 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2e880cd8-58f9-41e4-8795-4f5e978e18f5-sys\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:53:22.013320 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.013274 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-host-var-lib-cni-multus\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:53:22.013461 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.013314 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5e591466-fcf6-469d-9c3d-cde0767f4dde-dbus\") pod \"global-pull-secret-syncer-vb26w\" (UID: \"5e591466-fcf6-469d-9c3d-cde0767f4dde\") " pod="kube-system/global-pull-secret-syncer-vb26w" Oct 30 08:53:22.013461 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.013342 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1f51db5a-5720-4dea-ad02-f336b35d43b1-ovnkube-config\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:53:22.013461 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.013365 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-host-run-multus-certs\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:53:22.013461 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.013406 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/82a05e91-5a9d-4670-9882-c12ee332ca04-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6r2gg\" (UID: \"82a05e91-5a9d-4670-9882-c12ee332ca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6r2gg" Oct 30 08:53:22.013461 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.013444 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2e880cd8-58f9-41e4-8795-4f5e978e18f5-lib-modules\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:53:22.013670 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.013476 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e880cd8-58f9-41e4-8795-4f5e978e18f5-var-lib-kubelet\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:53:22.013670 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.013542 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-run-systemd\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:53:22.013670 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.013567 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-host-run-ovn-kubernetes\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:53:22.013670 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.013593 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1f51db5a-5720-4dea-ad02-f336b35d43b1-env-overrides\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:53:22.013670 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.013627 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-hostroot\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:53:22.013670 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.013657 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6f09a608-14be-447e-8e6e-eaf0a0031554-cni-binary-copy\") pod \"multus-additional-cni-plugins-v8qzv\" (UID: \"6f09a608-14be-447e-8e6e-eaf0a0031554\") " pod="openshift-multus/multus-additional-cni-plugins-v8qzv" Oct 30 08:53:22.013919 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.013678 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1b64ad2b-20eb-4a29-93b9-ccc9a217135d-host\") pod \"node-ca-xqjlj\" (UID: \"1b64ad2b-20eb-4a29-93b9-ccc9a217135d\") " pod="openshift-image-registry/node-ca-xqjlj" Oct 30 08:53:22.013919 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.013694 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-var-lib-openvswitch\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:53:22.013919 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.013725 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-run-ovn\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:53:22.013919 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.013753 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-multus-cni-dir\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:53:22.013919 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.013778 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-host-run-k8s-cni-cncf-io\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:53:22.013919 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.013803 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/82a05e91-5a9d-4670-9882-c12ee332ca04-registration-dir\") pod \"aws-ebs-csi-driver-node-6r2gg\" (UID: \"82a05e91-5a9d-4670-9882-c12ee332ca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6r2gg" Oct 30 08:53:22.013919 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.013830 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/82a05e91-5a9d-4670-9882-c12ee332ca04-sys-fs\") pod \"aws-ebs-csi-driver-node-6r2gg\" (UID: \"82a05e91-5a9d-4670-9882-c12ee332ca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6r2gg" Oct 30 08:53:22.013919 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.013853 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8x4w\" (UniqueName: \"kubernetes.io/projected/4c94bc06-63d6-454f-9a56-1253c0315672-kube-api-access-x8x4w\") pod \"network-metrics-daemon-z66mv\" (UID: \"4c94bc06-63d6-454f-9a56-1253c0315672\") " pod="openshift-multus/network-metrics-daemon-z66mv" Oct 30 08:53:22.013919 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.013876 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-host-kubelet\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:53:22.013919 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.013913 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-host-run-netns\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:53:22.014353 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.013941 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3024e1a0-8ea6-4fc3-bb0f-e5997585eb86-host-slash\") pod \"iptables-alerter-6cncl\" (UID: \"3024e1a0-8ea6-4fc3-bb0f-e5997585eb86\") " pod="openshift-network-operator/iptables-alerter-6cncl" Oct 30 08:53:22.014353 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.013961 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtm2w\" (UniqueName: \"kubernetes.io/projected/1b64ad2b-20eb-4a29-93b9-ccc9a217135d-kube-api-access-mtm2w\") pod \"node-ca-xqjlj\" (UID: \"1b64ad2b-20eb-4a29-93b9-ccc9a217135d\") " pod="openshift-image-registry/node-ca-xqjlj" Oct 30 08:53:22.014353 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.013982 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c94bc06-63d6-454f-9a56-1253c0315672-metrics-certs\") pod \"network-metrics-daemon-z66mv\" (UID: \"4c94bc06-63d6-454f-9a56-1253c0315672\") " pod="openshift-multus/network-metrics-daemon-z66mv" Oct 30 08:53:22.014353 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.014005 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2e880cd8-58f9-41e4-8795-4f5e978e18f5-etc-systemd\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:53:22.014353 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.014036 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5e591466-fcf6-469d-9c3d-cde0767f4dde-original-pull-secret\") pod \"global-pull-secret-syncer-vb26w\" (UID: \"5e591466-fcf6-469d-9c3d-cde0767f4dde\") " pod="kube-system/global-pull-secret-syncer-vb26w" Oct 30 08:53:22.014353 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.014068 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/634749a3-dbc0-41f4-9e97-bf3d25f9b6a8-konnectivity-ca\") pod \"konnectivity-agent-h5vr9\" (UID: \"634749a3-dbc0-41f4-9e97-bf3d25f9b6a8\") " pod="kube-system/konnectivity-agent-h5vr9" Oct 30 08:53:22.014353 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.014091 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-log-socket\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:53:22.014353 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.014125 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1f51db5a-5720-4dea-ad02-f336b35d43b1-ovn-node-metrics-cert\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:53:22.014353 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.014149 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-multus-conf-dir\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:53:22.014353 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.014172 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6f09a608-14be-447e-8e6e-eaf0a0031554-cnibin\") pod \"multus-additional-cni-plugins-v8qzv\" (UID: \"6f09a608-14be-447e-8e6e-eaf0a0031554\") " pod="openshift-multus/multus-additional-cni-plugins-v8qzv" Oct 30 08:53:22.014353 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.014195 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2e880cd8-58f9-41e4-8795-4f5e978e18f5-run\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:53:22.014353 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.014232 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2e880cd8-58f9-41e4-8795-4f5e978e18f5-etc-tuned\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:53:22.014353 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.014250 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-etc-openvswitch\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:53:22.014353 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.014269 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5679q\" (UniqueName: \"kubernetes.io/projected/1f51db5a-5720-4dea-ad02-f336b35d43b1-kube-api-access-5679q\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:53:22.014353 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.014289 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-host-var-lib-cni-bin\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:53:22.014353 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.014312 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6f09a608-14be-447e-8e6e-eaf0a0031554-system-cni-dir\") pod \"multus-additional-cni-plugins-v8qzv\" (UID: \"6f09a608-14be-447e-8e6e-eaf0a0031554\") " pod="openshift-multus/multus-additional-cni-plugins-v8qzv" Oct 30 08:53:22.014902 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.014331 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6f09a608-14be-447e-8e6e-eaf0a0031554-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v8qzv\" (UID: \"6f09a608-14be-447e-8e6e-eaf0a0031554\") " pod="openshift-multus/multus-additional-cni-plugins-v8qzv" Oct 30 08:53:22.014902 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.014346 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1b64ad2b-20eb-4a29-93b9-ccc9a217135d-serviceca\") pod \"node-ca-xqjlj\" (UID: \"1b64ad2b-20eb-4a29-93b9-ccc9a217135d\") " pod="openshift-image-registry/node-ca-xqjlj" Oct 30 08:53:22.014902 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.014413 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e880cd8-58f9-41e4-8795-4f5e978e18f5-tmp\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:53:22.014902 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.014443 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-run-openvswitch\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:53:22.014902 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.014507 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-host-cni-netd\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:53:22.014902 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.014537 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-multus-socket-dir-parent\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:53:22.014902 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.014564 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzmnn\" (UniqueName: \"kubernetes.io/projected/6f09a608-14be-447e-8e6e-eaf0a0031554-kube-api-access-nzmnn\") pod \"multus-additional-cni-plugins-v8qzv\" (UID: \"6f09a608-14be-447e-8e6e-eaf0a0031554\") " pod="openshift-multus/multus-additional-cni-plugins-v8qzv" Oct 30 08:53:22.014902 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.014592 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9c67a6b6-a3d8-49bc-a678-b203ef4201a9-hosts-file\") pod \"node-resolver-lq2p2\" (UID: \"9c67a6b6-a3d8-49bc-a678-b203ef4201a9\") " pod="openshift-dns/node-resolver-lq2p2" Oct 30 08:53:22.014902 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.014610 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2e880cd8-58f9-41e4-8795-4f5e978e18f5-etc-sysctl-d\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:53:22.014902 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.014631 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-host-run-netns\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:53:22.014902 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.014662 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-etc-kubernetes\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:53:22.014902 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.014681 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/82a05e91-5a9d-4670-9882-c12ee332ca04-etc-selinux\") pod \"aws-ebs-csi-driver-node-6r2gg\" (UID: \"82a05e91-5a9d-4670-9882-c12ee332ca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6r2gg" Oct 30 08:53:22.014902 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.014707 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2e880cd8-58f9-41e4-8795-4f5e978e18f5-etc-modprobe-d\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:53:22.014902 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.014750 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2e880cd8-58f9-41e4-8795-4f5e978e18f5-etc-sysctl-conf\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:53:22.014902 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.014765 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-host-slash\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:53:22.014902 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.014787 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1f51db5a-5720-4dea-ad02-f336b35d43b1-ovnkube-script-lib\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:53:22.015427 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.014815 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfhrk\" (UniqueName: \"kubernetes.io/projected/0278c964-1ae2-42e7-a5b3-1df265e86edc-kube-api-access-rfhrk\") pod \"network-check-target-mz2kl\" (UID: \"0278c964-1ae2-42e7-a5b3-1df265e86edc\") " pod="openshift-network-diagnostics/network-check-target-mz2kl" Oct 30 08:53:22.015427 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.014835 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/82a05e91-5a9d-4670-9882-c12ee332ca04-device-dir\") pod \"aws-ebs-csi-driver-node-6r2gg\" (UID: \"82a05e91-5a9d-4670-9882-c12ee332ca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6r2gg" Oct 30 08:53:22.015427 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.014872 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/634749a3-dbc0-41f4-9e97-bf3d25f9b6a8-agent-certs\") pod \"konnectivity-agent-h5vr9\" (UID: \"634749a3-dbc0-41f4-9e97-bf3d25f9b6a8\") " pod="kube-system/konnectivity-agent-h5vr9" Oct 30 08:53:22.015427 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.014895 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmxcr\" (UniqueName: \"kubernetes.io/projected/9c67a6b6-a3d8-49bc-a678-b203ef4201a9-kube-api-access-rmxcr\") pod \"node-resolver-lq2p2\" (UID: \"9c67a6b6-a3d8-49bc-a678-b203ef4201a9\") " pod="openshift-dns/node-resolver-lq2p2" Oct 30 08:53:22.015427 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.014911 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-systemd-units\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:53:22.015427 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.014932 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-cnibin\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:53:22.015427 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.014960 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-host-var-lib-kubelet\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:53:22.015427 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.014981 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5d95\" (UniqueName: \"kubernetes.io/projected/3024e1a0-8ea6-4fc3-bb0f-e5997585eb86-kube-api-access-v5d95\") pod \"iptables-alerter-6cncl\" (UID: \"3024e1a0-8ea6-4fc3-bb0f-e5997585eb86\") " pod="openshift-network-operator/iptables-alerter-6cncl" Oct 30 08:53:22.015427 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.014996 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6f09a608-14be-447e-8e6e-eaf0a0031554-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v8qzv\" (UID: \"6f09a608-14be-447e-8e6e-eaf0a0031554\") " pod="openshift-multus/multus-additional-cni-plugins-v8qzv" Oct 30 08:53:22.015427 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.015021 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/82a05e91-5a9d-4670-9882-c12ee332ca04-socket-dir\") pod \"aws-ebs-csi-driver-node-6r2gg\" (UID: \"82a05e91-5a9d-4670-9882-c12ee332ca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6r2gg" Oct 30 08:53:22.015427 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.015050 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2e880cd8-58f9-41e4-8795-4f5e978e18f5-host\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:53:22.015427 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.015088 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcnc5\" (UniqueName: \"kubernetes.io/projected/2e880cd8-58f9-41e4-8795-4f5e978e18f5-kube-api-access-vcnc5\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:53:22.015427 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.015115 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-node-log\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:53:22.015427 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.015142 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-host-cni-bin\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:53:22.015427 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.015164 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:53:22.015427 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.015199 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6d1fb12b-1a27-4574-be55-140884eb868a-multus-daemon-config\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:53:22.015862 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.015221 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6f09a608-14be-447e-8e6e-eaf0a0031554-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v8qzv\" (UID: \"6f09a608-14be-447e-8e6e-eaf0a0031554\") " pod="openshift-multus/multus-additional-cni-plugins-v8qzv" Oct 30 08:53:22.015862 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.015244 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-system-cni-dir\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:53:22.015862 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.015282 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6d1fb12b-1a27-4574-be55-140884eb868a-cni-binary-copy\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:53:22.015862 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.015322 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9wwl\" (UniqueName: \"kubernetes.io/projected/6d1fb12b-1a27-4574-be55-140884eb868a-kube-api-access-m9wwl\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:53:22.015862 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.015348 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3024e1a0-8ea6-4fc3-bb0f-e5997585eb86-iptables-alerter-script\") pod \"iptables-alerter-6cncl\" (UID: \"3024e1a0-8ea6-4fc3-bb0f-e5997585eb86\") " pod="openshift-network-operator/iptables-alerter-6cncl" Oct 30 08:53:22.066269 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.066251 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 08:53:22.066512 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.066497 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 08:53:22.066581 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.066499 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 08:53:22.066751 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.066728 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 08:53:22.066751 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.066739 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 08:53:22.066751 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.066743 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 08:53:22.066940 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.066791 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 08:53:22.066940 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.066835 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 08:53:22.066940 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.066842 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 08:53:22.066940 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.066864 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 08:53:22.066940 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.066884 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 08:53:22.067139 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.066947 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 08:53:22.067139 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.066974 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 08:53:22.067139 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.066990 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 08:53:22.067139 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.067021 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 08:53:22.067139 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.067057 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 08:53:22.067357 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.067152 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 08:53:22.067357 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.067165 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 08:53:22.067357 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.067178 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 08:53:22.067357 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.067178 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 08:53:22.067357 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.067204 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-twrgj\"" Oct 30 08:53:22.067357 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.067298 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 08:53:22.067357 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.067338 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-gbg4x\"" Oct 30 08:53:22.067357 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.067342 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-6v9sh\"" Oct 30 08:53:22.067623 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.067363 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-qhn6k\"" Oct 30 08:53:22.067623 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.067370 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pw6p6\"" Oct 30 08:53:22.067623 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.067391 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2jshw\"" Oct 30 08:53:22.067623 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.067403 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kd468\"" Oct 30 08:53:22.067623 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.067429 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-cnfjl\"" Oct 30 08:53:22.070535 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.070517 2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 08:53:22.070573 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.070549 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-r72ld" Oct 30 08:53:22.115969 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.115947 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2e880cd8-58f9-41e4-8795-4f5e978e18f5-etc-sysctl-conf\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:53:22.116041 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.115973 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-host-slash\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:53:22.116041 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.115991 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1f51db5a-5720-4dea-ad02-f336b35d43b1-ovnkube-script-lib\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:53:22.116041 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.116012 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfhrk\" (UniqueName: \"kubernetes.io/projected/0278c964-1ae2-42e7-a5b3-1df265e86edc-kube-api-access-rfhrk\") pod \"network-check-target-mz2kl\" (UID: \"0278c964-1ae2-42e7-a5b3-1df265e86edc\") " pod="openshift-network-diagnostics/network-check-target-mz2kl" Oct 30 08:53:22.116175 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.116047 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/82a05e91-5a9d-4670-9882-c12ee332ca04-device-dir\") pod \"aws-ebs-csi-driver-node-6r2gg\" (UID: \"82a05e91-5a9d-4670-9882-c12ee332ca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6r2gg" Oct 30 08:53:22.116175 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.116102 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/82a05e91-5a9d-4670-9882-c12ee332ca04-device-dir\") pod \"aws-ebs-csi-driver-node-6r2gg\" (UID: \"82a05e91-5a9d-4670-9882-c12ee332ca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6r2gg" Oct 30 08:53:22.116175 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.116099 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-host-slash\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:53:22.116175 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.116141 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/634749a3-dbc0-41f4-9e97-bf3d25f9b6a8-agent-certs\") pod \"konnectivity-agent-h5vr9\" (UID: \"634749a3-dbc0-41f4-9e97-bf3d25f9b6a8\") " pod="kube-system/konnectivity-agent-h5vr9" Oct 30 08:53:22.116175 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.116147 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2e880cd8-58f9-41e4-8795-4f5e978e18f5-etc-sysctl-conf\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:53:22.116175 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.116163 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmxcr\" (UniqueName: \"kubernetes.io/projected/9c67a6b6-a3d8-49bc-a678-b203ef4201a9-kube-api-access-rmxcr\") pod \"node-resolver-lq2p2\" (UID: \"9c67a6b6-a3d8-49bc-a678-b203ef4201a9\") " pod="openshift-dns/node-resolver-lq2p2" Oct 30 08:53:22.116487 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.116183 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-systemd-units\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:53:22.116487 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.116199 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-cnibin\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:53:22.116487 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.116218 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-host-var-lib-kubelet\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:53:22.116487 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.116250 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5d95\" (UniqueName: \"kubernetes.io/projected/3024e1a0-8ea6-4fc3-bb0f-e5997585eb86-kube-api-access-v5d95\") pod \"iptables-alerter-6cncl\" (UID: \"3024e1a0-8ea6-4fc3-bb0f-e5997585eb86\") " pod="openshift-network-operator/iptables-alerter-6cncl" Oct 30 08:53:22.116487 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.116262 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-systemd-units\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:53:22.116487 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.116275 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6f09a608-14be-447e-8e6e-eaf0a0031554-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v8qzv\" (UID: \"6f09a608-14be-447e-8e6e-eaf0a0031554\") " pod="openshift-multus/multus-additional-cni-plugins-v8qzv" Oct 30 08:53:22.116487 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.116300 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/82a05e91-5a9d-4670-9882-c12ee332ca04-socket-dir\") pod \"aws-ebs-csi-driver-node-6r2gg\" (UID: \"82a05e91-5a9d-4670-9882-c12ee332ca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6r2gg" Oct 30 08:53:22.116487 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.116311 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-cnibin\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:53:22.116487 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.116321 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2e880cd8-58f9-41e4-8795-4f5e978e18f5-host\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:53:22.116487 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.116318 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-host-var-lib-kubelet\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:53:22.116487 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.116366 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcnc5\" (UniqueName: \"kubernetes.io/projected/2e880cd8-58f9-41e4-8795-4f5e978e18f5-kube-api-access-vcnc5\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:53:22.116487 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.116414 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-node-log\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:53:22.116487 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.116439 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-host-cni-bin\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:53:22.116487 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.116455 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/82a05e91-5a9d-4670-9882-c12ee332ca04-socket-dir\") pod \"aws-ebs-csi-driver-node-6r2gg\" (UID: \"82a05e91-5a9d-4670-9882-c12ee332ca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6r2gg" Oct 30 08:53:22.116487 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.116479 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6f09a608-14be-447e-8e6e-eaf0a0031554-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v8qzv\" (UID: \"6f09a608-14be-447e-8e6e-eaf0a0031554\") " pod="openshift-multus/multus-additional-cni-plugins-v8qzv" Oct 30 08:53:22.116487 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.116463 2575 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 08:53:22.117208 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.116503 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-node-log\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:53:22.117208 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.116484 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:53:22.117208 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.116541 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6d1fb12b-1a27-4574-be55-140884eb868a-multus-daemon-config\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:53:22.117208 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.116546 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2e880cd8-58f9-41e4-8795-4f5e978e18f5-host\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:53:22.117208 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.116565 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:53:22.117208 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.116572 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-host-cni-bin\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:53:22.117208 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.116565 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6f09a608-14be-447e-8e6e-eaf0a0031554-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v8qzv\" (UID: \"6f09a608-14be-447e-8e6e-eaf0a0031554\") " pod="openshift-multus/multus-additional-cni-plugins-v8qzv" Oct 30 08:53:22.117208 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.116605 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-system-cni-dir\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:53:22.117208 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.116616 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1f51db5a-5720-4dea-ad02-f336b35d43b1-ovnkube-script-lib\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:53:22.117208 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.116639 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6d1fb12b-1a27-4574-be55-140884eb868a-cni-binary-copy\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:53:22.117208 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.116664 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9wwl\" (UniqueName: \"kubernetes.io/projected/6d1fb12b-1a27-4574-be55-140884eb868a-kube-api-access-m9wwl\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:53:22.117208 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.116710 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3024e1a0-8ea6-4fc3-bb0f-e5997585eb86-iptables-alerter-script\") pod \"iptables-alerter-6cncl\" (UID: \"3024e1a0-8ea6-4fc3-bb0f-e5997585eb86\") " pod="openshift-network-operator/iptables-alerter-6cncl" Oct 30 08:53:22.117208 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.116735 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8chm\" (UniqueName: \"kubernetes.io/projected/82a05e91-5a9d-4670-9882-c12ee332ca04-kube-api-access-s8chm\") pod \"aws-ebs-csi-driver-node-6r2gg\" (UID: \"82a05e91-5a9d-4670-9882-c12ee332ca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6r2gg" Oct 30 08:53:22.117208 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.116740 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-system-cni-dir\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:53:22.117208 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.116757 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c67a6b6-a3d8-49bc-a678-b203ef4201a9-tmp-dir\") pod \"node-resolver-lq2p2\" (UID: \"9c67a6b6-a3d8-49bc-a678-b203ef4201a9\") " pod="openshift-dns/node-resolver-lq2p2" Oct 30 08:53:22.117208 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.116778 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2e880cd8-58f9-41e4-8795-4f5e978e18f5-etc-sysconfig\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:53:22.117208 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.116803 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2e880cd8-58f9-41e4-8795-4f5e978e18f5-etc-kubernetes\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:53:22.118007 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.116826 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5e591466-fcf6-469d-9c3d-cde0767f4dde-kubelet-config\") pod \"global-pull-secret-syncer-vb26w\" (UID: \"5e591466-fcf6-469d-9c3d-cde0767f4dde\") " pod="kube-system/global-pull-secret-syncer-vb26w" Oct 30 08:53:22.118007 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.116849 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2e880cd8-58f9-41e4-8795-4f5e978e18f5-sys\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:53:22.118007 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.116874 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-host-var-lib-cni-multus\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:53:22.118007 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.116899 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5e591466-fcf6-469d-9c3d-cde0767f4dde-dbus\") pod \"global-pull-secret-syncer-vb26w\" (UID: \"5e591466-fcf6-469d-9c3d-cde0767f4dde\") " pod="kube-system/global-pull-secret-syncer-vb26w" Oct 30 08:53:22.118007 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.116922 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1f51db5a-5720-4dea-ad02-f336b35d43b1-ovnkube-config\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:53:22.118007 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.116950 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-host-run-multus-certs\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:53:22.118007 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.117010 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/82a05e91-5a9d-4670-9882-c12ee332ca04-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6r2gg\" (UID: \"82a05e91-5a9d-4670-9882-c12ee332ca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6r2gg" Oct 30 08:53:22.118007 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.117018 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2e880cd8-58f9-41e4-8795-4f5e978e18f5-sys\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:53:22.118007 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.117036 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2e880cd8-58f9-41e4-8795-4f5e978e18f5-lib-modules\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:53:22.118007 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.117062 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e880cd8-58f9-41e4-8795-4f5e978e18f5-var-lib-kubelet\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:53:22.118007 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.117072 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c67a6b6-a3d8-49bc-a678-b203ef4201a9-tmp-dir\") pod \"node-resolver-lq2p2\" (UID: \"9c67a6b6-a3d8-49bc-a678-b203ef4201a9\") " pod="openshift-dns/node-resolver-lq2p2" Oct 30 08:53:22.118007 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.117087 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-run-systemd\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:53:22.118007 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.117097 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2e880cd8-58f9-41e4-8795-4f5e978e18f5-etc-sysconfig\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:53:22.118007 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.117124 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-host-run-ovn-kubernetes\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:53:22.118007 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.117130 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-run-systemd\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:53:22.118007 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.117135 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6d1fb12b-1a27-4574-be55-140884eb868a-multus-daemon-config\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:53:22.118007 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.117153 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1f51db5a-5720-4dea-ad02-f336b35d43b1-env-overrides\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:53:22.118007 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.117165 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2e880cd8-58f9-41e4-8795-4f5e978e18f5-etc-kubernetes\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:53:22.118858 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.117175 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-host-var-lib-cni-multus\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:53:22.118858 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.117206 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5e591466-fcf6-469d-9c3d-cde0767f4dde-kubelet-config\") pod \"global-pull-secret-syncer-vb26w\" (UID: \"5e591466-fcf6-469d-9c3d-cde0767f4dde\") " pod="kube-system/global-pull-secret-syncer-vb26w" Oct 30 08:53:22.118858 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.117213 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3024e1a0-8ea6-4fc3-bb0f-e5997585eb86-iptables-alerter-script\") pod \"iptables-alerter-6cncl\" (UID: \"3024e1a0-8ea6-4fc3-bb0f-e5997585eb86\") " pod="openshift-network-operator/iptables-alerter-6cncl" Oct 30 08:53:22.118858 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.117235 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/82a05e91-5a9d-4670-9882-c12ee332ca04-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6r2gg\" (UID: \"82a05e91-5a9d-4670-9882-c12ee332ca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6r2gg" Oct 30 08:53:22.118858 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.117237 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-hostroot\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:53:22.118858 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.117269 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-hostroot\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:53:22.118858 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.117272 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e880cd8-58f9-41e4-8795-4f5e978e18f5-var-lib-kubelet\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:53:22.118858 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.117289 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6f09a608-14be-447e-8e6e-eaf0a0031554-cni-binary-copy\") pod \"multus-additional-cni-plugins-v8qzv\" (UID: \"6f09a608-14be-447e-8e6e-eaf0a0031554\") " pod="openshift-multus/multus-additional-cni-plugins-v8qzv" Oct 30 08:53:22.118858 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.117314 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-host-run-ovn-kubernetes\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:53:22.118858 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.117316 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2e880cd8-58f9-41e4-8795-4f5e978e18f5-lib-modules\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:53:22.118858 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.117316 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1b64ad2b-20eb-4a29-93b9-ccc9a217135d-host\") pod \"node-ca-xqjlj\" (UID: \"1b64ad2b-20eb-4a29-93b9-ccc9a217135d\") " pod="openshift-image-registry/node-ca-xqjlj" Oct 30 08:53:22.118858 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.117372 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-var-lib-openvswitch\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:53:22.118858 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.117410 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-run-ovn\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:53:22.118858 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.117436 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-multus-cni-dir\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:53:22.118858 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.117460 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-host-run-k8s-cni-cncf-io\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:53:22.118858 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.117464 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5e591466-fcf6-469d-9c3d-cde0767f4dde-dbus\") pod \"global-pull-secret-syncer-vb26w\" (UID: \"5e591466-fcf6-469d-9c3d-cde0767f4dde\") " pod="kube-system/global-pull-secret-syncer-vb26w" Oct 30 08:53:22.118858 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.117485 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/82a05e91-5a9d-4670-9882-c12ee332ca04-registration-dir\") pod \"aws-ebs-csi-driver-node-6r2gg\" (UID: \"82a05e91-5a9d-4670-9882-c12ee332ca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6r2gg" Oct 30 08:53:22.119730 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.117508 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/82a05e91-5a9d-4670-9882-c12ee332ca04-sys-fs\") pod \"aws-ebs-csi-driver-node-6r2gg\" (UID: \"82a05e91-5a9d-4670-9882-c12ee332ca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6r2gg" Oct 30 08:53:22.119730 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.117537 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8x4w\" (UniqueName: \"kubernetes.io/projected/4c94bc06-63d6-454f-9a56-1253c0315672-kube-api-access-x8x4w\") pod \"network-metrics-daemon-z66mv\" (UID: \"4c94bc06-63d6-454f-9a56-1253c0315672\") " pod="openshift-multus/network-metrics-daemon-z66mv" Oct 30 08:53:22.119730 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.117565 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-host-kubelet\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:53:22.119730 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.117589 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-host-run-netns\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:53:22.119730 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.117613 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3024e1a0-8ea6-4fc3-bb0f-e5997585eb86-host-slash\") pod \"iptables-alerter-6cncl\" (UID: \"3024e1a0-8ea6-4fc3-bb0f-e5997585eb86\") " pod="openshift-network-operator/iptables-alerter-6cncl" Oct 30 08:53:22.119730 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.117642 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtm2w\" (UniqueName: \"kubernetes.io/projected/1b64ad2b-20eb-4a29-93b9-ccc9a217135d-kube-api-access-mtm2w\") pod \"node-ca-xqjlj\" (UID: \"1b64ad2b-20eb-4a29-93b9-ccc9a217135d\") " pod="openshift-image-registry/node-ca-xqjlj" Oct 30 08:53:22.119730 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.117648 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1f51db5a-5720-4dea-ad02-f336b35d43b1-env-overrides\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:53:22.119730 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.117666 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c94bc06-63d6-454f-9a56-1253c0315672-metrics-certs\") pod \"network-metrics-daemon-z66mv\" (UID: \"4c94bc06-63d6-454f-9a56-1253c0315672\") " pod="openshift-multus/network-metrics-daemon-z66mv" Oct 30 08:53:22.119730 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.117688 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/82a05e91-5a9d-4670-9882-c12ee332ca04-registration-dir\") pod \"aws-ebs-csi-driver-node-6r2gg\" (UID: \"82a05e91-5a9d-4670-9882-c12ee332ca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6r2gg" Oct 30 08:53:22.119730 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.117702 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-host-kubelet\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:53:22.119730 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.117730 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2e880cd8-58f9-41e4-8795-4f5e978e18f5-etc-systemd\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:53:22.119730 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.117739 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-host-run-netns\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:53:22.119730 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.117742 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1b64ad2b-20eb-4a29-93b9-ccc9a217135d-host\") pod \"node-ca-xqjlj\" (UID: \"1b64ad2b-20eb-4a29-93b9-ccc9a217135d\") " pod="openshift-image-registry/node-ca-xqjlj" Oct 30 08:53:22.119730 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.117758 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5e591466-fcf6-469d-9c3d-cde0767f4dde-original-pull-secret\") pod \"global-pull-secret-syncer-vb26w\" (UID: \"5e591466-fcf6-469d-9c3d-cde0767f4dde\") " pod="kube-system/global-pull-secret-syncer-vb26w" Oct 30 08:53:22.119730 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.117780 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6f09a608-14be-447e-8e6e-eaf0a0031554-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v8qzv\" (UID: \"6f09a608-14be-447e-8e6e-eaf0a0031554\") " pod="openshift-multus/multus-additional-cni-plugins-v8qzv" Oct 30 08:53:22.119730 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.117787 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/634749a3-dbc0-41f4-9e97-bf3d25f9b6a8-konnectivity-ca\") pod \"konnectivity-agent-h5vr9\" (UID: \"634749a3-dbc0-41f4-9e97-bf3d25f9b6a8\") " pod="kube-system/konnectivity-agent-h5vr9" Oct 30 08:53:22.119730 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:22.117788 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 08:53:22.119730 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.117815 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-log-socket\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:53:22.120608 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.117839 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1f51db5a-5720-4dea-ad02-f336b35d43b1-ovn-node-metrics-cert\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:53:22.120608 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.117852 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2e880cd8-58f9-41e4-8795-4f5e978e18f5-etc-systemd\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:53:22.120608 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:22.117889 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c94bc06-63d6-454f-9a56-1253c0315672-metrics-certs podName:4c94bc06-63d6-454f-9a56-1253c0315672 nodeName:}" failed. No retries permitted until 2026-10-30 08:53:22.617837125 +0000 UTC m=+2.129808979 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4c94bc06-63d6-454f-9a56-1253c0315672-metrics-certs") pod "network-metrics-daemon-z66mv" (UID: "4c94bc06-63d6-454f-9a56-1253c0315672") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 08:53:22.120608 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.117912 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/82a05e91-5a9d-4670-9882-c12ee332ca04-sys-fs\") pod \"aws-ebs-csi-driver-node-6r2gg\" (UID: \"82a05e91-5a9d-4670-9882-c12ee332ca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6r2gg" Oct 30 08:53:22.120608 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.117920 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6f09a608-14be-447e-8e6e-eaf0a0031554-cni-binary-copy\") pod \"multus-additional-cni-plugins-v8qzv\" (UID: \"6f09a608-14be-447e-8e6e-eaf0a0031554\") " pod="openshift-multus/multus-additional-cni-plugins-v8qzv" Oct 30 08:53:22.120608 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.117927 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-var-lib-openvswitch\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:53:22.120608 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.117510 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-host-run-multus-certs\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:53:22.120608 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.117958 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1f51db5a-5720-4dea-ad02-f336b35d43b1-ovnkube-config\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:53:22.120608 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.117985 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-host-run-k8s-cni-cncf-io\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:53:22.120608 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.117984 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3024e1a0-8ea6-4fc3-bb0f-e5997585eb86-host-slash\") pod \"iptables-alerter-6cncl\" (UID: \"3024e1a0-8ea6-4fc3-bb0f-e5997585eb86\") " pod="openshift-network-operator/iptables-alerter-6cncl" Oct 30 08:53:22.120608 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.118007 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-multus-conf-dir\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:53:22.120608 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.118033 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6f09a608-14be-447e-8e6e-eaf0a0031554-cnibin\") pod \"multus-additional-cni-plugins-v8qzv\" (UID: \"6f09a608-14be-447e-8e6e-eaf0a0031554\") " pod="openshift-multus/multus-additional-cni-plugins-v8qzv" Oct 30 08:53:22.120608 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.118056 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2e880cd8-58f9-41e4-8795-4f5e978e18f5-run\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:53:22.120608 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.118064 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-multus-conf-dir\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:53:22.120608 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.118076 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-multus-cni-dir\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:53:22.120608 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.118081 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2e880cd8-58f9-41e4-8795-4f5e978e18f5-etc-tuned\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:53:22.120608 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.118034 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-run-ovn\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:53:22.121166 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.118111 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-etc-openvswitch\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:53:22.121166 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:22.118134 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 08:53:22.121166 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.118138 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5679q\" (UniqueName: \"kubernetes.io/projected/1f51db5a-5720-4dea-ad02-f336b35d43b1-kube-api-access-5679q\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:53:22.121166 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.118145 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6f09a608-14be-447e-8e6e-eaf0a0031554-cnibin\") pod \"multus-additional-cni-plugins-v8qzv\" (UID: \"6f09a608-14be-447e-8e6e-eaf0a0031554\") " pod="openshift-multus/multus-additional-cni-plugins-v8qzv" Oct 30 08:53:22.121166 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.118171 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-host-var-lib-cni-bin\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:53:22.121166 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:22.118201 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e591466-fcf6-469d-9c3d-cde0767f4dde-original-pull-secret podName:5e591466-fcf6-469d-9c3d-cde0767f4dde nodeName:}" failed. No retries permitted until 2026-10-30 08:53:22.618182716 +0000 UTC m=+2.130154564 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5e591466-fcf6-469d-9c3d-cde0767f4dde-original-pull-secret") pod "global-pull-secret-syncer-vb26w" (UID: "5e591466-fcf6-469d-9c3d-cde0767f4dde") : object "kube-system"/"original-pull-secret" not registered Oct 30 08:53:22.121166 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.118210 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-host-var-lib-cni-bin\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:53:22.121166 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.118230 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6f09a608-14be-447e-8e6e-eaf0a0031554-system-cni-dir\") pod \"multus-additional-cni-plugins-v8qzv\" (UID: \"6f09a608-14be-447e-8e6e-eaf0a0031554\") " pod="openshift-multus/multus-additional-cni-plugins-v8qzv" Oct 30 08:53:22.121166 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.118256 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6f09a608-14be-447e-8e6e-eaf0a0031554-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v8qzv\" (UID: \"6f09a608-14be-447e-8e6e-eaf0a0031554\") " pod="openshift-multus/multus-additional-cni-plugins-v8qzv" Oct 30 08:53:22.121166 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.118197 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-etc-openvswitch\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:53:22.121166 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.118282 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1b64ad2b-20eb-4a29-93b9-ccc9a217135d-serviceca\") pod \"node-ca-xqjlj\" (UID: \"1b64ad2b-20eb-4a29-93b9-ccc9a217135d\") " pod="openshift-image-registry/node-ca-xqjlj" Oct 30 08:53:22.121166 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.118304 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e880cd8-58f9-41e4-8795-4f5e978e18f5-tmp\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:53:22.121166 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.118323 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-run-openvswitch\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:53:22.121166 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.118343 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-host-cni-netd\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:53:22.121166 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.118363 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-multus-socket-dir-parent\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:53:22.121166 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.118403 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzmnn\" (UniqueName: \"kubernetes.io/projected/6f09a608-14be-447e-8e6e-eaf0a0031554-kube-api-access-nzmnn\") pod \"multus-additional-cni-plugins-v8qzv\" (UID: \"6f09a608-14be-447e-8e6e-eaf0a0031554\") " pod="openshift-multus/multus-additional-cni-plugins-v8qzv" Oct 30 08:53:22.121166 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.118435 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6f09a608-14be-447e-8e6e-eaf0a0031554-system-cni-dir\") pod \"multus-additional-cni-plugins-v8qzv\" (UID: \"6f09a608-14be-447e-8e6e-eaf0a0031554\") " pod="openshift-multus/multus-additional-cni-plugins-v8qzv" Oct 30 08:53:22.121687 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.118438 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9c67a6b6-a3d8-49bc-a678-b203ef4201a9-hosts-file\") pod \"node-resolver-lq2p2\" (UID: \"9c67a6b6-a3d8-49bc-a678-b203ef4201a9\") " pod="openshift-dns/node-resolver-lq2p2" Oct 30 08:53:22.121687 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.118479 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9c67a6b6-a3d8-49bc-a678-b203ef4201a9-hosts-file\") pod \"node-resolver-lq2p2\" (UID: \"9c67a6b6-a3d8-49bc-a678-b203ef4201a9\") " pod="openshift-dns/node-resolver-lq2p2" Oct 30 08:53:22.121687 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.118247 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-log-socket\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:53:22.121687 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.118478 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2e880cd8-58f9-41e4-8795-4f5e978e18f5-etc-sysctl-d\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:53:22.121687 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.118520 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-host-run-netns\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:53:22.121687 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.118548 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-etc-kubernetes\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:53:22.121687 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.118569 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/82a05e91-5a9d-4670-9882-c12ee332ca04-etc-selinux\") pod \"aws-ebs-csi-driver-node-6r2gg\" (UID: \"82a05e91-5a9d-4670-9882-c12ee332ca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6r2gg" Oct 30 08:53:22.121687 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.118591 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2e880cd8-58f9-41e4-8795-4f5e978e18f5-etc-modprobe-d\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:53:22.121687 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.118679 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/82a05e91-5a9d-4670-9882-c12ee332ca04-etc-selinux\") pod \"aws-ebs-csi-driver-node-6r2gg\" (UID: \"82a05e91-5a9d-4670-9882-c12ee332ca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6r2gg" Oct 30 08:53:22.121687 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.118693 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-run-openvswitch\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:53:22.121687 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.118742 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-host-cni-netd\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:53:22.121687 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.118780 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2e880cd8-58f9-41e4-8795-4f5e978e18f5-etc-modprobe-d\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:53:22.121687 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.118800 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-multus-socket-dir-parent\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:53:22.121687 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.118816 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-host-run-netns\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:53:22.121687 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.118853 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2e880cd8-58f9-41e4-8795-4f5e978e18f5-run\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:53:22.121687 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.118890 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-etc-kubernetes\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:53:22.121687 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.118896 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2e880cd8-58f9-41e4-8795-4f5e978e18f5-etc-sysctl-d\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:53:22.121687 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.119858 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6d1fb12b-1a27-4574-be55-140884eb868a-cni-binary-copy\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:53:22.122160 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.119872 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/634749a3-dbc0-41f4-9e97-bf3d25f9b6a8-konnectivity-ca\") pod \"konnectivity-agent-h5vr9\" (UID: \"634749a3-dbc0-41f4-9e97-bf3d25f9b6a8\") " pod="kube-system/konnectivity-agent-h5vr9" Oct 30 08:53:22.122160 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.119971 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1b64ad2b-20eb-4a29-93b9-ccc9a217135d-serviceca\") pod \"node-ca-xqjlj\" (UID: \"1b64ad2b-20eb-4a29-93b9-ccc9a217135d\") " pod="openshift-image-registry/node-ca-xqjlj" Oct 30 08:53:22.122160 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.120043 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6f09a608-14be-447e-8e6e-eaf0a0031554-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v8qzv\" (UID: \"6f09a608-14be-447e-8e6e-eaf0a0031554\") " pod="openshift-multus/multus-additional-cni-plugins-v8qzv" Oct 30 08:53:22.122160 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.120938 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2e880cd8-58f9-41e4-8795-4f5e978e18f5-etc-tuned\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:53:22.122160 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.121025 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1f51db5a-5720-4dea-ad02-f336b35d43b1-ovn-node-metrics-cert\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:53:22.122160 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.121034 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e880cd8-58f9-41e4-8795-4f5e978e18f5-tmp\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:53:22.122160 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.121149 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/634749a3-dbc0-41f4-9e97-bf3d25f9b6a8-agent-certs\") pod \"konnectivity-agent-h5vr9\" (UID: \"634749a3-dbc0-41f4-9e97-bf3d25f9b6a8\") " pod="kube-system/konnectivity-agent-h5vr9" Oct 30 08:53:22.134865 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:22.134849 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 08:53:22.134865 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:22.134868 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 08:53:22.134963 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:22.134877 2575 projected.go:196] Error preparing data for projected volume kube-api-access-rfhrk for pod openshift-network-diagnostics/network-check-target-mz2kl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 08:53:22.134963 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:22.134920 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0278c964-1ae2-42e7-a5b3-1df265e86edc-kube-api-access-rfhrk podName:0278c964-1ae2-42e7-a5b3-1df265e86edc nodeName:}" failed. No retries permitted until 2026-10-30 08:53:22.634908071 +0000 UTC m=+2.146879910 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-rfhrk" (UniqueName: "kubernetes.io/projected/0278c964-1ae2-42e7-a5b3-1df265e86edc-kube-api-access-rfhrk") pod "network-check-target-mz2kl" (UID: "0278c964-1ae2-42e7-a5b3-1df265e86edc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 08:53:22.136770 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.136746 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmxcr\" (UniqueName: \"kubernetes.io/projected/9c67a6b6-a3d8-49bc-a678-b203ef4201a9-kube-api-access-rmxcr\") pod \"node-resolver-lq2p2\" (UID: \"9c67a6b6-a3d8-49bc-a678-b203ef4201a9\") " pod="openshift-dns/node-resolver-lq2p2" Oct 30 08:53:22.137543 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.137522 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcnc5\" (UniqueName: \"kubernetes.io/projected/2e880cd8-58f9-41e4-8795-4f5e978e18f5-kube-api-access-vcnc5\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:53:22.137855 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.137833 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5d95\" (UniqueName: \"kubernetes.io/projected/3024e1a0-8ea6-4fc3-bb0f-e5997585eb86-kube-api-access-v5d95\") pod \"iptables-alerter-6cncl\" (UID: \"3024e1a0-8ea6-4fc3-bb0f-e5997585eb86\") " pod="openshift-network-operator/iptables-alerter-6cncl" Oct 30 08:53:22.138269 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.138254 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtm2w\" (UniqueName: \"kubernetes.io/projected/1b64ad2b-20eb-4a29-93b9-ccc9a217135d-kube-api-access-mtm2w\") pod \"node-ca-xqjlj\" (UID: \"1b64ad2b-20eb-4a29-93b9-ccc9a217135d\") " pod="openshift-image-registry/node-ca-xqjlj" Oct 30 08:53:22.139045 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.139025 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9wwl\" (UniqueName: \"kubernetes.io/projected/6d1fb12b-1a27-4574-be55-140884eb868a-kube-api-access-m9wwl\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:53:22.141019 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.141000 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8chm\" (UniqueName: \"kubernetes.io/projected/82a05e91-5a9d-4670-9882-c12ee332ca04-kube-api-access-s8chm\") pod \"aws-ebs-csi-driver-node-6r2gg\" (UID: \"82a05e91-5a9d-4670-9882-c12ee332ca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6r2gg" Oct 30 08:53:22.141152 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.141130 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8x4w\" (UniqueName: \"kubernetes.io/projected/4c94bc06-63d6-454f-9a56-1253c0315672-kube-api-access-x8x4w\") pod \"network-metrics-daemon-z66mv\" (UID: \"4c94bc06-63d6-454f-9a56-1253c0315672\") " pod="openshift-multus/network-metrics-daemon-z66mv" Oct 30 08:53:22.141203 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.141188 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5679q\" (UniqueName: \"kubernetes.io/projected/1f51db5a-5720-4dea-ad02-f336b35d43b1-kube-api-access-5679q\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:53:22.141592 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.141577 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzmnn\" (UniqueName: \"kubernetes.io/projected/6f09a608-14be-447e-8e6e-eaf0a0031554-kube-api-access-nzmnn\") pod \"multus-additional-cni-plugins-v8qzv\" (UID: \"6f09a608-14be-447e-8e6e-eaf0a0031554\") " pod="openshift-multus/multus-additional-cni-plugins-v8qzv" Oct 30 08:53:22.278543 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.278467 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-h5vr9" Oct 30 08:53:22.280627 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.280610 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rd9pd" Oct 30 08:53:22.288161 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.288143 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:53:22.288795 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.288773 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6cncl" Oct 30 08:53:22.289541 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:22.289518 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod634749a3_dbc0_41f4_9e97_bf3d25f9b6a8.slice/crio-4fdbbdeb1b6c4536eeb090fbdcd9727974b1d75f3492b1e0893d1d85c10d2841 WatchSource:0}: Error finding container 4fdbbdeb1b6c4536eeb090fbdcd9727974b1d75f3492b1e0893d1d85c10d2841: Status 404 returned error can't find the container with id 4fdbbdeb1b6c4536eeb090fbdcd9727974b1d75f3492b1e0893d1d85c10d2841 Oct 30 08:53:22.292992 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:22.292963 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d1fb12b_1a27_4574_be55_140884eb868a.slice/crio-de2b1f8738fd9844a42cd79ccf8775beed1e975e0eca66054f0003b5c421b56a WatchSource:0}: Error finding container de2b1f8738fd9844a42cd79ccf8775beed1e975e0eca66054f0003b5c421b56a: Status 404 returned error can't find the container with id de2b1f8738fd9844a42cd79ccf8775beed1e975e0eca66054f0003b5c421b56a Oct 30 08:53:22.299275 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:22.299251 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f51db5a_5720_4dea_ad02_f336b35d43b1.slice/crio-76fe9ba212bdc314a14ad3f84c0a7b8705dcd55e5c1c8f582dbe89fba52cb848 WatchSource:0}: Error finding container 76fe9ba212bdc314a14ad3f84c0a7b8705dcd55e5c1c8f582dbe89fba52cb848: Status 404 returned error can't find the container with id 76fe9ba212bdc314a14ad3f84c0a7b8705dcd55e5c1c8f582dbe89fba52cb848 Oct 30 08:53:22.301269 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:22.301248 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3024e1a0_8ea6_4fc3_bb0f_e5997585eb86.slice/crio-76206ead68e5c114f1d118341427645fd525e49d6c489cb683ff7175344afab2 WatchSource:0}: Error finding container 76206ead68e5c114f1d118341427645fd525e49d6c489cb683ff7175344afab2: Status 404 returned error can't find the container with id 76206ead68e5c114f1d118341427645fd525e49d6c489cb683ff7175344afab2 Oct 30 08:53:22.303141 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.303120 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lq2p2" Oct 30 08:53:22.306033 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.306013 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xqjlj" Oct 30 08:53:22.309948 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.309929 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-v8qzv" Oct 30 08:53:22.315942 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.315925 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6r2gg" Oct 30 08:53:22.316711 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:22.316691 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c67a6b6_a3d8_49bc_a678_b203ef4201a9.slice/crio-3d97d7db0511b7383090a21ad240f629c2cf23dfb9d0aab848d0cfb968f73b1c WatchSource:0}: Error finding container 3d97d7db0511b7383090a21ad240f629c2cf23dfb9d0aab848d0cfb968f73b1c: Status 404 returned error can't find the container with id 3d97d7db0511b7383090a21ad240f629c2cf23dfb9d0aab848d0cfb968f73b1c Oct 30 08:53:22.317948 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.317924 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:53:22.321779 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:22.321755 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b64ad2b_20eb_4a29_93b9_ccc9a217135d.slice/crio-98cb424b776de449e3a0f9c8a3b06b916346b73a4334818adaa121b5ca85e264 WatchSource:0}: Error finding container 98cb424b776de449e3a0f9c8a3b06b916346b73a4334818adaa121b5ca85e264: Status 404 returned error can't find the container with id 98cb424b776de449e3a0f9c8a3b06b916346b73a4334818adaa121b5ca85e264 Oct 30 08:53:22.322367 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:22.322344 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f09a608_14be_447e_8e6e_eaf0a0031554.slice/crio-6e0f69f96c4428a4db9835f68797d23a70e0b4ae1cb9e9ddd77552999fb86836 WatchSource:0}: Error finding container 6e0f69f96c4428a4db9835f68797d23a70e0b4ae1cb9e9ddd77552999fb86836: Status 404 returned error can't find the container with id 6e0f69f96c4428a4db9835f68797d23a70e0b4ae1cb9e9ddd77552999fb86836 Oct 30 08:53:22.327433 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:22.327358 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82a05e91_5a9d_4670_9882_c12ee332ca04.slice/crio-146d0980a6c4321fc991f6b14d8fc6c97e8ef968209481d013207827abdfae9b WatchSource:0}: Error finding container 146d0980a6c4321fc991f6b14d8fc6c97e8ef968209481d013207827abdfae9b: Status 404 returned error can't find the container with id 146d0980a6c4321fc991f6b14d8fc6c97e8ef968209481d013207827abdfae9b Oct 30 08:53:22.463733 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.463701 2575 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 08:53:22.621572 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.621493 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c94bc06-63d6-454f-9a56-1253c0315672-metrics-certs\") pod \"network-metrics-daemon-z66mv\" (UID: \"4c94bc06-63d6-454f-9a56-1253c0315672\") " pod="openshift-multus/network-metrics-daemon-z66mv" Oct 30 08:53:22.621572 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.621528 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5e591466-fcf6-469d-9c3d-cde0767f4dde-original-pull-secret\") pod \"global-pull-secret-syncer-vb26w\" (UID: \"5e591466-fcf6-469d-9c3d-cde0767f4dde\") " pod="kube-system/global-pull-secret-syncer-vb26w" Oct 30 08:53:22.621748 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:22.621675 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 08:53:22.621748 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:22.621739 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c94bc06-63d6-454f-9a56-1253c0315672-metrics-certs podName:4c94bc06-63d6-454f-9a56-1253c0315672 nodeName:}" failed. No retries permitted until 2026-10-30 08:53:23.621721797 +0000 UTC m=+3.133693654 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4c94bc06-63d6-454f-9a56-1253c0315672-metrics-certs") pod "network-metrics-daemon-z66mv" (UID: "4c94bc06-63d6-454f-9a56-1253c0315672") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 08:53:22.621844 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:22.621806 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 08:53:22.621903 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:22.621877 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e591466-fcf6-469d-9c3d-cde0767f4dde-original-pull-secret podName:5e591466-fcf6-469d-9c3d-cde0767f4dde nodeName:}" failed. No retries permitted until 2026-10-30 08:53:23.621861063 +0000 UTC m=+3.133832897 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5e591466-fcf6-469d-9c3d-cde0767f4dde-original-pull-secret") pod "global-pull-secret-syncer-vb26w" (UID: "5e591466-fcf6-469d-9c3d-cde0767f4dde") : object "kube-system"/"original-pull-secret" not registered Oct 30 08:53:22.723435 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:22.723006 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfhrk\" (UniqueName: \"kubernetes.io/projected/0278c964-1ae2-42e7-a5b3-1df265e86edc-kube-api-access-rfhrk\") pod \"network-check-target-mz2kl\" (UID: \"0278c964-1ae2-42e7-a5b3-1df265e86edc\") " pod="openshift-network-diagnostics/network-check-target-mz2kl" Oct 30 08:53:22.723435 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:22.723140 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 08:53:22.723435 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:22.723154 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 08:53:22.723435 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:22.723164 2575 projected.go:196] Error preparing data for projected volume kube-api-access-rfhrk for pod openshift-network-diagnostics/network-check-target-mz2kl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 08:53:22.723435 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:22.723208 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0278c964-1ae2-42e7-a5b3-1df265e86edc-kube-api-access-rfhrk podName:0278c964-1ae2-42e7-a5b3-1df265e86edc nodeName:}" failed. No retries permitted until 2026-10-30 08:53:23.723195033 +0000 UTC m=+3.235166881 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-rfhrk" (UniqueName: "kubernetes.io/projected/0278c964-1ae2-42e7-a5b3-1df265e86edc-kube-api-access-rfhrk") pod "network-check-target-mz2kl" (UID: "0278c964-1ae2-42e7-a5b3-1df265e86edc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 08:53:23.022845 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:23.022675 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" event={"ID":"2e880cd8-58f9-41e4-8795-4f5e978e18f5","Type":"ContainerStarted","Data":"aae15da4241b6222587872c4d1109d898b3716d59b6474a312a993eeaa641339"} Oct 30 08:53:23.027047 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:23.026960 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6r2gg" event={"ID":"82a05e91-5a9d-4670-9882-c12ee332ca04","Type":"ContainerStarted","Data":"146d0980a6c4321fc991f6b14d8fc6c97e8ef968209481d013207827abdfae9b"} Oct 30 08:53:23.034087 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:23.034026 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v8qzv" event={"ID":"6f09a608-14be-447e-8e6e-eaf0a0031554","Type":"ContainerStarted","Data":"6e0f69f96c4428a4db9835f68797d23a70e0b4ae1cb9e9ddd77552999fb86836"} Oct 30 08:53:23.036783 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:23.036706 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lq2p2" event={"ID":"9c67a6b6-a3d8-49bc-a678-b203ef4201a9","Type":"ContainerStarted","Data":"3d97d7db0511b7383090a21ad240f629c2cf23dfb9d0aab848d0cfb968f73b1c"} Oct 30 08:53:23.047420 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:23.047376 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" event={"ID":"1f51db5a-5720-4dea-ad02-f336b35d43b1","Type":"ContainerStarted","Data":"76fe9ba212bdc314a14ad3f84c0a7b8705dcd55e5c1c8f582dbe89fba52cb848"} Oct 30 08:53:23.053569 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:23.053463 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rd9pd" event={"ID":"6d1fb12b-1a27-4574-be55-140884eb868a","Type":"ContainerStarted","Data":"de2b1f8738fd9844a42cd79ccf8775beed1e975e0eca66054f0003b5c421b56a"} Oct 30 08:53:23.058892 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:23.058864 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xqjlj" event={"ID":"1b64ad2b-20eb-4a29-93b9-ccc9a217135d","Type":"ContainerStarted","Data":"98cb424b776de449e3a0f9c8a3b06b916346b73a4334818adaa121b5ca85e264"} Oct 30 08:53:23.067595 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:23.067514 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6cncl" event={"ID":"3024e1a0-8ea6-4fc3-bb0f-e5997585eb86","Type":"ContainerStarted","Data":"76206ead68e5c114f1d118341427645fd525e49d6c489cb683ff7175344afab2"} Oct 30 08:53:23.073714 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:23.073611 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 08:48:21 +0000 UTC" deadline="2028-06-12 09:34:56.598225242 +0000 UTC" Oct 30 08:53:23.073714 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:23.073670 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14184h41m33.524559092s" Oct 30 08:53:23.077888 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:23.077823 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-h5vr9" event={"ID":"634749a3-dbc0-41f4-9e97-bf3d25f9b6a8","Type":"ContainerStarted","Data":"4fdbbdeb1b6c4536eeb090fbdcd9727974b1d75f3492b1e0893d1d85c10d2841"} Oct 30 08:53:23.169866 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:23.169829 2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 08:53:23.379281 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:23.379210 2575 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 08:53:23.631872 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:23.631794 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c94bc06-63d6-454f-9a56-1253c0315672-metrics-certs\") pod \"network-metrics-daemon-z66mv\" (UID: \"4c94bc06-63d6-454f-9a56-1253c0315672\") " pod="openshift-multus/network-metrics-daemon-z66mv" Oct 30 08:53:23.631872 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:23.631840 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5e591466-fcf6-469d-9c3d-cde0767f4dde-original-pull-secret\") pod \"global-pull-secret-syncer-vb26w\" (UID: \"5e591466-fcf6-469d-9c3d-cde0767f4dde\") " pod="kube-system/global-pull-secret-syncer-vb26w" Oct 30 08:53:23.632089 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:23.632006 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 08:53:23.632089 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:23.632065 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e591466-fcf6-469d-9c3d-cde0767f4dde-original-pull-secret podName:5e591466-fcf6-469d-9c3d-cde0767f4dde nodeName:}" failed. No retries permitted until 2026-10-30 08:53:25.632046977 +0000 UTC m=+5.144018828 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5e591466-fcf6-469d-9c3d-cde0767f4dde-original-pull-secret") pod "global-pull-secret-syncer-vb26w" (UID: "5e591466-fcf6-469d-9c3d-cde0767f4dde") : object "kube-system"/"original-pull-secret" not registered Oct 30 08:53:23.632514 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:23.632495 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 08:53:23.632618 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:23.632558 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c94bc06-63d6-454f-9a56-1253c0315672-metrics-certs podName:4c94bc06-63d6-454f-9a56-1253c0315672 nodeName:}" failed. No retries permitted until 2026-10-30 08:53:25.632542627 +0000 UTC m=+5.144514477 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4c94bc06-63d6-454f-9a56-1253c0315672-metrics-certs") pod "network-metrics-daemon-z66mv" (UID: "4c94bc06-63d6-454f-9a56-1253c0315672") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 08:53:23.732900 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:23.732765 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfhrk\" (UniqueName: \"kubernetes.io/projected/0278c964-1ae2-42e7-a5b3-1df265e86edc-kube-api-access-rfhrk\") pod \"network-check-target-mz2kl\" (UID: \"0278c964-1ae2-42e7-a5b3-1df265e86edc\") " pod="openshift-network-diagnostics/network-check-target-mz2kl" Oct 30 08:53:23.733058 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:23.732922 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 08:53:23.733058 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:23.732939 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 08:53:23.733058 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:23.732952 2575 projected.go:196] Error preparing data for projected volume kube-api-access-rfhrk for pod openshift-network-diagnostics/network-check-target-mz2kl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 08:53:23.733058 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:23.733006 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0278c964-1ae2-42e7-a5b3-1df265e86edc-kube-api-access-rfhrk podName:0278c964-1ae2-42e7-a5b3-1df265e86edc nodeName:}" failed. No retries permitted until 2026-10-30 08:53:25.732988 +0000 UTC m=+5.244959853 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-rfhrk" (UniqueName: "kubernetes.io/projected/0278c964-1ae2-42e7-a5b3-1df265e86edc-kube-api-access-rfhrk") pod "network-check-target-mz2kl" (UID: "0278c964-1ae2-42e7-a5b3-1df265e86edc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 08:53:23.981052 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:23.980972 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z66mv" Oct 30 08:53:23.981211 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:23.981140 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z66mv" podUID="4c94bc06-63d6-454f-9a56-1253c0315672" Oct 30 08:53:23.981559 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:23.981538 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mz2kl" Oct 30 08:53:23.981653 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:23.981633 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mz2kl" podUID="0278c964-1ae2-42e7-a5b3-1df265e86edc" Oct 30 08:53:23.981710 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:23.981701 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vb26w" Oct 30 08:53:23.981798 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:23.981771 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vb26w" podUID="5e591466-fcf6-469d-9c3d-cde0767f4dde" Oct 30 08:53:24.074279 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:24.074219 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 08:48:21 +0000 UTC" deadline="2028-05-27 22:52:02.106418165 +0000 UTC" Oct 30 08:53:24.074279 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:24.074275 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13813h58m38.032146392s" Oct 30 08:53:25.002552 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:25.002514 2575 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 08:53:25.648118 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:25.647373 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c94bc06-63d6-454f-9a56-1253c0315672-metrics-certs\") pod \"network-metrics-daemon-z66mv\" (UID: \"4c94bc06-63d6-454f-9a56-1253c0315672\") " pod="openshift-multus/network-metrics-daemon-z66mv" Oct 30 08:53:25.648118 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:25.647443 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5e591466-fcf6-469d-9c3d-cde0767f4dde-original-pull-secret\") pod \"global-pull-secret-syncer-vb26w\" (UID: \"5e591466-fcf6-469d-9c3d-cde0767f4dde\") " pod="kube-system/global-pull-secret-syncer-vb26w" Oct 30 08:53:25.648118 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:25.647594 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 08:53:25.648118 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:25.647656 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e591466-fcf6-469d-9c3d-cde0767f4dde-original-pull-secret podName:5e591466-fcf6-469d-9c3d-cde0767f4dde nodeName:}" failed. No retries permitted until 2026-10-30 08:53:29.647639076 +0000 UTC m=+9.159610927 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5e591466-fcf6-469d-9c3d-cde0767f4dde-original-pull-secret") pod "global-pull-secret-syncer-vb26w" (UID: "5e591466-fcf6-469d-9c3d-cde0767f4dde") : object "kube-system"/"original-pull-secret" not registered Oct 30 08:53:25.648118 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:25.648029 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 08:53:25.648118 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:25.648074 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c94bc06-63d6-454f-9a56-1253c0315672-metrics-certs podName:4c94bc06-63d6-454f-9a56-1253c0315672 nodeName:}" failed. No retries permitted until 2026-10-30 08:53:29.648058571 +0000 UTC m=+9.160030465 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4c94bc06-63d6-454f-9a56-1253c0315672-metrics-certs") pod "network-metrics-daemon-z66mv" (UID: "4c94bc06-63d6-454f-9a56-1253c0315672") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 08:53:25.749156 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:25.748317 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfhrk\" (UniqueName: \"kubernetes.io/projected/0278c964-1ae2-42e7-a5b3-1df265e86edc-kube-api-access-rfhrk\") pod \"network-check-target-mz2kl\" (UID: \"0278c964-1ae2-42e7-a5b3-1df265e86edc\") " pod="openshift-network-diagnostics/network-check-target-mz2kl" Oct 30 08:53:25.749156 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:25.748646 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 08:53:25.749156 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:25.748797 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 08:53:25.749156 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:25.748809 2575 projected.go:196] Error preparing data for projected volume kube-api-access-rfhrk for pod openshift-network-diagnostics/network-check-target-mz2kl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 08:53:25.749156 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:25.748863 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0278c964-1ae2-42e7-a5b3-1df265e86edc-kube-api-access-rfhrk podName:0278c964-1ae2-42e7-a5b3-1df265e86edc nodeName:}" failed. No retries permitted until 2026-10-30 08:53:29.748844437 +0000 UTC m=+9.260816287 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-rfhrk" (UniqueName: "kubernetes.io/projected/0278c964-1ae2-42e7-a5b3-1df265e86edc-kube-api-access-rfhrk") pod "network-check-target-mz2kl" (UID: "0278c964-1ae2-42e7-a5b3-1df265e86edc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 08:53:25.980994 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:25.980173 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vb26w" Oct 30 08:53:25.980994 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:25.980302 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vb26w" podUID="5e591466-fcf6-469d-9c3d-cde0767f4dde" Oct 30 08:53:25.980994 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:25.980374 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z66mv" Oct 30 08:53:25.980994 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:25.980475 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z66mv" podUID="4c94bc06-63d6-454f-9a56-1253c0315672" Oct 30 08:53:25.980994 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:25.980520 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mz2kl" Oct 30 08:53:25.980994 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:25.980584 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mz2kl" podUID="0278c964-1ae2-42e7-a5b3-1df265e86edc" Oct 30 08:53:26.103374 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:26.102840 2575 generic.go:356] "Generic (PLEG): container finished" podID="4bdabdb48827c33749d04f26c951dd86" containerID="eba4e2e85a1762338bc652c746c629ea5babefcbf4a7fb78490ce9c8471eef14" exitCode=0 Oct 30 08:53:26.103374 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:26.102887 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-244.ec2.internal" event={"ID":"4bdabdb48827c33749d04f26c951dd86","Type":"ContainerDied","Data":"eba4e2e85a1762338bc652c746c629ea5babefcbf4a7fb78490ce9c8471eef14"} Oct 30 08:53:26.108174 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:26.108142 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-244.ec2.internal" event={"ID":"e62eebd98344a1611517e7c60f1efe61","Type":"ContainerStarted","Data":"bb7c92bde4da25da139e27136c707e4d71f36fc96fae42f1c3c7b14c08b93c50"} Oct 30 08:53:26.113141 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:26.113114 2575 generic.go:356] "Generic (PLEG): container finished" podID="6f09a608-14be-447e-8e6e-eaf0a0031554" containerID="d39760d59d325da5ae5b924fb1a3cfb60944b06ed908ff98091b141ecad3ec6e" exitCode=0 Oct 30 08:53:26.113226 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:26.113158 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v8qzv" event={"ID":"6f09a608-14be-447e-8e6e-eaf0a0031554","Type":"ContainerDied","Data":"d39760d59d325da5ae5b924fb1a3cfb60944b06ed908ff98091b141ecad3ec6e"} Oct 30 08:53:26.214753 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:26.214672 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-244.ec2.internal" podStartSLOduration=5.214652522 podStartE2EDuration="5.214652522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 08:53:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:53:26.174550834 +0000 UTC m=+5.686522691" watchObservedRunningTime="2026-10-30 08:53:26.214652522 +0000 UTC m=+5.726624381" Oct 30 08:53:27.122348 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:27.122194 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-h5vr9" event={"ID":"634749a3-dbc0-41f4-9e97-bf3d25f9b6a8","Type":"ContainerStarted","Data":"1db3abf1215290f459ff578fe8f20174742e2baafd9baf6420930d7c655c0be0"} Oct 30 08:53:27.140070 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:27.140039 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-244.ec2.internal" event={"ID":"4bdabdb48827c33749d04f26c951dd86","Type":"ContainerStarted","Data":"f855a0a99383931db9e4001ebaa42e179be5a5a4f8c67110609294b20b96a912"} Oct 30 08:53:27.194926 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:27.194869 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-h5vr9" podStartSLOduration=2.059325211 podStartE2EDuration="6.194849531s" totalImagesPullingTime="4.13552432s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:53:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:53:27.159223704 +0000 UTC m=+6.671195562" watchObservedRunningTime="2026-10-30 08:53:27.194849531 +0000 UTC m=+6.706821388" Oct 30 08:53:27.980598 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:27.980535 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mz2kl" Oct 30 08:53:27.980739 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:27.980624 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mz2kl" podUID="0278c964-1ae2-42e7-a5b3-1df265e86edc" Oct 30 08:53:27.980977 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:27.980960 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vb26w" Oct 30 08:53:27.981051 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:27.981035 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vb26w" podUID="5e591466-fcf6-469d-9c3d-cde0767f4dde" Oct 30 08:53:27.981109 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:27.981078 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z66mv" Oct 30 08:53:27.981158 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:27.981129 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z66mv" podUID="4c94bc06-63d6-454f-9a56-1253c0315672" Oct 30 08:53:28.147479 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:28.147442 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xqjlj" event={"ID":"1b64ad2b-20eb-4a29-93b9-ccc9a217135d","Type":"ContainerStarted","Data":"08e98fb84e7a8f538a6c24859e15c35e01d82e3a62266cf226a5b9c5c5af11bb"} Oct 30 08:53:28.183364 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:28.182733 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-xqjlj" podStartSLOduration=1.9081041810000001 podStartE2EDuration="7.182714121s" totalImagesPullingTime="5.27460994s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 08:53:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:53:28.182576682 +0000 UTC m=+7.694548541" watchObservedRunningTime="2026-10-30 08:53:28.182714121 +0000 UTC m=+7.694685981" Oct 30 08:53:28.183364 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:28.183198 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-244.ec2.internal" podStartSLOduration=7.183185571 podStartE2EDuration="7.183185571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 08:53:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:53:27.195581926 +0000 UTC m=+6.707553784" watchObservedRunningTime="2026-10-30 08:53:28.183185571 +0000 UTC m=+7.695157429" Oct 30 08:53:29.153352 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:29.153286 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6r2gg" event={"ID":"82a05e91-5a9d-4670-9882-c12ee332ca04","Type":"ContainerStarted","Data":"75e5ec5d569bb938785e10bb2f1aa74a213d1d1f68da7be20eeb717990dedeb6"} Oct 30 08:53:29.681869 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:29.681836 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c94bc06-63d6-454f-9a56-1253c0315672-metrics-certs\") pod \"network-metrics-daemon-z66mv\" (UID: \"4c94bc06-63d6-454f-9a56-1253c0315672\") " pod="openshift-multus/network-metrics-daemon-z66mv" Oct 30 08:53:29.681971 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:29.681885 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5e591466-fcf6-469d-9c3d-cde0767f4dde-original-pull-secret\") pod \"global-pull-secret-syncer-vb26w\" (UID: \"5e591466-fcf6-469d-9c3d-cde0767f4dde\") " pod="kube-system/global-pull-secret-syncer-vb26w" Oct 30 08:53:29.682040 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:29.682022 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 08:53:29.682099 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:29.682083 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e591466-fcf6-469d-9c3d-cde0767f4dde-original-pull-secret podName:5e591466-fcf6-469d-9c3d-cde0767f4dde nodeName:}" failed. No retries permitted until 2026-10-30 08:53:37.682065013 +0000 UTC m=+17.194036870 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5e591466-fcf6-469d-9c3d-cde0767f4dde-original-pull-secret") pod "global-pull-secret-syncer-vb26w" (UID: "5e591466-fcf6-469d-9c3d-cde0767f4dde") : object "kube-system"/"original-pull-secret" not registered Oct 30 08:53:29.682165 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:29.682154 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 08:53:29.682219 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:29.682184 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c94bc06-63d6-454f-9a56-1253c0315672-metrics-certs podName:4c94bc06-63d6-454f-9a56-1253c0315672 nodeName:}" failed. No retries permitted until 2026-10-30 08:53:37.68217634 +0000 UTC m=+17.194148175 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4c94bc06-63d6-454f-9a56-1253c0315672-metrics-certs") pod "network-metrics-daemon-z66mv" (UID: "4c94bc06-63d6-454f-9a56-1253c0315672") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 08:53:29.700265 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:29.700237 2575 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 08:53:29.782539 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:29.782475 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfhrk\" (UniqueName: \"kubernetes.io/projected/0278c964-1ae2-42e7-a5b3-1df265e86edc-kube-api-access-rfhrk\") pod \"network-check-target-mz2kl\" (UID: \"0278c964-1ae2-42e7-a5b3-1df265e86edc\") " pod="openshift-network-diagnostics/network-check-target-mz2kl" Oct 30 08:53:29.782717 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:29.782610 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 08:53:29.782717 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:29.782626 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 08:53:29.782717 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:29.782635 2575 projected.go:196] Error preparing data for projected volume kube-api-access-rfhrk for pod openshift-network-diagnostics/network-check-target-mz2kl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 08:53:29.782717 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:29.782683 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0278c964-1ae2-42e7-a5b3-1df265e86edc-kube-api-access-rfhrk podName:0278c964-1ae2-42e7-a5b3-1df265e86edc nodeName:}" failed. No retries permitted until 2026-10-30 08:53:37.782667203 +0000 UTC m=+17.294639067 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-rfhrk" (UniqueName: "kubernetes.io/projected/0278c964-1ae2-42e7-a5b3-1df265e86edc-kube-api-access-rfhrk") pod "network-check-target-mz2kl" (UID: "0278c964-1ae2-42e7-a5b3-1df265e86edc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 08:53:29.980653 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:29.980620 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mz2kl" Oct 30 08:53:29.980820 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:29.980755 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mz2kl" podUID="0278c964-1ae2-42e7-a5b3-1df265e86edc" Oct 30 08:53:29.980820 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:29.980724 2575 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T08:53:29.700255625Z","UUID":"106e5a2c-94cb-433b-9e45-4c52b2cd2bf1","Handler":null,"Name":"","Endpoint":""} Oct 30 08:53:29.981596 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:29.981520 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vb26w" Oct 30 08:53:29.981877 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:29.981824 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vb26w" podUID="5e591466-fcf6-469d-9c3d-cde0767f4dde" Oct 30 08:53:29.981980 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:29.981947 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z66mv" Oct 30 08:53:29.982609 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:29.982586 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z66mv" podUID="4c94bc06-63d6-454f-9a56-1253c0315672" Oct 30 08:53:29.986001 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:29.985982 2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 08:53:29.986083 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:29.986026 2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 08:53:30.157291 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:30.157243 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6r2gg" event={"ID":"82a05e91-5a9d-4670-9882-c12ee332ca04","Type":"ContainerStarted","Data":"6a7c2c91e3ecf4a72e6a90e8f836f8e3906b236bc4d53a0fc0456ddeaae3c16b"} Oct 30 08:53:30.159756 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:30.159726 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rd9pd" event={"ID":"6d1fb12b-1a27-4574-be55-140884eb868a","Type":"ContainerStarted","Data":"e3b96ee56b03dbeeb4c23eaa784df4d049cdab204b523ed291541bd0b42b5965"} Oct 30 08:53:30.197909 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:30.197840 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-rd9pd" podStartSLOduration=1.4315811379999999 podStartE2EDuration="9.197823511s" totalImagesPullingTime="7.766242373s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 08:53:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:53:30.197471739 +0000 UTC m=+9.709443653" watchObservedRunningTime="2026-10-30 08:53:30.197823511 +0000 UTC m=+9.709795483" Oct 30 08:53:30.369997 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:30.369920 2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-h5vr9" Oct 30 08:53:30.370542 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:30.370520 2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-h5vr9" Oct 30 08:53:31.164884 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:31.164800 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6r2gg" event={"ID":"82a05e91-5a9d-4670-9882-c12ee332ca04","Type":"ContainerStarted","Data":"c1d762993f222d4e190810298b9f316900a1070ae825a3f4c3a7efc65ac9a19f"} Oct 30 08:53:31.167767 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:31.167739 2575 generic.go:356] "Generic (PLEG): container finished" podID="6f09a608-14be-447e-8e6e-eaf0a0031554" containerID="1baa210418f0f6cf9ada0551c4e4448cc8d5811f3233d77f6434937d9c1013e9" exitCode=0 Oct 30 08:53:31.167873 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:31.167805 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v8qzv" event={"ID":"6f09a608-14be-447e-8e6e-eaf0a0031554","Type":"ContainerDied","Data":"1baa210418f0f6cf9ada0551c4e4448cc8d5811f3233d77f6434937d9c1013e9"} Oct 30 08:53:31.170714 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:31.170683 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lq2p2" event={"ID":"9c67a6b6-a3d8-49bc-a678-b203ef4201a9","Type":"ContainerStarted","Data":"16a259c7a7165c16773d157ea5541a683bd804b9e6fbdb3d0973cdea693307ce"} Oct 30 08:53:31.212838 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:31.212590 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6r2gg" podStartSLOduration=2.136574088 podStartE2EDuration="10.212574599s" totalImagesPullingTime="8.076000511s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 08:53:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:53:31.203526449 +0000 UTC m=+10.715498310" watchObservedRunningTime="2026-10-30 08:53:31.212574599 +0000 UTC m=+10.724546458" Oct 30 08:53:31.285554 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:31.285501 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-lq2p2" podStartSLOduration=1.957692313 podStartE2EDuration="10.285485437s" totalImagesPullingTime="8.327793124s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 08:53:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:53:31.284889354 +0000 UTC m=+10.796861214" watchObservedRunningTime="2026-10-30 08:53:31.285485437 +0000 UTC m=+10.797457297" Oct 30 08:53:31.981124 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:31.981089 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mz2kl" Oct 30 08:53:31.981302 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:31.981127 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vb26w" Oct 30 08:53:31.981302 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:31.981218 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mz2kl" podUID="0278c964-1ae2-42e7-a5b3-1df265e86edc" Oct 30 08:53:31.981302 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:31.981270 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vb26w" podUID="5e591466-fcf6-469d-9c3d-cde0767f4dde" Oct 30 08:53:31.981302 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:31.981290 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z66mv" Oct 30 08:53:31.981536 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:31.981374 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z66mv" podUID="4c94bc06-63d6-454f-9a56-1253c0315672" Oct 30 08:53:32.175101 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:32.175022 2575 generic.go:356] "Generic (PLEG): container finished" podID="6f09a608-14be-447e-8e6e-eaf0a0031554" containerID="c7eed2a753a92df048bef1cd2fdaa2c55d74d6a0b919fbaa4d5c6b801b0b04ab" exitCode=0 Oct 30 08:53:32.175493 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:32.175127 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v8qzv" event={"ID":"6f09a608-14be-447e-8e6e-eaf0a0031554","Type":"ContainerDied","Data":"c7eed2a753a92df048bef1cd2fdaa2c55d74d6a0b919fbaa4d5c6b801b0b04ab"} Oct 30 08:53:32.177730 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:32.177678 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6cncl" event={"ID":"3024e1a0-8ea6-4fc3-bb0f-e5997585eb86","Type":"ContainerStarted","Data":"6c5b6772488069425596531ac65678e71e8f902f251132736aded62cfea18aeb"} Oct 30 08:53:33.181632 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:33.181586 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" event={"ID":"2e880cd8-58f9-41e4-8795-4f5e978e18f5","Type":"ContainerStarted","Data":"2e05113cf1fd2067ddda17944c683af4ba8a30dbb3f026bcba5c95f6bef991f1"} Oct 30 08:53:33.184747 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:33.184716 2575 generic.go:356] "Generic (PLEG): container finished" podID="6f09a608-14be-447e-8e6e-eaf0a0031554" containerID="90af3f9863d2e545aa8499e9a5902b3c59a667f6449935585f24e6e9e83f1321" exitCode=0 Oct 30 08:53:33.184876 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:33.184809 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v8qzv" event={"ID":"6f09a608-14be-447e-8e6e-eaf0a0031554","Type":"ContainerDied","Data":"90af3f9863d2e545aa8499e9a5902b3c59a667f6449935585f24e6e9e83f1321"} Oct 30 08:53:33.218901 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:33.218848 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-6cncl" podStartSLOduration=4.134433092 podStartE2EDuration="12.218831529s" totalImagesPullingTime="8.084398437s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 08:53:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:53:32.252796785 +0000 UTC m=+11.764768646" watchObservedRunningTime="2026-10-30 08:53:33.218831529 +0000 UTC m=+12.730803387" Oct 30 08:53:33.219217 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:33.219179 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" podStartSLOduration=2.24856576 podStartE2EDuration="12.219172274s" totalImagesPullingTime="9.970606514s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 08:53:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:53:33.218699372 +0000 UTC m=+12.730671229" watchObservedRunningTime="2026-10-30 08:53:33.219172274 +0000 UTC m=+12.731144130" Oct 30 08:53:33.981375 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:33.980862 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z66mv" Oct 30 08:53:33.981375 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:33.980898 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mz2kl" Oct 30 08:53:33.981375 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:33.981018 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z66mv" podUID="4c94bc06-63d6-454f-9a56-1253c0315672" Oct 30 08:53:33.981375 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:33.981089 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mz2kl" podUID="0278c964-1ae2-42e7-a5b3-1df265e86edc" Oct 30 08:53:33.981375 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:33.981162 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vb26w" Oct 30 08:53:33.981375 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:33.981262 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vb26w" podUID="5e591466-fcf6-469d-9c3d-cde0767f4dde" Oct 30 08:53:35.192495 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:35.192455 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v8qzv" event={"ID":"6f09a608-14be-447e-8e6e-eaf0a0031554","Type":"ContainerStarted","Data":"91d841116d556c6ec4d9b3341cfd1f7675ce2b2da1d53365a8357011128fca52"} Oct 30 08:53:35.231165 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:35.231139 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-h5vr9" Oct 30 08:53:35.231287 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:35.231262 2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 08:53:35.232129 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:35.232110 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-h5vr9" Oct 30 08:53:35.980312 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:35.980103 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mz2kl" Oct 30 08:53:35.980499 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:35.980165 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vb26w" Oct 30 08:53:35.980499 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:35.980417 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mz2kl" podUID="0278c964-1ae2-42e7-a5b3-1df265e86edc" Oct 30 08:53:35.980499 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:35.980484 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vb26w" podUID="5e591466-fcf6-469d-9c3d-cde0767f4dde" Oct 30 08:53:35.980646 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:35.980173 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z66mv" Oct 30 08:53:35.980646 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:35.980584 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z66mv" podUID="4c94bc06-63d6-454f-9a56-1253c0315672" Oct 30 08:53:36.197802 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:36.197764 2575 generic.go:356] "Generic (PLEG): container finished" podID="6f09a608-14be-447e-8e6e-eaf0a0031554" containerID="91d841116d556c6ec4d9b3341cfd1f7675ce2b2da1d53365a8357011128fca52" exitCode=0 Oct 30 08:53:36.198189 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:36.197837 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v8qzv" event={"ID":"6f09a608-14be-447e-8e6e-eaf0a0031554","Type":"ContainerDied","Data":"91d841116d556c6ec4d9b3341cfd1f7675ce2b2da1d53365a8357011128fca52"} Oct 30 08:53:37.203301 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:37.203141 2575 generic.go:356] "Generic (PLEG): container finished" podID="6f09a608-14be-447e-8e6e-eaf0a0031554" containerID="edb7694d185427d98a0674cdd2fcac444f6c9025917bb049e0014effea8b5b2c" exitCode=0 Oct 30 08:53:37.203761 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:37.203223 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v8qzv" event={"ID":"6f09a608-14be-447e-8e6e-eaf0a0031554","Type":"ContainerDied","Data":"edb7694d185427d98a0674cdd2fcac444f6c9025917bb049e0014effea8b5b2c"} Oct 30 08:53:37.206201 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:37.206184 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6q7n_1f51db5a-5720-4dea-ad02-f336b35d43b1/ovn-acl-logging/0.log" Oct 30 08:53:37.206561 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:37.206535 2575 generic.go:356] "Generic (PLEG): container finished" podID="1f51db5a-5720-4dea-ad02-f336b35d43b1" containerID="3cb37b1a8e3d1e357fede29d8044d2c67903697d6ce9fc3859603c9669368854" exitCode=1 Oct 30 08:53:37.206629 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:37.206588 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" event={"ID":"1f51db5a-5720-4dea-ad02-f336b35d43b1","Type":"ContainerStarted","Data":"185d01c2d1f1e676b59ed4972a6cc29c9a9529b158a284041ae7192f0a4bdd78"} Oct 30 08:53:37.206800 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:37.206655 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" event={"ID":"1f51db5a-5720-4dea-ad02-f336b35d43b1","Type":"ContainerStarted","Data":"81864f51a2241aa3645f861bced282aa1b9c670c57d423201f11d4ba34bfaed0"} Oct 30 08:53:37.206800 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:37.206701 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" event={"ID":"1f51db5a-5720-4dea-ad02-f336b35d43b1","Type":"ContainerStarted","Data":"85950fe83a9aa85303ea768b87efff2f7c12542933a82d919119a4d16a5dcf4f"} Oct 30 08:53:37.206800 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:37.206746 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" event={"ID":"1f51db5a-5720-4dea-ad02-f336b35d43b1","Type":"ContainerStarted","Data":"27711943de4bfc350a208eeaf11538a736d71a09edc516cd0fdfd05251bd5ff5"} Oct 30 08:53:37.206800 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:37.206791 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" event={"ID":"1f51db5a-5720-4dea-ad02-f336b35d43b1","Type":"ContainerDied","Data":"3cb37b1a8e3d1e357fede29d8044d2c67903697d6ce9fc3859603c9669368854"} Oct 30 08:53:37.206930 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:37.206846 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" event={"ID":"1f51db5a-5720-4dea-ad02-f336b35d43b1","Type":"ContainerStarted","Data":"6d5bff2b3df4e48d7763b2d8ba85f7b06ad8ccd6516bc412691060bc4790e44f"} Oct 30 08:53:37.752461 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:37.752432 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c94bc06-63d6-454f-9a56-1253c0315672-metrics-certs\") pod \"network-metrics-daemon-z66mv\" (UID: \"4c94bc06-63d6-454f-9a56-1253c0315672\") " pod="openshift-multus/network-metrics-daemon-z66mv" Oct 30 08:53:37.752461 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:37.752462 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5e591466-fcf6-469d-9c3d-cde0767f4dde-original-pull-secret\") pod \"global-pull-secret-syncer-vb26w\" (UID: \"5e591466-fcf6-469d-9c3d-cde0767f4dde\") " pod="kube-system/global-pull-secret-syncer-vb26w" Oct 30 08:53:37.752645 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:37.752570 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 08:53:37.752645 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:37.752571 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 08:53:37.752645 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:37.752620 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c94bc06-63d6-454f-9a56-1253c0315672-metrics-certs podName:4c94bc06-63d6-454f-9a56-1253c0315672 nodeName:}" failed. No retries permitted until 2026-10-30 08:53:53.752604456 +0000 UTC m=+33.264576291 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4c94bc06-63d6-454f-9a56-1253c0315672-metrics-certs") pod "network-metrics-daemon-z66mv" (UID: "4c94bc06-63d6-454f-9a56-1253c0315672") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 08:53:37.752645 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:37.752632 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e591466-fcf6-469d-9c3d-cde0767f4dde-original-pull-secret podName:5e591466-fcf6-469d-9c3d-cde0767f4dde nodeName:}" failed. No retries permitted until 2026-10-30 08:53:53.752626406 +0000 UTC m=+33.264598244 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5e591466-fcf6-469d-9c3d-cde0767f4dde-original-pull-secret") pod "global-pull-secret-syncer-vb26w" (UID: "5e591466-fcf6-469d-9c3d-cde0767f4dde") : object "kube-system"/"original-pull-secret" not registered Oct 30 08:53:37.853492 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:37.853466 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfhrk\" (UniqueName: \"kubernetes.io/projected/0278c964-1ae2-42e7-a5b3-1df265e86edc-kube-api-access-rfhrk\") pod \"network-check-target-mz2kl\" (UID: \"0278c964-1ae2-42e7-a5b3-1df265e86edc\") " pod="openshift-network-diagnostics/network-check-target-mz2kl" Oct 30 08:53:37.853630 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:37.853614 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 08:53:37.853664 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:37.853635 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 08:53:37.853664 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:37.853644 2575 projected.go:196] Error preparing data for projected volume kube-api-access-rfhrk for pod openshift-network-diagnostics/network-check-target-mz2kl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 08:53:37.853731 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:37.853687 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0278c964-1ae2-42e7-a5b3-1df265e86edc-kube-api-access-rfhrk podName:0278c964-1ae2-42e7-a5b3-1df265e86edc nodeName:}" failed. No retries permitted until 2026-10-30 08:53:53.853675925 +0000 UTC m=+33.365647789 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-rfhrk" (UniqueName: "kubernetes.io/projected/0278c964-1ae2-42e7-a5b3-1df265e86edc-kube-api-access-rfhrk") pod "network-check-target-mz2kl" (UID: "0278c964-1ae2-42e7-a5b3-1df265e86edc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 08:53:37.980403 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:37.980355 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mz2kl" Oct 30 08:53:37.980521 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:37.980413 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vb26w" Oct 30 08:53:37.980521 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:37.980497 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z66mv" Oct 30 08:53:37.980521 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:37.980498 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mz2kl" podUID="0278c964-1ae2-42e7-a5b3-1df265e86edc" Oct 30 08:53:37.980620 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:37.980569 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vb26w" podUID="5e591466-fcf6-469d-9c3d-cde0767f4dde" Oct 30 08:53:37.980651 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:37.980623 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z66mv" podUID="4c94bc06-63d6-454f-9a56-1253c0315672" Oct 30 08:53:38.211473 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:38.211442 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v8qzv" event={"ID":"6f09a608-14be-447e-8e6e-eaf0a0031554","Type":"ContainerStarted","Data":"9a1a4e807a55d7fee83f254f5bca7152bbccfea3aca29e90f8c28ff435d69129"} Oct 30 08:53:38.251880 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:38.251839 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-v8qzv" podStartSLOduration=7.51563856 podStartE2EDuration="17.251827746s" totalImagesPullingTime="9.736189186s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 08:53:21 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:53:38.2513705 +0000 UTC m=+17.763342348" watchObservedRunningTime="2026-10-30 08:53:38.251827746 +0000 UTC m=+17.763799603" Oct 30 08:53:39.981001 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:39.980971 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mz2kl" Oct 30 08:53:39.981520 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:39.981085 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vb26w" Oct 30 08:53:39.981520 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:39.981093 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mz2kl" podUID="0278c964-1ae2-42e7-a5b3-1df265e86edc" Oct 30 08:53:39.981520 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:39.981164 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vb26w" podUID="5e591466-fcf6-469d-9c3d-cde0767f4dde" Oct 30 08:53:39.981520 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:39.981175 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z66mv" Oct 30 08:53:39.981520 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:39.981244 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z66mv" podUID="4c94bc06-63d6-454f-9a56-1253c0315672" Oct 30 08:53:40.217390 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:40.217352 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6q7n_1f51db5a-5720-4dea-ad02-f336b35d43b1/ovn-acl-logging/0.log" Oct 30 08:53:40.217720 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:40.217700 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" event={"ID":"1f51db5a-5720-4dea-ad02-f336b35d43b1","Type":"ContainerStarted","Data":"20c709c32aa1f2706a46483ee60e1bd4eab1f7bb4e1c8a9185386bff563b7545"} Oct 30 08:53:41.981087 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:41.981060 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mz2kl" Oct 30 08:53:41.981708 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:41.981154 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mz2kl" podUID="0278c964-1ae2-42e7-a5b3-1df265e86edc" Oct 30 08:53:41.981708 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:41.981166 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vb26w" Oct 30 08:53:41.981708 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:41.981243 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vb26w" podUID="5e591466-fcf6-469d-9c3d-cde0767f4dde" Oct 30 08:53:41.981708 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:41.981157 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z66mv" Oct 30 08:53:41.981708 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:41.981323 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z66mv" podUID="4c94bc06-63d6-454f-9a56-1253c0315672" Oct 30 08:53:42.224292 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:42.224268 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6q7n_1f51db5a-5720-4dea-ad02-f336b35d43b1/ovn-acl-logging/0.log" Oct 30 08:53:42.224660 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:42.224631 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" event={"ID":"1f51db5a-5720-4dea-ad02-f336b35d43b1","Type":"ContainerStarted","Data":"44c1c4180e85329e318038a2fcf986575dfd6b2f15b405eab287e87f1831a9b5"} Oct 30 08:53:42.224999 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:42.224984 2575 scope.go:122] "RemoveContainer" containerID="3cb37b1a8e3d1e357fede29d8044d2c67903697d6ce9fc3859603c9669368854" Oct 30 08:53:42.225081 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:42.224988 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:53:42.225185 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:42.225144 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:53:42.241941 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:42.241793 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:53:43.230573 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:43.230548 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6q7n_1f51db5a-5720-4dea-ad02-f336b35d43b1/ovn-acl-logging/0.log" Oct 30 08:53:43.231058 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:43.231009 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" event={"ID":"1f51db5a-5720-4dea-ad02-f336b35d43b1","Type":"ContainerStarted","Data":"be497a1d2bbeef9ca055efd2cc510e249af4e2a52bdb27b698e0af4f06af5f5f"} Oct 30 08:53:43.231426 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:43.231406 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:53:43.248968 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:43.248896 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:53:43.272170 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:43.272149 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vb26w"] Oct 30 08:53:43.272282 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:43.272245 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vb26w" Oct 30 08:53:43.272335 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:43.272319 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vb26w" podUID="5e591466-fcf6-469d-9c3d-cde0767f4dde" Oct 30 08:53:43.275141 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:43.275118 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mz2kl"] Oct 30 08:53:43.275272 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:43.275224 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mz2kl" Oct 30 08:53:43.275336 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:43.275314 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mz2kl" podUID="0278c964-1ae2-42e7-a5b3-1df265e86edc" Oct 30 08:53:43.275953 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:43.275924 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-z66mv"] Oct 30 08:53:43.276051 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:43.276037 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z66mv" Oct 30 08:53:43.276159 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:43.276141 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z66mv" podUID="4c94bc06-63d6-454f-9a56-1253c0315672" Oct 30 08:53:43.288055 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:43.288009 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" podStartSLOduration=7.8548875339999995 podStartE2EDuration="22.28799911s" totalImagesPullingTime="14.433111576s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 08:53:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:53:43.287363677 +0000 UTC m=+22.799335534" watchObservedRunningTime="2026-10-30 08:53:43.28799911 +0000 UTC m=+22.799970966" Oct 30 08:53:44.980419 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:44.980376 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z66mv" Oct 30 08:53:44.980419 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:44.980416 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vb26w" Oct 30 08:53:44.980929 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:44.980505 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z66mv" podUID="4c94bc06-63d6-454f-9a56-1253c0315672" Oct 30 08:53:44.980929 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:44.980561 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vb26w" podUID="5e591466-fcf6-469d-9c3d-cde0767f4dde" Oct 30 08:53:44.980929 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:44.980628 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mz2kl" Oct 30 08:53:44.980929 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:44.980724 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mz2kl" podUID="0278c964-1ae2-42e7-a5b3-1df265e86edc" Oct 30 08:53:46.980527 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:46.980495 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vb26w" Oct 30 08:53:46.980527 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:46.980507 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z66mv" Oct 30 08:53:46.981033 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:46.980507 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mz2kl" Oct 30 08:53:46.981033 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:46.980636 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vb26w" podUID="5e591466-fcf6-469d-9c3d-cde0767f4dde" Oct 30 08:53:46.981033 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:46.980748 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z66mv" podUID="4c94bc06-63d6-454f-9a56-1253c0315672" Oct 30 08:53:46.981033 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:46.980818 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mz2kl" podUID="0278c964-1ae2-42e7-a5b3-1df265e86edc" Oct 30 08:53:48.801054 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:48.801025 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.ec2.internal" event="NodeReady" Oct 30 08:53:48.801543 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:48.801154 2575 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 08:53:48.852816 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:48.852793 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-lmjss"] Oct 30 08:53:48.855267 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:48.855248 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lmjss" Oct 30 08:53:48.855979 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:48.855959 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-95kld"] Oct 30 08:53:48.858626 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:48.858609 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-95kld" Oct 30 08:53:48.869594 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:48.869575 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 08:53:48.869755 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:48.869740 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 08:53:48.869800 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:48.869740 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 08:53:48.869835 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:48.869800 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-cb5mw\"" Oct 30 08:53:48.875284 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:48.875263 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lmjss"] Oct 30 08:53:48.878500 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:48.878480 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-95kld"] Oct 30 08:53:48.936777 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:48.936752 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87k9j\" (UniqueName: \"kubernetes.io/projected/6edea6bc-4807-48ff-b879-a77fc07d92de-kube-api-access-87k9j\") pod \"dns-default-95kld\" (UID: \"6edea6bc-4807-48ff-b879-a77fc07d92de\") " pod="openshift-dns/dns-default-95kld" Oct 30 08:53:48.936866 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:48.936793 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhzh5\" (UniqueName: \"kubernetes.io/projected/deaa55d7-ac05-4aa1-909a-4237addf17ea-kube-api-access-nhzh5\") pod \"ingress-canary-lmjss\" (UID: \"deaa55d7-ac05-4aa1-909a-4237addf17ea\") " pod="openshift-ingress-canary/ingress-canary-lmjss" Oct 30 08:53:48.936866 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:48.936815 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6edea6bc-4807-48ff-b879-a77fc07d92de-tmp-dir\") pod \"dns-default-95kld\" (UID: \"6edea6bc-4807-48ff-b879-a77fc07d92de\") " pod="openshift-dns/dns-default-95kld" Oct 30 08:53:48.936948 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:48.936924 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6edea6bc-4807-48ff-b879-a77fc07d92de-config-volume\") pod \"dns-default-95kld\" (UID: \"6edea6bc-4807-48ff-b879-a77fc07d92de\") " pod="openshift-dns/dns-default-95kld" Oct 30 08:53:48.936980 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:48.936957 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/deaa55d7-ac05-4aa1-909a-4237addf17ea-cert\") pod \"ingress-canary-lmjss\" (UID: \"deaa55d7-ac05-4aa1-909a-4237addf17ea\") " pod="openshift-ingress-canary/ingress-canary-lmjss" Oct 30 08:53:48.936980 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:48.936974 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6edea6bc-4807-48ff-b879-a77fc07d92de-metrics-tls\") pod \"dns-default-95kld\" (UID: \"6edea6bc-4807-48ff-b879-a77fc07d92de\") " pod="openshift-dns/dns-default-95kld" Oct 30 08:53:48.970187 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:48.970166 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 08:53:48.970253 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:48.970178 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 08:53:48.970303 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:48.970288 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xqvv4\"" Oct 30 08:53:48.980551 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:48.980534 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z66mv" Oct 30 08:53:48.980589 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:48.980559 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vb26w" Oct 30 08:53:48.980756 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:48.980737 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mz2kl" Oct 30 08:53:48.995146 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:48.995126 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-r2tvf\"" Oct 30 08:53:48.995146 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:48.995141 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 08:53:48.995304 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:48.995152 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 08:53:48.995304 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:48.995157 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 08:53:48.995304 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:48.995200 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-kgscj\"" Oct 30 08:53:48.995304 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:48.995221 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-7pnl7\"" Oct 30 08:53:48.995304 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:48.995288 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 08:53:49.037797 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:49.037777 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6edea6bc-4807-48ff-b879-a77fc07d92de-config-volume\") pod \"dns-default-95kld\" (UID: \"6edea6bc-4807-48ff-b879-a77fc07d92de\") " pod="openshift-dns/dns-default-95kld" Oct 30 08:53:49.037882 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:49.037817 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/deaa55d7-ac05-4aa1-909a-4237addf17ea-cert\") pod \"ingress-canary-lmjss\" (UID: \"deaa55d7-ac05-4aa1-909a-4237addf17ea\") " pod="openshift-ingress-canary/ingress-canary-lmjss" Oct 30 08:53:49.037882 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:49.037835 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6edea6bc-4807-48ff-b879-a77fc07d92de-metrics-tls\") pod \"dns-default-95kld\" (UID: \"6edea6bc-4807-48ff-b879-a77fc07d92de\") " pod="openshift-dns/dns-default-95kld" Oct 30 08:53:49.038000 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:49.037907 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 08:53:49.038000 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:49.037949 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6edea6bc-4807-48ff-b879-a77fc07d92de-metrics-tls podName:6edea6bc-4807-48ff-b879-a77fc07d92de nodeName:}" failed. No retries permitted until 2026-10-30 08:53:49.537932875 +0000 UTC m=+29.049904724 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6edea6bc-4807-48ff-b879-a77fc07d92de-metrics-tls") pod "dns-default-95kld" (UID: "6edea6bc-4807-48ff-b879-a77fc07d92de") : secret "dns-default-metrics-tls" not found Oct 30 08:53:49.038000 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:49.037963 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87k9j\" (UniqueName: \"kubernetes.io/projected/6edea6bc-4807-48ff-b879-a77fc07d92de-kube-api-access-87k9j\") pod \"dns-default-95kld\" (UID: \"6edea6bc-4807-48ff-b879-a77fc07d92de\") " pod="openshift-dns/dns-default-95kld" Oct 30 08:53:49.038000 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:49.037994 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhzh5\" (UniqueName: \"kubernetes.io/projected/deaa55d7-ac05-4aa1-909a-4237addf17ea-kube-api-access-nhzh5\") pod \"ingress-canary-lmjss\" (UID: \"deaa55d7-ac05-4aa1-909a-4237addf17ea\") " pod="openshift-ingress-canary/ingress-canary-lmjss" Oct 30 08:53:49.038186 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:49.038039 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6edea6bc-4807-48ff-b879-a77fc07d92de-tmp-dir\") pod \"dns-default-95kld\" (UID: \"6edea6bc-4807-48ff-b879-a77fc07d92de\") " pod="openshift-dns/dns-default-95kld" Oct 30 08:53:49.038286 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:49.038268 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6edea6bc-4807-48ff-b879-a77fc07d92de-tmp-dir\") pod \"dns-default-95kld\" (UID: \"6edea6bc-4807-48ff-b879-a77fc07d92de\") " pod="openshift-dns/dns-default-95kld" Oct 30 08:53:49.038401 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:49.038368 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6edea6bc-4807-48ff-b879-a77fc07d92de-config-volume\") pod \"dns-default-95kld\" (UID: \"6edea6bc-4807-48ff-b879-a77fc07d92de\") " pod="openshift-dns/dns-default-95kld" Oct 30 08:53:49.042150 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:49.042036 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/deaa55d7-ac05-4aa1-909a-4237addf17ea-cert\") pod \"ingress-canary-lmjss\" (UID: \"deaa55d7-ac05-4aa1-909a-4237addf17ea\") " pod="openshift-ingress-canary/ingress-canary-lmjss" Oct 30 08:53:49.059420 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:49.059349 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87k9j\" (UniqueName: \"kubernetes.io/projected/6edea6bc-4807-48ff-b879-a77fc07d92de-kube-api-access-87k9j\") pod \"dns-default-95kld\" (UID: \"6edea6bc-4807-48ff-b879-a77fc07d92de\") " pod="openshift-dns/dns-default-95kld" Oct 30 08:53:49.059636 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:49.059620 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhzh5\" (UniqueName: \"kubernetes.io/projected/deaa55d7-ac05-4aa1-909a-4237addf17ea-kube-api-access-nhzh5\") pod \"ingress-canary-lmjss\" (UID: \"deaa55d7-ac05-4aa1-909a-4237addf17ea\") " pod="openshift-ingress-canary/ingress-canary-lmjss" Oct 30 08:53:49.167768 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:49.167736 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lmjss" Oct 30 08:53:49.463811 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:49.463576 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lmjss"] Oct 30 08:53:49.466312 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:49.466284 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeaa55d7_ac05_4aa1_909a_4237addf17ea.slice/crio-69fc2caf22a115675e3e0996034e0918c88e4902429d6e6f262f434487cdfd63 WatchSource:0}: Error finding container 69fc2caf22a115675e3e0996034e0918c88e4902429d6e6f262f434487cdfd63: Status 404 returned error can't find the container with id 69fc2caf22a115675e3e0996034e0918c88e4902429d6e6f262f434487cdfd63 Oct 30 08:53:49.542079 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:49.542058 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6edea6bc-4807-48ff-b879-a77fc07d92de-metrics-tls\") pod \"dns-default-95kld\" (UID: \"6edea6bc-4807-48ff-b879-a77fc07d92de\") " pod="openshift-dns/dns-default-95kld" Oct 30 08:53:49.542183 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:49.542171 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 08:53:49.542228 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:49.542217 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6edea6bc-4807-48ff-b879-a77fc07d92de-metrics-tls podName:6edea6bc-4807-48ff-b879-a77fc07d92de nodeName:}" failed. No retries permitted until 2026-10-30 08:53:50.542203788 +0000 UTC m=+30.054175627 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6edea6bc-4807-48ff-b879-a77fc07d92de-metrics-tls") pod "dns-default-95kld" (UID: "6edea6bc-4807-48ff-b879-a77fc07d92de") : secret "dns-default-metrics-tls" not found Oct 30 08:53:50.247978 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:50.247938 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lmjss" event={"ID":"deaa55d7-ac05-4aa1-909a-4237addf17ea","Type":"ContainerStarted","Data":"69fc2caf22a115675e3e0996034e0918c88e4902429d6e6f262f434487cdfd63"} Oct 30 08:53:50.548814 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:50.548736 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6edea6bc-4807-48ff-b879-a77fc07d92de-metrics-tls\") pod \"dns-default-95kld\" (UID: \"6edea6bc-4807-48ff-b879-a77fc07d92de\") " pod="openshift-dns/dns-default-95kld" Oct 30 08:53:50.548965 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:50.548865 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 08:53:50.548965 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:50.548927 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6edea6bc-4807-48ff-b879-a77fc07d92de-metrics-tls podName:6edea6bc-4807-48ff-b879-a77fc07d92de nodeName:}" failed. No retries permitted until 2026-10-30 08:53:52.548914461 +0000 UTC m=+32.060886299 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6edea6bc-4807-48ff-b879-a77fc07d92de-metrics-tls") pod "dns-default-95kld" (UID: "6edea6bc-4807-48ff-b879-a77fc07d92de") : secret "dns-default-metrics-tls" not found Oct 30 08:53:50.901717 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:50.901638 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-lc8jz"] Oct 30 08:53:50.959663 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:50.959626 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lc8jz"] Oct 30 08:53:50.959810 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:50.959752 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lc8jz" Oct 30 08:53:50.977035 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:50.977015 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-z7wg9-fvpzr"] Oct 30 08:53:50.986224 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:50.986201 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-z7wg9-fvpzr" Oct 30 08:53:50.995579 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:50.995555 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-z7wg9-fvpzr"] Oct 30 08:53:51.022357 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:51.022328 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 08:53:51.022546 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:51.022461 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 08:53:51.022546 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:51.022543 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-nq2rg\"" Oct 30 08:53:51.022667 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:51.022601 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 08:53:51.022667 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:51.022660 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 08:53:51.022815 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:51.022789 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wqhvd\"" Oct 30 08:53:51.022948 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:51.022931 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 08:53:51.038213 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:51.038172 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 08:53:51.053490 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:51.053458 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b79e9a37-068d-4387-a07b-84b094e6895a-crio-socket\") pod \"insights-runtime-extractor-lc8jz\" (UID: \"b79e9a37-068d-4387-a07b-84b094e6895a\") " pod="openshift-insights/insights-runtime-extractor-lc8jz" Oct 30 08:53:51.053613 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:51.053535 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2-service-ca-bundle\") pod \"periodic-gathering-z7wg9-fvpzr\" (UID: \"6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2\") " pod="openshift-insights/periodic-gathering-z7wg9-fvpzr" Oct 30 08:53:51.053664 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:51.053633 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b79e9a37-068d-4387-a07b-84b094e6895a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lc8jz\" (UID: \"b79e9a37-068d-4387-a07b-84b094e6895a\") " pod="openshift-insights/insights-runtime-extractor-lc8jz" Oct 30 08:53:51.053664 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:51.053656 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpl7f\" (UniqueName: \"kubernetes.io/projected/b79e9a37-068d-4387-a07b-84b094e6895a-kube-api-access-bpl7f\") pod \"insights-runtime-extractor-lc8jz\" (UID: \"b79e9a37-068d-4387-a07b-84b094e6895a\") " pod="openshift-insights/insights-runtime-extractor-lc8jz" Oct 30 08:53:51.053732 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:51.053690 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2-trusted-ca-bundle\") pod \"periodic-gathering-z7wg9-fvpzr\" (UID: \"6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2\") " pod="openshift-insights/periodic-gathering-z7wg9-fvpzr" Oct 30 08:53:51.053782 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:51.053768 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2-archives-path\") pod \"periodic-gathering-z7wg9-fvpzr\" (UID: \"6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2\") " pod="openshift-insights/periodic-gathering-z7wg9-fvpzr" Oct 30 08:53:51.053838 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:51.053802 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb9vs\" (UniqueName: \"kubernetes.io/projected/6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2-kube-api-access-tb9vs\") pod \"periodic-gathering-z7wg9-fvpzr\" (UID: \"6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2\") " pod="openshift-insights/periodic-gathering-z7wg9-fvpzr" Oct 30 08:53:51.053953 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:51.053930 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b79e9a37-068d-4387-a07b-84b094e6895a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lc8jz\" (UID: \"b79e9a37-068d-4387-a07b-84b094e6895a\") " pod="openshift-insights/insights-runtime-extractor-lc8jz" Oct 30 08:53:51.054022 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:51.053982 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b79e9a37-068d-4387-a07b-84b094e6895a-data-volume\") pod \"insights-runtime-extractor-lc8jz\" (UID: \"b79e9a37-068d-4387-a07b-84b094e6895a\") " pod="openshift-insights/insights-runtime-extractor-lc8jz" Oct 30 08:53:51.155281 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:51.155206 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2-archives-path\") pod \"periodic-gathering-z7wg9-fvpzr\" (UID: \"6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2\") " pod="openshift-insights/periodic-gathering-z7wg9-fvpzr" Oct 30 08:53:51.155281 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:51.155248 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb9vs\" (UniqueName: \"kubernetes.io/projected/6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2-kube-api-access-tb9vs\") pod \"periodic-gathering-z7wg9-fvpzr\" (UID: \"6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2\") " pod="openshift-insights/periodic-gathering-z7wg9-fvpzr" Oct 30 08:53:51.155522 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:51.155301 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b79e9a37-068d-4387-a07b-84b094e6895a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lc8jz\" (UID: \"b79e9a37-068d-4387-a07b-84b094e6895a\") " pod="openshift-insights/insights-runtime-extractor-lc8jz" Oct 30 08:53:51.155522 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:51.155339 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b79e9a37-068d-4387-a07b-84b094e6895a-data-volume\") pod \"insights-runtime-extractor-lc8jz\" (UID: \"b79e9a37-068d-4387-a07b-84b094e6895a\") " pod="openshift-insights/insights-runtime-extractor-lc8jz" Oct 30 08:53:51.155522 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:51.155399 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b79e9a37-068d-4387-a07b-84b094e6895a-crio-socket\") pod \"insights-runtime-extractor-lc8jz\" (UID: \"b79e9a37-068d-4387-a07b-84b094e6895a\") " pod="openshift-insights/insights-runtime-extractor-lc8jz" Oct 30 08:53:51.155522 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:51.155453 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2-service-ca-bundle\") pod \"periodic-gathering-z7wg9-fvpzr\" (UID: \"6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2\") " pod="openshift-insights/periodic-gathering-z7wg9-fvpzr" Oct 30 08:53:51.155522 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:51.155479 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b79e9a37-068d-4387-a07b-84b094e6895a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lc8jz\" (UID: \"b79e9a37-068d-4387-a07b-84b094e6895a\") " pod="openshift-insights/insights-runtime-extractor-lc8jz" Oct 30 08:53:51.155522 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:51.155503 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpl7f\" (UniqueName: \"kubernetes.io/projected/b79e9a37-068d-4387-a07b-84b094e6895a-kube-api-access-bpl7f\") pod \"insights-runtime-extractor-lc8jz\" (UID: \"b79e9a37-068d-4387-a07b-84b094e6895a\") " pod="openshift-insights/insights-runtime-extractor-lc8jz" Oct 30 08:53:51.155809 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:51.155538 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2-trusted-ca-bundle\") pod \"periodic-gathering-z7wg9-fvpzr\" (UID: \"6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2\") " pod="openshift-insights/periodic-gathering-z7wg9-fvpzr" Oct 30 08:53:51.155809 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:51.155658 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 08:53:51.155809 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:51.155737 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 08:53:51.155809 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:51.155737 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b79e9a37-068d-4387-a07b-84b094e6895a-crio-socket\") pod \"insights-runtime-extractor-lc8jz\" (UID: \"b79e9a37-068d-4387-a07b-84b094e6895a\") " pod="openshift-insights/insights-runtime-extractor-lc8jz" Oct 30 08:53:51.155809 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:51.155740 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b79e9a37-068d-4387-a07b-84b094e6895a-kube-rbac-proxy-cm podName:b79e9a37-068d-4387-a07b-84b094e6895a nodeName:}" failed. No retries permitted until 2026-10-30 08:53:51.655718941 +0000 UTC m=+31.167690789 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b79e9a37-068d-4387-a07b-84b094e6895a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lc8jz" (UID: "b79e9a37-068d-4387-a07b-84b094e6895a") : configmap "kube-rbac-proxy" not found Oct 30 08:53:51.156051 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:51.155822 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b79e9a37-068d-4387-a07b-84b094e6895a-insights-runtime-extractor-tls podName:b79e9a37-068d-4387-a07b-84b094e6895a nodeName:}" failed. No retries permitted until 2026-10-30 08:53:51.655806394 +0000 UTC m=+31.167778444 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b79e9a37-068d-4387-a07b-84b094e6895a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lc8jz" (UID: "b79e9a37-068d-4387-a07b-84b094e6895a") : secret "insights-runtime-extractor-tls" not found Oct 30 08:53:51.156051 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:51.155660 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2-archives-path\") pod \"periodic-gathering-z7wg9-fvpzr\" (UID: \"6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2\") " pod="openshift-insights/periodic-gathering-z7wg9-fvpzr" Oct 30 08:53:51.156051 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:51.156048 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b79e9a37-068d-4387-a07b-84b094e6895a-data-volume\") pod \"insights-runtime-extractor-lc8jz\" (UID: \"b79e9a37-068d-4387-a07b-84b094e6895a\") " pod="openshift-insights/insights-runtime-extractor-lc8jz" Oct 30 08:53:51.156242 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:51.156218 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2-service-ca-bundle\") pod \"periodic-gathering-z7wg9-fvpzr\" (UID: \"6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2\") " pod="openshift-insights/periodic-gathering-z7wg9-fvpzr" Oct 30 08:53:51.156425 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:51.156405 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2-trusted-ca-bundle\") pod \"periodic-gathering-z7wg9-fvpzr\" (UID: \"6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2\") " pod="openshift-insights/periodic-gathering-z7wg9-fvpzr" Oct 30 08:53:51.175353 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:51.175329 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpl7f\" (UniqueName: \"kubernetes.io/projected/b79e9a37-068d-4387-a07b-84b094e6895a-kube-api-access-bpl7f\") pod \"insights-runtime-extractor-lc8jz\" (UID: \"b79e9a37-068d-4387-a07b-84b094e6895a\") " pod="openshift-insights/insights-runtime-extractor-lc8jz" Oct 30 08:53:51.175554 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:51.175537 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb9vs\" (UniqueName: \"kubernetes.io/projected/6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2-kube-api-access-tb9vs\") pod \"periodic-gathering-z7wg9-fvpzr\" (UID: \"6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2\") " pod="openshift-insights/periodic-gathering-z7wg9-fvpzr" Oct 30 08:53:51.328559 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:51.328533 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-z7wg9-fvpzr" Oct 30 08:53:51.582421 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:51.582371 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-nf6sx"] Oct 30 08:53:51.603201 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:51.603173 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-nlv7n"] Oct 30 08:53:51.603357 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:51.603266 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-nf6sx" Oct 30 08:53:51.617204 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:51.617150 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-nf6sx"] Oct 30 08:53:51.617316 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:51.617257 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-nlv7n"] Oct 30 08:53:51.617359 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:51.617340 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nlv7n" Oct 30 08:53:51.621041 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:51.621013 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 08:53:51.621175 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:51.621155 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-5px9g\"" Oct 30 08:53:51.626980 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:51.626963 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 08:53:51.626980 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:51.626965 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 08:53:51.627110 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:51.627008 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 08:53:51.646707 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:51.646684 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 08:53:51.646707 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:51.646697 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 08:53:51.660418 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:51.660393 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b79e9a37-068d-4387-a07b-84b094e6895a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lc8jz\" (UID: \"b79e9a37-068d-4387-a07b-84b094e6895a\") " pod="openshift-insights/insights-runtime-extractor-lc8jz" Oct 30 08:53:51.660512 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:51.660448 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b79e9a37-068d-4387-a07b-84b094e6895a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lc8jz\" (UID: \"b79e9a37-068d-4387-a07b-84b094e6895a\") " pod="openshift-insights/insights-runtime-extractor-lc8jz" Oct 30 08:53:51.660573 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:51.660526 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 08:53:51.660573 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:51.660553 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 08:53:51.660648 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:51.660594 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b79e9a37-068d-4387-a07b-84b094e6895a-kube-rbac-proxy-cm podName:b79e9a37-068d-4387-a07b-84b094e6895a nodeName:}" failed. No retries permitted until 2026-10-30 08:53:52.660579141 +0000 UTC m=+32.172550996 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b79e9a37-068d-4387-a07b-84b094e6895a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lc8jz" (UID: "b79e9a37-068d-4387-a07b-84b094e6895a") : configmap "kube-rbac-proxy" not found Oct 30 08:53:51.660648 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:51.660608 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b79e9a37-068d-4387-a07b-84b094e6895a-insights-runtime-extractor-tls podName:b79e9a37-068d-4387-a07b-84b094e6895a nodeName:}" failed. No retries permitted until 2026-10-30 08:53:52.660602652 +0000 UTC m=+32.172574490 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b79e9a37-068d-4387-a07b-84b094e6895a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lc8jz" (UID: "b79e9a37-068d-4387-a07b-84b094e6895a") : secret "insights-runtime-extractor-tls" not found Oct 30 08:53:51.672134 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:51.671920 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-z7wg9-fvpzr"] Oct 30 08:53:51.675047 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:51.675020 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6296ad27_59fd_4726_8ca9_0ccdbf0fb4e2.slice/crio-50d630405f1e6ff1b6bbb8e451da2ab8317b82f08e08cfce7f1fdf5cd0ea7728 WatchSource:0}: Error finding container 50d630405f1e6ff1b6bbb8e451da2ab8317b82f08e08cfce7f1fdf5cd0ea7728: Status 404 returned error can't find the container with id 50d630405f1e6ff1b6bbb8e451da2ab8317b82f08e08cfce7f1fdf5cd0ea7728 Oct 30 08:53:51.721237 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:51.721202 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-h5xkm\"" Oct 30 08:53:51.761745 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:51.761719 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/bab893eb-ddd1-41b0-80a7-3c4907b7a8ad-signing-key\") pod \"service-ca-5fffc97bdf-nf6sx\" (UID: \"bab893eb-ddd1-41b0-80a7-3c4907b7a8ad\") " pod="openshift-service-ca/service-ca-5fffc97bdf-nf6sx" Oct 30 08:53:51.761859 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:51.761749 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/bab893eb-ddd1-41b0-80a7-3c4907b7a8ad-signing-cabundle\") pod \"service-ca-5fffc97bdf-nf6sx\" (UID: \"bab893eb-ddd1-41b0-80a7-3c4907b7a8ad\") " pod="openshift-service-ca/service-ca-5fffc97bdf-nf6sx" Oct 30 08:53:51.761859 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:51.761770 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnszd\" (UniqueName: \"kubernetes.io/projected/54ef0e46-200b-4e7d-a007-ad959dec4486-kube-api-access-vnszd\") pod \"migrator-66df5d496b-nlv7n\" (UID: \"54ef0e46-200b-4e7d-a007-ad959dec4486\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nlv7n" Oct 30 08:53:51.761938 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:51.761889 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dhlm\" (UniqueName: \"kubernetes.io/projected/bab893eb-ddd1-41b0-80a7-3c4907b7a8ad-kube-api-access-6dhlm\") pod \"service-ca-5fffc97bdf-nf6sx\" (UID: \"bab893eb-ddd1-41b0-80a7-3c4907b7a8ad\") " pod="openshift-service-ca/service-ca-5fffc97bdf-nf6sx" Oct 30 08:53:51.862363 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:51.862331 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dhlm\" (UniqueName: \"kubernetes.io/projected/bab893eb-ddd1-41b0-80a7-3c4907b7a8ad-kube-api-access-6dhlm\") pod \"service-ca-5fffc97bdf-nf6sx\" (UID: \"bab893eb-ddd1-41b0-80a7-3c4907b7a8ad\") " pod="openshift-service-ca/service-ca-5fffc97bdf-nf6sx" Oct 30 08:53:51.862504 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:51.862431 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/bab893eb-ddd1-41b0-80a7-3c4907b7a8ad-signing-key\") pod \"service-ca-5fffc97bdf-nf6sx\" (UID: \"bab893eb-ddd1-41b0-80a7-3c4907b7a8ad\") " pod="openshift-service-ca/service-ca-5fffc97bdf-nf6sx" Oct 30 08:53:51.862504 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:51.862448 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/bab893eb-ddd1-41b0-80a7-3c4907b7a8ad-signing-cabundle\") pod \"service-ca-5fffc97bdf-nf6sx\" (UID: \"bab893eb-ddd1-41b0-80a7-3c4907b7a8ad\") " pod="openshift-service-ca/service-ca-5fffc97bdf-nf6sx" Oct 30 08:53:51.862504 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:51.862464 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnszd\" (UniqueName: \"kubernetes.io/projected/54ef0e46-200b-4e7d-a007-ad959dec4486-kube-api-access-vnszd\") pod \"migrator-66df5d496b-nlv7n\" (UID: \"54ef0e46-200b-4e7d-a007-ad959dec4486\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nlv7n" Oct 30 08:53:51.863087 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:51.863062 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/bab893eb-ddd1-41b0-80a7-3c4907b7a8ad-signing-cabundle\") pod \"service-ca-5fffc97bdf-nf6sx\" (UID: \"bab893eb-ddd1-41b0-80a7-3c4907b7a8ad\") " pod="openshift-service-ca/service-ca-5fffc97bdf-nf6sx" Oct 30 08:53:51.864713 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:51.864693 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/bab893eb-ddd1-41b0-80a7-3c4907b7a8ad-signing-key\") pod \"service-ca-5fffc97bdf-nf6sx\" (UID: \"bab893eb-ddd1-41b0-80a7-3c4907b7a8ad\") " pod="openshift-service-ca/service-ca-5fffc97bdf-nf6sx" Oct 30 08:53:51.880439 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:51.880419 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dhlm\" (UniqueName: \"kubernetes.io/projected/bab893eb-ddd1-41b0-80a7-3c4907b7a8ad-kube-api-access-6dhlm\") pod \"service-ca-5fffc97bdf-nf6sx\" (UID: \"bab893eb-ddd1-41b0-80a7-3c4907b7a8ad\") " pod="openshift-service-ca/service-ca-5fffc97bdf-nf6sx" Oct 30 08:53:51.880542 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:51.880525 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnszd\" (UniqueName: \"kubernetes.io/projected/54ef0e46-200b-4e7d-a007-ad959dec4486-kube-api-access-vnszd\") pod \"migrator-66df5d496b-nlv7n\" (UID: \"54ef0e46-200b-4e7d-a007-ad959dec4486\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nlv7n" Oct 30 08:53:51.933336 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:51.933291 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-nf6sx" Oct 30 08:53:51.948348 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:51.948319 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nlv7n" Oct 30 08:53:52.254531 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:52.254495 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-z7wg9-fvpzr" event={"ID":"6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2","Type":"ContainerStarted","Data":"50d630405f1e6ff1b6bbb8e451da2ab8317b82f08e08cfce7f1fdf5cd0ea7728"} Oct 30 08:53:52.422586 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:52.422554 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-nf6sx"] Oct 30 08:53:52.426664 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:52.426628 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbab893eb_ddd1_41b0_80a7_3c4907b7a8ad.slice/crio-e49d0e65b9a49b1ead0f025a919a2e9ba5148272a2f16f6a75a9bf3d2f3e9050 WatchSource:0}: Error finding container e49d0e65b9a49b1ead0f025a919a2e9ba5148272a2f16f6a75a9bf3d2f3e9050: Status 404 returned error can't find the container with id e49d0e65b9a49b1ead0f025a919a2e9ba5148272a2f16f6a75a9bf3d2f3e9050 Oct 30 08:53:52.436135 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:52.436112 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-nlv7n"] Oct 30 08:53:52.438985 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:52.438961 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54ef0e46_200b_4e7d_a007_ad959dec4486.slice/crio-2c27bf024d4005a39e4dab9ca5956a67cc53513bcdeff79e4b17cca7f7df2898 WatchSource:0}: Error finding container 2c27bf024d4005a39e4dab9ca5956a67cc53513bcdeff79e4b17cca7f7df2898: Status 404 returned error can't find the container with id 2c27bf024d4005a39e4dab9ca5956a67cc53513bcdeff79e4b17cca7f7df2898 Oct 30 08:53:52.569677 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:52.569638 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6edea6bc-4807-48ff-b879-a77fc07d92de-metrics-tls\") pod \"dns-default-95kld\" (UID: \"6edea6bc-4807-48ff-b879-a77fc07d92de\") " pod="openshift-dns/dns-default-95kld" Oct 30 08:53:52.569824 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:52.569793 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 08:53:52.569875 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:52.569855 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6edea6bc-4807-48ff-b879-a77fc07d92de-metrics-tls podName:6edea6bc-4807-48ff-b879-a77fc07d92de nodeName:}" failed. No retries permitted until 2026-10-30 08:53:56.569837289 +0000 UTC m=+36.081809140 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6edea6bc-4807-48ff-b879-a77fc07d92de-metrics-tls") pod "dns-default-95kld" (UID: "6edea6bc-4807-48ff-b879-a77fc07d92de") : secret "dns-default-metrics-tls" not found Oct 30 08:53:52.670340 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:52.670303 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b79e9a37-068d-4387-a07b-84b094e6895a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lc8jz\" (UID: \"b79e9a37-068d-4387-a07b-84b094e6895a\") " pod="openshift-insights/insights-runtime-extractor-lc8jz" Oct 30 08:53:52.670520 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:52.670405 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b79e9a37-068d-4387-a07b-84b094e6895a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lc8jz\" (UID: \"b79e9a37-068d-4387-a07b-84b094e6895a\") " pod="openshift-insights/insights-runtime-extractor-lc8jz" Oct 30 08:53:52.670520 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:52.670452 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 08:53:52.670520 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:52.670520 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b79e9a37-068d-4387-a07b-84b094e6895a-kube-rbac-proxy-cm podName:b79e9a37-068d-4387-a07b-84b094e6895a nodeName:}" failed. No retries permitted until 2026-10-30 08:53:54.670504933 +0000 UTC m=+34.182476778 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b79e9a37-068d-4387-a07b-84b094e6895a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lc8jz" (UID: "b79e9a37-068d-4387-a07b-84b094e6895a") : configmap "kube-rbac-proxy" not found Oct 30 08:53:52.670661 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:52.670521 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 08:53:52.670661 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:52.670574 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b79e9a37-068d-4387-a07b-84b094e6895a-insights-runtime-extractor-tls podName:b79e9a37-068d-4387-a07b-84b094e6895a nodeName:}" failed. No retries permitted until 2026-10-30 08:53:54.670558377 +0000 UTC m=+34.182530226 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b79e9a37-068d-4387-a07b-84b094e6895a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lc8jz" (UID: "b79e9a37-068d-4387-a07b-84b094e6895a") : secret "insights-runtime-extractor-tls" not found Oct 30 08:53:53.259103 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:53.259023 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nlv7n" event={"ID":"54ef0e46-200b-4e7d-a007-ad959dec4486","Type":"ContainerStarted","Data":"2c27bf024d4005a39e4dab9ca5956a67cc53513bcdeff79e4b17cca7f7df2898"} Oct 30 08:53:53.260509 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:53.260442 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-nf6sx" event={"ID":"bab893eb-ddd1-41b0-80a7-3c4907b7a8ad","Type":"ContainerStarted","Data":"e49d0e65b9a49b1ead0f025a919a2e9ba5148272a2f16f6a75a9bf3d2f3e9050"} Oct 30 08:53:53.262294 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:53.262271 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lmjss" event={"ID":"deaa55d7-ac05-4aa1-909a-4237addf17ea","Type":"ContainerStarted","Data":"a9c481d428d5b9d9da52d786e02911fe5c33110067303885a879fc2ed1911882"} Oct 30 08:53:53.299473 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:53.299429 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-lmjss" podStartSLOduration=2.630216066 podStartE2EDuration="5.299414093s" totalImagesPullingTime="2.669198027s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:53:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:53:53.298932285 +0000 UTC m=+32.810904144" watchObservedRunningTime="2026-10-30 08:53:53.299414093 +0000 UTC m=+32.811385950" Oct 30 08:53:53.780640 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:53.780561 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c94bc06-63d6-454f-9a56-1253c0315672-metrics-certs\") pod \"network-metrics-daemon-z66mv\" (UID: \"4c94bc06-63d6-454f-9a56-1253c0315672\") " pod="openshift-multus/network-metrics-daemon-z66mv" Oct 30 08:53:53.780640 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:53.780632 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5e591466-fcf6-469d-9c3d-cde0767f4dde-original-pull-secret\") pod \"global-pull-secret-syncer-vb26w\" (UID: \"5e591466-fcf6-469d-9c3d-cde0767f4dde\") " pod="kube-system/global-pull-secret-syncer-vb26w" Oct 30 08:53:53.781054 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:53.780743 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 08:53:53.781054 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:53.780826 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c94bc06-63d6-454f-9a56-1253c0315672-metrics-certs podName:4c94bc06-63d6-454f-9a56-1253c0315672 nodeName:}" failed. No retries permitted until 2026-10-30 08:54:25.780804128 +0000 UTC m=+65.292775980 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4c94bc06-63d6-454f-9a56-1253c0315672-metrics-certs") pod "network-metrics-daemon-z66mv" (UID: "4c94bc06-63d6-454f-9a56-1253c0315672") : secret "metrics-daemon-secret" not found Oct 30 08:53:53.782978 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:53.782955 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5e591466-fcf6-469d-9c3d-cde0767f4dde-original-pull-secret\") pod \"global-pull-secret-syncer-vb26w\" (UID: \"5e591466-fcf6-469d-9c3d-cde0767f4dde\") " pod="kube-system/global-pull-secret-syncer-vb26w" Oct 30 08:53:53.798087 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:53.798061 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vb26w" Oct 30 08:53:53.881282 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:53.881251 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfhrk\" (UniqueName: \"kubernetes.io/projected/0278c964-1ae2-42e7-a5b3-1df265e86edc-kube-api-access-rfhrk\") pod \"network-check-target-mz2kl\" (UID: \"0278c964-1ae2-42e7-a5b3-1df265e86edc\") " pod="openshift-network-diagnostics/network-check-target-mz2kl" Oct 30 08:53:53.883807 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:53.883781 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfhrk\" (UniqueName: \"kubernetes.io/projected/0278c964-1ae2-42e7-a5b3-1df265e86edc-kube-api-access-rfhrk\") pod \"network-check-target-mz2kl\" (UID: \"0278c964-1ae2-42e7-a5b3-1df265e86edc\") " pod="openshift-network-diagnostics/network-check-target-mz2kl" Oct 30 08:53:54.102680 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:54.102611 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mz2kl" Oct 30 08:53:54.107758 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:54.107669 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vb26w"] Oct 30 08:53:54.110843 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:54.110815 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e591466_fcf6_469d_9c3d_cde0767f4dde.slice/crio-47967a90d27e4dc73c73fd6ce741e8a8db648d8ce3961b2154549ef3e02a2ffa WatchSource:0}: Error finding container 47967a90d27e4dc73c73fd6ce741e8a8db648d8ce3961b2154549ef3e02a2ffa: Status 404 returned error can't find the container with id 47967a90d27e4dc73c73fd6ce741e8a8db648d8ce3961b2154549ef3e02a2ffa Oct 30 08:53:54.265946 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:54.265912 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vb26w" event={"ID":"5e591466-fcf6-469d-9c3d-cde0767f4dde","Type":"ContainerStarted","Data":"47967a90d27e4dc73c73fd6ce741e8a8db648d8ce3961b2154549ef3e02a2ffa"} Oct 30 08:53:54.267372 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:54.267340 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nlv7n" event={"ID":"54ef0e46-200b-4e7d-a007-ad959dec4486","Type":"ContainerStarted","Data":"28229b2d022dc1e53da014e316bdcf2814007883a8325f9947e0926579647fcc"} Oct 30 08:53:54.303496 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:54.303451 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nlv7n" podStartSLOduration=2.036807594 podStartE2EDuration="3.303434633s" totalImagesPullingTime="1.266627039s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:53:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:53:54.303031462 +0000 UTC m=+33.815003320" watchObservedRunningTime="2026-10-30 08:53:54.303434633 +0000 UTC m=+33.815406488" Oct 30 08:53:54.411313 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:54.411282 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mz2kl"] Oct 30 08:53:54.413810 ip-10-0-143-244 kubenswrapper[2575]: W1030 08:53:54.413785 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0278c964_1ae2_42e7_a5b3_1df265e86edc.slice/crio-ce09d28db0ca7bdee23ac345de34a445d1b56433d9e1b22fa6df5fb18ab11d77 WatchSource:0}: Error finding container ce09d28db0ca7bdee23ac345de34a445d1b56433d9e1b22fa6df5fb18ab11d77: Status 404 returned error can't find the container with id ce09d28db0ca7bdee23ac345de34a445d1b56433d9e1b22fa6df5fb18ab11d77 Oct 30 08:53:54.686721 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:54.686637 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b79e9a37-068d-4387-a07b-84b094e6895a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lc8jz\" (UID: \"b79e9a37-068d-4387-a07b-84b094e6895a\") " pod="openshift-insights/insights-runtime-extractor-lc8jz" Oct 30 08:53:54.686870 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:54.686739 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b79e9a37-068d-4387-a07b-84b094e6895a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lc8jz\" (UID: \"b79e9a37-068d-4387-a07b-84b094e6895a\") " pod="openshift-insights/insights-runtime-extractor-lc8jz" Oct 30 08:53:54.686870 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:54.686834 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 08:53:54.686957 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:54.686896 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b79e9a37-068d-4387-a07b-84b094e6895a-kube-rbac-proxy-cm podName:b79e9a37-068d-4387-a07b-84b094e6895a nodeName:}" failed. No retries permitted until 2026-10-30 08:53:58.686878639 +0000 UTC m=+38.198850491 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b79e9a37-068d-4387-a07b-84b094e6895a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lc8jz" (UID: "b79e9a37-068d-4387-a07b-84b094e6895a") : configmap "kube-rbac-proxy" not found Oct 30 08:53:54.687012 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:54.686980 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 08:53:54.687068 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:54.687013 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b79e9a37-068d-4387-a07b-84b094e6895a-insights-runtime-extractor-tls podName:b79e9a37-068d-4387-a07b-84b094e6895a nodeName:}" failed. No retries permitted until 2026-10-30 08:53:58.687002746 +0000 UTC m=+38.198974584 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b79e9a37-068d-4387-a07b-84b094e6895a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lc8jz" (UID: "b79e9a37-068d-4387-a07b-84b094e6895a") : secret "insights-runtime-extractor-tls" not found Oct 30 08:53:55.272126 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:55.272086 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mz2kl" event={"ID":"0278c964-1ae2-42e7-a5b3-1df265e86edc","Type":"ContainerStarted","Data":"ce09d28db0ca7bdee23ac345de34a445d1b56433d9e1b22fa6df5fb18ab11d77"} Oct 30 08:53:55.274434 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:55.274403 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-z7wg9-fvpzr" event={"ID":"6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2","Type":"ContainerStarted","Data":"859c0d2a7fbfc11982a8900f9a8d63d376ba7f82a1a0e8ce7b0955524c93589f"} Oct 30 08:53:55.313051 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:55.313005 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-z7wg9-fvpzr" podStartSLOduration=2.642174083 podStartE2EDuration="5.31299337s" totalImagesPullingTime="2.670819287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:53:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:53:55.312435253 +0000 UTC m=+34.824407113" watchObservedRunningTime="2026-10-30 08:53:55.31299337 +0000 UTC m=+34.824965226" Oct 30 08:53:56.283233 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:56.283170 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-nf6sx" event={"ID":"bab893eb-ddd1-41b0-80a7-3c4907b7a8ad","Type":"ContainerStarted","Data":"cdc69722bae6e8ce35c7a897e90a17b1eac071bfb949d96629ff64539ff63da8"} Oct 30 08:53:56.320146 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:56.320096 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-nf6sx" podStartSLOduration=2.269443146 podStartE2EDuration="5.320084747s" totalImagesPullingTime="3.050641601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:53:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:53:56.319566203 +0000 UTC m=+35.831538060" watchObservedRunningTime="2026-10-30 08:53:56.320084747 +0000 UTC m=+35.832056601" Oct 30 08:53:56.603069 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:56.602698 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6edea6bc-4807-48ff-b879-a77fc07d92de-metrics-tls\") pod \"dns-default-95kld\" (UID: \"6edea6bc-4807-48ff-b879-a77fc07d92de\") " pod="openshift-dns/dns-default-95kld" Oct 30 08:53:56.603612 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:56.603553 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 08:53:56.604259 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:56.603861 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6edea6bc-4807-48ff-b879-a77fc07d92de-metrics-tls podName:6edea6bc-4807-48ff-b879-a77fc07d92de nodeName:}" failed. No retries permitted until 2026-10-30 08:54:04.603836661 +0000 UTC m=+44.115808510 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6edea6bc-4807-48ff-b879-a77fc07d92de-metrics-tls") pod "dns-default-95kld" (UID: "6edea6bc-4807-48ff-b879-a77fc07d92de") : secret "dns-default-metrics-tls" not found Oct 30 08:53:56.719914 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:56.719875 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h5vr9_634749a3-dbc0-41f4-9e97-bf3d25f9b6a8/konnectivity-agent/0.log" limit=1024 Oct 30 08:53:56.827979 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:56.827935 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h5vr9_634749a3-dbc0-41f4-9e97-bf3d25f9b6a8/konnectivity-agent/0.log" limit=1024 Oct 30 08:53:56.933076 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:56.932998 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h5vr9_634749a3-dbc0-41f4-9e97-bf3d25f9b6a8/konnectivity-agent/0.log" limit=1024 Oct 30 08:53:57.059882 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:57.059836 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h5vr9_634749a3-dbc0-41f4-9e97-bf3d25f9b6a8/konnectivity-agent/0.log" limit=1024 Oct 30 08:53:57.170132 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:57.170093 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h5vr9_634749a3-dbc0-41f4-9e97-bf3d25f9b6a8/konnectivity-agent/0.log" limit=1024 Oct 30 08:53:57.232334 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:57.232310 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lq2p2_9c67a6b6-a3d8-49bc-a678-b203ef4201a9/dns-node-resolver/0.log" Oct 30 08:53:57.274854 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:57.274822 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h5vr9_634749a3-dbc0-41f4-9e97-bf3d25f9b6a8/konnectivity-agent/0.log" limit=1024 Oct 30 08:53:57.503958 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:57.503876 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h5vr9_634749a3-dbc0-41f4-9e97-bf3d25f9b6a8/konnectivity-agent/0.log" limit=1024 Oct 30 08:53:58.294734 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:58.294693 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mz2kl" event={"ID":"0278c964-1ae2-42e7-a5b3-1df265e86edc","Type":"ContainerStarted","Data":"ef0e673fb3a0f19fa97fe7bb6526a975f4f7b65dd5e411d6a1da7746c7423e16"} Oct 30 08:53:58.294890 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:58.294861 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mz2kl" Oct 30 08:53:58.333998 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:58.333952 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mz2kl" podStartSLOduration=33.593884354 podStartE2EDuration="37.333940375s" totalImagesPullingTime="3.740056021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:53:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:53:58.333797193 +0000 UTC m=+37.845769069" watchObservedRunningTime="2026-10-30 08:53:58.333940375 +0000 UTC m=+37.845912226" Oct 30 08:53:58.431533 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:58.431454 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xqjlj_1b64ad2b-20eb-4a29-93b9-ccc9a217135d/node-ca/0.log" Oct 30 08:53:58.718359 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:58.718267 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b79e9a37-068d-4387-a07b-84b094e6895a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lc8jz\" (UID: \"b79e9a37-068d-4387-a07b-84b094e6895a\") " pod="openshift-insights/insights-runtime-extractor-lc8jz" Oct 30 08:53:58.718841 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:58.718421 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b79e9a37-068d-4387-a07b-84b094e6895a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lc8jz\" (UID: \"b79e9a37-068d-4387-a07b-84b094e6895a\") " pod="openshift-insights/insights-runtime-extractor-lc8jz" Oct 30 08:53:58.718841 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:58.718563 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 08:53:58.718841 ip-10-0-143-244 kubenswrapper[2575]: E1030 08:53:58.718626 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b79e9a37-068d-4387-a07b-84b094e6895a-insights-runtime-extractor-tls podName:b79e9a37-068d-4387-a07b-84b094e6895a nodeName:}" failed. No retries permitted until 2026-10-30 08:54:06.718608411 +0000 UTC m=+46.230580270 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b79e9a37-068d-4387-a07b-84b094e6895a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lc8jz" (UID: "b79e9a37-068d-4387-a07b-84b094e6895a") : secret "insights-runtime-extractor-tls" not found Oct 30 08:53:58.719033 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:58.718935 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b79e9a37-068d-4387-a07b-84b094e6895a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lc8jz\" (UID: \"b79e9a37-068d-4387-a07b-84b094e6895a\") " pod="openshift-insights/insights-runtime-extractor-lc8jz" Oct 30 08:53:58.834420 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:58.831343 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lmjss_deaa55d7-ac05-4aa1-909a-4237addf17ea/serve-healthcheck-canary/0.log" Oct 30 08:53:59.431619 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:59.431591 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-nlv7n_54ef0e46-200b-4e7d-a007-ad959dec4486/migrator/0.log" Oct 30 08:53:59.685349 ip-10-0-143-244 kubenswrapper[2575]: I1030 08:53:59.685238 2575 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 08:53:59.685409 ip-10-0-143-244 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 08:53:59.696848 ip-10-0-143-244 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 08:53:59.697013 ip-10-0-143-244 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 08:53:59.697060 ip-10-0-143-244 systemd[1]: kubelet.service: Consumed 2.341s CPU time. Oct 30 08:53:59.709792 ip-10-0-143-244 systemd[1]: Starting Kubernetes Kubelet... Oct 30 08:53:59.794316 ip-10-0-143-244 kubenswrapper[4562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 08:53:59.794316 ip-10-0-143-244 kubenswrapper[4562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 08:53:59.794316 ip-10-0-143-244 kubenswrapper[4562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 08:53:59.794316 ip-10-0-143-244 kubenswrapper[4562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 08:53:59.796448 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796358 4562 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 08:53:59.796448 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796447 4562 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 08:53:59.796448 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796452 4562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 08:53:59.796573 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796456 4562 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 08:53:59.796573 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796460 4562 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 08:53:59.796573 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796463 4562 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 08:53:59.796573 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796465 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 08:53:59.796573 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796468 4562 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 08:53:59.796573 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796471 4562 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 08:53:59.796573 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796474 4562 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 08:53:59.796573 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796476 4562 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 08:53:59.796573 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796479 4562 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 08:53:59.796573 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796482 4562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 08:53:59.796573 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796485 4562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 08:53:59.796573 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796491 4562 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 08:53:59.796573 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796494 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 08:53:59.796573 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796497 4562 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 08:53:59.796573 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796500 4562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 08:53:59.796573 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796502 4562 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 08:53:59.796573 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796505 4562 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 08:53:59.796573 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796507 4562 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 08:53:59.796573 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796510 4562 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 08:53:59.797046 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796513 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 08:53:59.797046 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796516 4562 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 08:53:59.797046 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796518 4562 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 08:53:59.797046 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796522 4562 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 08:53:59.797046 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796525 4562 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 08:53:59.797046 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796527 4562 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 08:53:59.797046 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796531 4562 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 08:53:59.797046 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796533 4562 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 08:53:59.797046 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796536 4562 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 08:53:59.797046 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796539 4562 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 08:53:59.797046 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796542 4562 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 08:53:59.797046 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796544 4562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 08:53:59.797046 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796547 4562 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 08:53:59.797046 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796549 4562 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 08:53:59.797046 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796552 4562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 08:53:59.797046 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796554 4562 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 08:53:59.797046 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796557 4562 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 08:53:59.797046 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796560 4562 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 08:53:59.797046 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796562 4562 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 08:53:59.797046 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796565 4562 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 08:53:59.797573 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796568 4562 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 08:53:59.797573 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796570 4562 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 08:53:59.797573 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796574 4562 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 08:53:59.797573 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796577 4562 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 08:53:59.797573 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796579 4562 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 08:53:59.797573 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796582 4562 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 08:53:59.797573 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796585 4562 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 08:53:59.797573 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796588 4562 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 08:53:59.797573 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796599 4562 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 08:53:59.797573 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796602 4562 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 08:53:59.797573 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796605 4562 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 08:53:59.797573 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796607 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 08:53:59.797573 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796610 4562 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 08:53:59.797573 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796612 4562 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 08:53:59.797573 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796615 4562 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 08:53:59.797573 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796622 4562 feature_gate.go:386] unrecognized feature gate: Example Oct 30 08:53:59.797573 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796624 4562 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 08:53:59.797573 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796627 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 08:53:59.797573 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796630 4562 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 08:53:59.797573 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796632 4562 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 08:53:59.798074 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796635 4562 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 08:53:59.798074 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796643 4562 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 08:53:59.798074 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796646 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 08:53:59.798074 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796649 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 08:53:59.798074 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796651 4562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 08:53:59.798074 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796654 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 08:53:59.798074 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796657 4562 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 08:53:59.798074 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796659 4562 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 08:53:59.798074 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796662 4562 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 08:53:59.798074 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796665 4562 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 08:53:59.798074 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796668 4562 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 08:53:59.798074 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796670 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 08:53:59.798074 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796673 4562 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 08:53:59.798074 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796675 4562 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 08:53:59.798074 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796678 4562 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 08:53:59.798074 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796681 4562 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 08:53:59.798074 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796684 4562 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 08:53:59.798074 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796686 4562 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 08:53:59.798074 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796689 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 08:53:59.798074 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796692 4562 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 08:53:59.798592 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796695 4562 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 08:53:59.798592 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796697 4562 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 08:53:59.798592 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796700 4562 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 08:53:59.798592 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796702 4562 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 08:53:59.798592 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796706 4562 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 08:53:59.798592 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796710 4562 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 08:53:59.798592 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796716 4562 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 08:53:59.798592 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796721 4562 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 08:53:59.798592 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796724 4562 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 08:53:59.798592 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796727 4562 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 08:53:59.798592 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796729 4562 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 08:53:59.798592 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796732 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 08:53:59.798592 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796734 4562 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 08:53:59.798592 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796737 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 08:53:59.798592 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796739 4562 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 08:53:59.798592 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796742 4562 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 08:53:59.798592 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796744 4562 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 08:53:59.798592 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796747 4562 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 08:53:59.798592 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796750 4562 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 08:53:59.799118 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796752 4562 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 08:53:59.799118 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796755 4562 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 08:53:59.799118 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796757 4562 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 08:53:59.799118 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796760 4562 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 08:53:59.799118 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.796763 4562 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 08:53:59.799118 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797575 4562 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 08:53:59.799118 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797584 4562 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 08:53:59.799118 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797587 4562 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 08:53:59.799118 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797591 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 08:53:59.799118 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797594 4562 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 08:53:59.799118 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797597 4562 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 08:53:59.799118 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797600 4562 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 08:53:59.799118 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797603 4562 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 08:53:59.799118 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797607 4562 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 08:53:59.799118 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797609 4562 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 08:53:59.799118 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797612 4562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 08:53:59.799118 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797615 4562 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 08:53:59.799118 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797617 4562 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 08:53:59.799118 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797620 4562 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 08:53:59.799118 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797622 4562 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 08:53:59.799651 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797626 4562 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 08:53:59.799651 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797628 4562 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 08:53:59.799651 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797631 4562 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 08:53:59.799651 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797634 4562 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 08:53:59.799651 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797636 4562 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 08:53:59.799651 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797639 4562 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 08:53:59.799651 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797642 4562 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 08:53:59.799651 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797645 4562 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 08:53:59.799651 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797647 4562 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 08:53:59.799651 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797650 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 08:53:59.799651 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797652 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 08:53:59.799651 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797655 4562 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 08:53:59.799651 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797657 4562 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 08:53:59.799651 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797660 4562 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 08:53:59.799651 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797663 4562 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 08:53:59.799651 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797666 4562 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 08:53:59.799651 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797668 4562 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 08:53:59.799651 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797671 4562 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 08:53:59.799651 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797674 4562 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 08:53:59.799651 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797676 4562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 08:53:59.800149 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797679 4562 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 08:53:59.800149 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797682 4562 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 08:53:59.800149 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797684 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 08:53:59.800149 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797687 4562 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 08:53:59.800149 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797689 4562 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 08:53:59.800149 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797692 4562 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 08:53:59.800149 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797695 4562 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 08:53:59.800149 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797697 4562 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 08:53:59.800149 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797700 4562 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 08:53:59.800149 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797703 4562 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 08:53:59.800149 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797705 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 08:53:59.800149 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797707 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 08:53:59.800149 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797710 4562 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 08:53:59.800149 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797713 4562 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 08:53:59.800149 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797715 4562 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 08:53:59.800149 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797718 4562 feature_gate.go:386] unrecognized feature gate: Example Oct 30 08:53:59.800149 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797720 4562 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 08:53:59.800149 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797723 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 08:53:59.800149 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797726 4562 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 08:53:59.800667 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797728 4562 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 08:53:59.800667 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797731 4562 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 08:53:59.800667 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797733 4562 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 08:53:59.800667 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797736 4562 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 08:53:59.800667 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797740 4562 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 08:53:59.800667 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797744 4562 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 08:53:59.800667 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797748 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 08:53:59.800667 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797751 4562 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 08:53:59.800667 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797754 4562 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 08:53:59.800667 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797756 4562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 08:53:59.800667 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797759 4562 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 08:53:59.800667 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797762 4562 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 08:53:59.800667 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797766 4562 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 08:53:59.800667 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797769 4562 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 08:53:59.800667 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797772 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 08:53:59.800667 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797775 4562 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 08:53:59.800667 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797777 4562 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 08:53:59.800667 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797780 4562 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 08:53:59.800667 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797783 4562 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 08:53:59.801137 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797786 4562 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 08:53:59.801137 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797789 4562 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 08:53:59.801137 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797791 4562 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 08:53:59.801137 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797794 4562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 08:53:59.801137 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797797 4562 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 08:53:59.801137 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797799 4562 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 08:53:59.801137 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797802 4562 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 08:53:59.801137 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797805 4562 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 08:53:59.801137 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797807 4562 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 08:53:59.801137 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797810 4562 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 08:53:59.801137 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797812 4562 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 08:53:59.801137 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797815 4562 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 08:53:59.801137 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797818 4562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 08:53:59.801137 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797820 4562 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 08:53:59.801137 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797823 4562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 08:53:59.801137 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797825 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 08:53:59.801137 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797828 4562 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 08:53:59.801137 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797831 4562 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 08:53:59.801137 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797833 4562 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 08:53:59.801137 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797836 4562 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 08:53:59.801655 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797839 4562 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 08:53:59.801655 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797841 4562 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 08:53:59.801655 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797844 4562 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 08:53:59.801655 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797847 4562 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 08:53:59.801655 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797849 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 08:53:59.801655 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797852 4562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 08:53:59.801655 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797854 4562 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 08:53:59.801655 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797856 4562 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 08:53:59.801655 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797859 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 08:53:59.801655 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797861 4562 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 08:53:59.801655 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797864 4562 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 08:53:59.801655 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797866 4562 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 08:53:59.801655 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.797869 4562 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 08:53:59.801655 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798327 4562 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 08:53:59.801655 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798332 4562 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 08:53:59.801655 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798335 4562 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 08:53:59.801655 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798337 4562 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 08:53:59.801655 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798340 4562 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 08:53:59.801655 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798343 4562 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 08:53:59.802144 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798345 4562 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 08:53:59.802144 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798348 4562 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 08:53:59.802144 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798351 4562 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 08:53:59.802144 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798353 4562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 08:53:59.802144 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798356 4562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 08:53:59.802144 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798361 4562 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 08:53:59.802144 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798363 4562 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 08:53:59.802144 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798366 4562 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 08:53:59.802144 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798369 4562 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 08:53:59.802144 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798372 4562 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 08:53:59.802144 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798376 4562 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 08:53:59.802144 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798394 4562 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 08:53:59.802144 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798397 4562 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 08:53:59.802144 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798400 4562 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 08:53:59.802144 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798404 4562 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 08:53:59.802144 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798407 4562 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 08:53:59.802144 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798410 4562 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 08:53:59.802144 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798412 4562 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 08:53:59.802144 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798415 4562 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 08:53:59.802625 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798418 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 08:53:59.802625 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798420 4562 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 08:53:59.802625 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798423 4562 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 08:53:59.802625 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798426 4562 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 08:53:59.802625 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798429 4562 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 08:53:59.802625 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798432 4562 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 08:53:59.802625 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798434 4562 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 08:53:59.802625 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798437 4562 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 08:53:59.802625 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798440 4562 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 08:53:59.802625 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798443 4562 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 08:53:59.802625 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798445 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 08:53:59.802625 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798448 4562 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 08:53:59.802625 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798451 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 08:53:59.802625 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798453 4562 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 08:53:59.802625 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798456 4562 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 08:53:59.802625 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798458 4562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 08:53:59.802625 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798461 4562 feature_gate.go:386] unrecognized feature gate: Example Oct 30 08:53:59.802625 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798464 4562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 08:53:59.802625 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798468 4562 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 08:53:59.802625 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798470 4562 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 08:53:59.803123 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798473 4562 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 08:53:59.803123 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798475 4562 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 08:53:59.803123 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798478 4562 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 08:53:59.803123 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798481 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 08:53:59.803123 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798483 4562 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 08:53:59.803123 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798486 4562 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 08:53:59.803123 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798488 4562 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 08:53:59.803123 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798492 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 08:53:59.803123 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798494 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 08:53:59.803123 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798497 4562 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 08:53:59.803123 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798500 4562 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 08:53:59.803123 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798502 4562 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 08:53:59.803123 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798505 4562 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 08:53:59.803123 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798507 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 08:53:59.803123 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798510 4562 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 08:53:59.803123 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798512 4562 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 08:53:59.803123 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798515 4562 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 08:53:59.803123 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798518 4562 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 08:53:59.803123 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798520 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 08:53:59.803600 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798523 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 08:53:59.803600 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798526 4562 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 08:53:59.803600 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798528 4562 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 08:53:59.803600 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798531 4562 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 08:53:59.803600 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798534 4562 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 08:53:59.803600 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798536 4562 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 08:53:59.803600 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798539 4562 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 08:53:59.803600 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798544 4562 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 08:53:59.803600 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798548 4562 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 08:53:59.803600 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798551 4562 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 08:53:59.803600 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798554 4562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 08:53:59.803600 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798558 4562 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 08:53:59.803600 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798560 4562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 08:53:59.803600 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798563 4562 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 08:53:59.803600 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798566 4562 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 08:53:59.803600 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798569 4562 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 08:53:59.803600 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798571 4562 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 08:53:59.803600 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798574 4562 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 08:53:59.803600 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798576 4562 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 08:53:59.803600 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798579 4562 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 08:53:59.804232 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798583 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 08:53:59.804232 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798586 4562 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 08:53:59.804232 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798588 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 08:53:59.804232 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798591 4562 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 08:53:59.804232 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798594 4562 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 08:53:59.804232 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798597 4562 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 08:53:59.804232 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798600 4562 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 08:53:59.804232 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798602 4562 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 08:53:59.804232 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798605 4562 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 08:53:59.804232 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798608 4562 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 08:53:59.804232 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798610 4562 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 08:53:59.804232 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798613 4562 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 08:53:59.804232 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798615 4562 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 08:53:59.804232 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798618 4562 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 08:53:59.804232 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798622 4562 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 08:53:59.804232 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798625 4562 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 08:53:59.804232 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798627 4562 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 08:53:59.804232 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798630 4562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 08:53:59.804232 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798633 4562 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 08:53:59.804232 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798636 4562 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 08:53:59.804997 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798639 4562 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 08:53:59.804997 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.798641 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 08:53:59.804997 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.798758 4562 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 08:53:59.804997 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.798770 4562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 08:53:59.804997 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.798777 4562 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 08:53:59.804997 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.798782 4562 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 08:53:59.804997 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.798786 4562 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 08:53:59.804997 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.798790 4562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 08:53:59.804997 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.798794 4562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 08:53:59.804997 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.798799 4562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 08:53:59.804997 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.798802 4562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 08:53:59.804997 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.798805 4562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 08:53:59.804997 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.798810 4562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 08:53:59.804997 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.798813 4562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 08:53:59.804997 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.798816 4562 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 08:53:59.804997 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.798819 4562 flags.go:64] FLAG: --cgroup-root="" Oct 30 08:53:59.804997 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.798822 4562 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 08:53:59.804997 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.798826 4562 flags.go:64] FLAG: --client-ca-file="" Oct 30 08:53:59.804997 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.798828 4562 flags.go:64] FLAG: --cloud-provider="external" Oct 30 08:53:59.804997 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.798831 4562 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 08:53:59.804997 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.798836 4562 flags.go:64] FLAG: --cluster-domain="" Oct 30 08:53:59.804997 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.798839 4562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 08:53:59.804997 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.798843 4562 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 08:53:59.804997 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.798846 4562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 08:53:59.805762 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.798849 4562 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 08:53:59.805762 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.798853 4562 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 08:53:59.805762 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.798856 4562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 08:53:59.805762 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.798860 4562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 08:53:59.805762 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.798863 4562 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 08:53:59.805762 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.798866 4562 flags.go:64] FLAG: --contention-profiling="false" Oct 30 08:53:59.805762 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.798870 4562 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 08:53:59.805762 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.798873 4562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 08:53:59.805762 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.798876 4562 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 08:53:59.805762 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.798879 4562 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 08:53:59.805762 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.798883 4562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 08:53:59.805762 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.798888 4562 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 08:53:59.805762 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.798891 4562 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 08:53:59.805762 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.798894 4562 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 08:53:59.805762 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.798897 4562 flags.go:64] FLAG: --enable-server="true" Oct 30 08:53:59.805762 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.798900 4562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 08:53:59.805762 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.798904 4562 flags.go:64] FLAG: --event-burst="100" Oct 30 08:53:59.805762 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.798908 4562 flags.go:64] FLAG: --event-qps="50" Oct 30 08:53:59.805762 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.798911 4562 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 08:53:59.805762 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.798914 4562 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 08:53:59.805762 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.798918 4562 flags.go:64] FLAG: --eviction-hard="" Oct 30 08:53:59.805762 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.798922 4562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 08:53:59.805762 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.798925 4562 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 08:53:59.805762 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.798928 4562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 08:53:59.805762 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.798931 4562 flags.go:64] FLAG: --eviction-soft="" Oct 30 08:53:59.806460 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.798934 4562 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 08:53:59.806460 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.798937 4562 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 08:53:59.806460 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.798940 4562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 08:53:59.806460 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.798943 4562 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 08:53:59.806460 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.798946 4562 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 08:53:59.806460 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.798949 4562 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 08:53:59.806460 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.798952 4562 flags.go:64] FLAG: --feature-gates="" Oct 30 08:53:59.806460 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.798956 4562 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 08:53:59.806460 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.798960 4562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 08:53:59.806460 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.798963 4562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 08:53:59.806460 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.798966 4562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 08:53:59.806460 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.798972 4562 flags.go:64] FLAG: --healthz-port="10248" Oct 30 08:53:59.806460 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.798976 4562 flags.go:64] FLAG: --help="false" Oct 30 08:53:59.806460 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.798979 4562 flags.go:64] FLAG: --hostname-override="ip-10-0-143-244.ec2.internal" Oct 30 08:53:59.806460 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.798982 4562 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 08:53:59.806460 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.798985 4562 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 08:53:59.806460 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.798988 4562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 08:53:59.806460 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.798992 4562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 08:53:59.806460 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.798997 4562 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 08:53:59.806460 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799000 4562 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 08:53:59.806460 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799003 4562 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 08:53:59.806460 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799006 4562 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 08:53:59.806460 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799008 4562 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 08:53:59.807043 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799011 4562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 08:53:59.807043 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799015 4562 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 08:53:59.807043 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799018 4562 flags.go:64] FLAG: --kube-reserved="" Oct 30 08:53:59.807043 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799021 4562 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 08:53:59.807043 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799026 4562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 08:53:59.807043 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799029 4562 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 08:53:59.807043 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799032 4562 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 08:53:59.807043 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799035 4562 flags.go:64] FLAG: --lock-file="" Oct 30 08:53:59.807043 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799037 4562 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 08:53:59.807043 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799040 4562 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 08:53:59.807043 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799044 4562 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 08:53:59.807043 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799049 4562 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 08:53:59.807043 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799052 4562 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 08:53:59.807043 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799055 4562 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 08:53:59.807043 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799058 4562 flags.go:64] FLAG: --logging-format="text" Oct 30 08:53:59.807043 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799061 4562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 08:53:59.807043 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799064 4562 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 08:53:59.807043 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799068 4562 flags.go:64] FLAG: --manifest-url="" Oct 30 08:53:59.807043 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799071 4562 flags.go:64] FLAG: --manifest-url-header="" Oct 30 08:53:59.807043 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799075 4562 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 08:53:59.807043 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799078 4562 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 08:53:59.807043 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799082 4562 flags.go:64] FLAG: --max-pods="110" Oct 30 08:53:59.807043 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799086 4562 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 08:53:59.807043 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799089 4562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 08:53:59.807043 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799092 4562 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 08:53:59.807683 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799095 4562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 08:53:59.807683 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799098 4562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 08:53:59.807683 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799102 4562 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 08:53:59.807683 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799105 4562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 08:53:59.807683 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799112 4562 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 08:53:59.807683 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799115 4562 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 08:53:59.807683 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799118 4562 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 08:53:59.807683 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799122 4562 flags.go:64] FLAG: --pod-cidr="" Oct 30 08:53:59.807683 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799125 4562 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 08:53:59.807683 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799128 4562 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 08:53:59.807683 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799131 4562 flags.go:64] FLAG: --pods-per-core="0" Oct 30 08:53:59.807683 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799136 4562 flags.go:64] FLAG: --port="10250" Oct 30 08:53:59.807683 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799139 4562 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 08:53:59.807683 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799142 4562 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f7ef334f56ea5bee" Oct 30 08:53:59.807683 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799145 4562 flags.go:64] FLAG: --qos-reserved="" Oct 30 08:53:59.807683 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799149 4562 flags.go:64] FLAG: --read-only-port="10255" Oct 30 08:53:59.807683 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799152 4562 flags.go:64] FLAG: --register-node="true" Oct 30 08:53:59.807683 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799155 4562 flags.go:64] FLAG: --register-with-taints="" Oct 30 08:53:59.807683 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799159 4562 flags.go:64] FLAG: --registry-burst="10" Oct 30 08:53:59.807683 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799162 4562 flags.go:64] FLAG: --registry-qps="5" Oct 30 08:53:59.807683 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799165 4562 flags.go:64] FLAG: --reserved-cpus="" Oct 30 08:53:59.807683 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799168 4562 flags.go:64] FLAG: --reserved-memory="" Oct 30 08:53:59.807683 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799171 4562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 08:53:59.807683 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799175 4562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 08:53:59.807683 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799178 4562 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 08:53:59.807683 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799181 4562 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 08:53:59.808311 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799184 4562 flags.go:64] FLAG: --runonce="false" Oct 30 08:53:59.808311 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799187 4562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 08:53:59.808311 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799190 4562 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 08:53:59.808311 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799193 4562 flags.go:64] FLAG: --seccomp-default="false" Oct 30 08:53:59.808311 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799196 4562 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 08:53:59.808311 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799199 4562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 08:53:59.808311 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799202 4562 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 08:53:59.808311 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799205 4562 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 08:53:59.808311 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799210 4562 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 08:53:59.808311 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799212 4562 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 08:53:59.808311 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799215 4562 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 08:53:59.808311 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799218 4562 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 08:53:59.808311 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799221 4562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 08:53:59.808311 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799224 4562 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 08:53:59.808311 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799227 4562 flags.go:64] FLAG: --system-cgroups="" Oct 30 08:53:59.808311 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799230 4562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 08:53:59.808311 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799235 4562 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 08:53:59.808311 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799239 4562 flags.go:64] FLAG: --tls-cert-file="" Oct 30 08:53:59.808311 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799242 4562 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 08:53:59.808311 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799246 4562 flags.go:64] FLAG: --tls-min-version="" Oct 30 08:53:59.808311 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799249 4562 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 08:53:59.808311 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799252 4562 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 08:53:59.808311 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799255 4562 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 08:53:59.808311 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799258 4562 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 08:53:59.808311 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799262 4562 flags.go:64] FLAG: --v="2" Oct 30 08:53:59.809160 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799266 4562 flags.go:64] FLAG: --version="false" Oct 30 08:53:59.809160 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799270 4562 flags.go:64] FLAG: --vmodule="" Oct 30 08:53:59.809160 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799274 4562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 08:53:59.809160 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799278 4562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 08:53:59.809160 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799420 4562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 08:53:59.809160 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799425 4562 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 08:53:59.809160 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799428 4562 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 08:53:59.809160 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799431 4562 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 08:53:59.809160 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799434 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 08:53:59.809160 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799437 4562 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 08:53:59.809160 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799439 4562 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 08:53:59.809160 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799443 4562 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 08:53:59.809160 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799445 4562 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 08:53:59.809160 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799448 4562 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 08:53:59.809160 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799451 4562 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 08:53:59.809160 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799456 4562 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 08:53:59.809160 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799458 4562 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 08:53:59.809160 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799461 4562 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 08:53:59.809160 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799464 4562 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 08:53:59.809160 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799467 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 08:53:59.809762 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799469 4562 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 08:53:59.809762 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799472 4562 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 08:53:59.809762 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799476 4562 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 08:53:59.809762 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799479 4562 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 08:53:59.809762 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799484 4562 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 08:53:59.809762 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799487 4562 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 08:53:59.809762 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799490 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 08:53:59.809762 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799493 4562 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 08:53:59.809762 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799496 4562 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 08:53:59.809762 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799500 4562 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 08:53:59.809762 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799502 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 08:53:59.809762 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799505 4562 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 08:53:59.809762 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799508 4562 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 08:53:59.809762 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799510 4562 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 08:53:59.809762 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799513 4562 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 08:53:59.809762 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799515 4562 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 08:53:59.809762 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799518 4562 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 08:53:59.809762 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799521 4562 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 08:53:59.809762 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799523 4562 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 08:53:59.810254 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799526 4562 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 08:53:59.810254 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799528 4562 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 08:53:59.810254 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799531 4562 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 08:53:59.810254 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799534 4562 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 08:53:59.810254 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799536 4562 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 08:53:59.810254 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799539 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 08:53:59.810254 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799542 4562 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 08:53:59.810254 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799544 4562 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 08:53:59.810254 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799549 4562 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 08:53:59.810254 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799552 4562 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 08:53:59.810254 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799555 4562 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 08:53:59.810254 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799557 4562 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 08:53:59.810254 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799560 4562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 08:53:59.810254 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799563 4562 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 08:53:59.810254 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799568 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 08:53:59.810254 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799571 4562 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 08:53:59.810254 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799574 4562 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 08:53:59.810254 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799578 4562 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 08:53:59.810254 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799580 4562 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 08:53:59.810254 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799583 4562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 08:53:59.810816 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799586 4562 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 08:53:59.810816 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799588 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 08:53:59.810816 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799591 4562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 08:53:59.810816 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799594 4562 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 08:53:59.810816 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799597 4562 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 08:53:59.810816 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799599 4562 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 08:53:59.810816 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799602 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 08:53:59.810816 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799604 4562 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 08:53:59.810816 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799607 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 08:53:59.810816 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799610 4562 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 08:53:59.810816 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799612 4562 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 08:53:59.810816 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799615 4562 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 08:53:59.810816 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799618 4562 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 08:53:59.810816 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799620 4562 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 08:53:59.810816 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799623 4562 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 08:53:59.810816 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799625 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 08:53:59.810816 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799628 4562 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 08:53:59.810816 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799631 4562 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 08:53:59.810816 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799634 4562 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 08:53:59.811308 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799637 4562 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 08:53:59.811308 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799641 4562 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 08:53:59.811308 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799644 4562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 08:53:59.811308 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799646 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 08:53:59.811308 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799649 4562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 08:53:59.811308 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799651 4562 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 08:53:59.811308 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799654 4562 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 08:53:59.811308 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799657 4562 feature_gate.go:386] unrecognized feature gate: Example Oct 30 08:53:59.811308 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799659 4562 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 08:53:59.811308 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799662 4562 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 08:53:59.811308 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799667 4562 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 08:53:59.811308 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799669 4562 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 08:53:59.811308 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799672 4562 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 08:53:59.811308 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799674 4562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 08:53:59.811308 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799677 4562 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 08:53:59.811308 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799679 4562 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 08:53:59.811308 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799682 4562 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 08:53:59.811308 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799685 4562 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 08:53:59.811308 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799688 4562 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 08:53:59.811934 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799691 4562 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 08:53:59.811934 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799695 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 08:53:59.811934 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799698 4562 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 08:53:59.811934 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799701 4562 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 08:53:59.811934 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799704 4562 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 08:53:59.811934 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799707 4562 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 08:53:59.811934 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799709 4562 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 08:53:59.811934 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799712 4562 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 08:53:59.811934 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799714 4562 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 08:53:59.811934 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799717 4562 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 08:53:59.811934 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799719 4562 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 08:53:59.811934 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799722 4562 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 08:53:59.811934 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799725 4562 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 08:53:59.811934 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.799752 4562 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 08:53:59.811934 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799786 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 08:53:59.811934 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799790 4562 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 08:53:59.812370 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799793 4562 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 08:53:59.812370 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799797 4562 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 08:53:59.812370 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799801 4562 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 08:53:59.812370 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799805 4562 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 08:53:59.812370 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799807 4562 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 08:53:59.812370 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799810 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 08:53:59.812370 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799813 4562 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 08:53:59.812370 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799815 4562 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 08:53:59.812370 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799819 4562 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 08:53:59.812370 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799822 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 08:53:59.812370 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799824 4562 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 08:53:59.812370 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799827 4562 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 08:53:59.812370 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799830 4562 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 08:53:59.812370 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799832 4562 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 08:53:59.812370 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799835 4562 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 08:53:59.812370 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799837 4562 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 08:53:59.812370 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799840 4562 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 08:53:59.812370 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799843 4562 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 08:53:59.812370 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799845 4562 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 08:53:59.812928 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799848 4562 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 08:53:59.812928 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799851 4562 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 08:53:59.812928 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799854 4562 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 08:53:59.812928 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799856 4562 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 08:53:59.812928 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799859 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 08:53:59.812928 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799861 4562 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 08:53:59.812928 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799864 4562 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 08:53:59.812928 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799866 4562 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 08:53:59.812928 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799869 4562 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 08:53:59.812928 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799872 4562 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 08:53:59.812928 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799875 4562 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 08:53:59.812928 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799878 4562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 08:53:59.812928 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799880 4562 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 08:53:59.812928 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799883 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 08:53:59.812928 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799885 4562 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 08:53:59.812928 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799888 4562 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 08:53:59.812928 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799891 4562 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 08:53:59.812928 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799893 4562 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 08:53:59.812928 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799896 4562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 08:53:59.812928 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799898 4562 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 08:53:59.813447 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799901 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 08:53:59.813447 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799905 4562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 08:53:59.813447 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799908 4562 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 08:53:59.813447 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799910 4562 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 08:53:59.813447 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799913 4562 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 08:53:59.813447 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799916 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 08:53:59.813447 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799918 4562 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 08:53:59.813447 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799921 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 08:53:59.813447 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799924 4562 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 08:53:59.813447 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799927 4562 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 08:53:59.813447 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799929 4562 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 08:53:59.813447 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799932 4562 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 08:53:59.813447 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799934 4562 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 08:53:59.813447 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799937 4562 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 08:53:59.813447 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799939 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 08:53:59.813447 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799942 4562 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 08:53:59.813447 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799945 4562 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 08:53:59.813447 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799947 4562 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 08:53:59.813447 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799951 4562 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 08:53:59.813923 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799953 4562 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 08:53:59.813923 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799956 4562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 08:53:59.813923 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799959 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 08:53:59.813923 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799961 4562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 08:53:59.813923 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799964 4562 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 08:53:59.813923 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799967 4562 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 08:53:59.813923 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799969 4562 feature_gate.go:386] unrecognized feature gate: Example Oct 30 08:53:59.813923 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799972 4562 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 08:53:59.813923 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799974 4562 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 08:53:59.813923 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799977 4562 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 08:53:59.813923 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799980 4562 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 08:53:59.813923 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799982 4562 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 08:53:59.813923 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799985 4562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 08:53:59.813923 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799987 4562 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 08:53:59.813923 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799992 4562 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 08:53:59.813923 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799995 4562 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 08:53:59.813923 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.799997 4562 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 08:53:59.813923 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.800000 4562 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 08:53:59.813923 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.800004 4562 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 08:53:59.814593 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.800008 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 08:53:59.814593 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.800011 4562 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 08:53:59.814593 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.800014 4562 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 08:53:59.814593 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.800017 4562 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 08:53:59.814593 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.800019 4562 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 08:53:59.814593 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.800023 4562 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 08:53:59.814593 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.800025 4562 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 08:53:59.814593 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.800029 4562 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 08:53:59.814593 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.800031 4562 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 08:53:59.814593 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.800034 4562 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 08:53:59.814593 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.800036 4562 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 08:53:59.814593 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.800039 4562 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 08:53:59.814593 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.800042 4562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 08:53:59.814593 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.800044 4562 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 08:53:59.814593 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.800047 4562 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 08:53:59.814593 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.800049 4562 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 08:53:59.814593 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.800052 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 08:53:59.814593 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.800055 4562 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 08:53:59.814593 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.800057 4562 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 08:53:59.814593 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.800060 4562 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 08:53:59.815110 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.800063 4562 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 08:53:59.815110 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.800065 4562 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 08:53:59.815110 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.800068 4562 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 08:53:59.815110 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.800072 4562 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 08:53:59.815110 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.800074 4562 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 08:53:59.815110 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.800077 4562 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 08:53:59.815110 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.800079 4562 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 08:53:59.815110 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.803520 4562 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 08:53:59.815110 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.803536 4562 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 08:53:59.815110 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.803552 4562 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 08:53:59.815110 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.803556 4562 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 08:53:59.815110 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803610 4562 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 08:53:59.815110 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803616 4562 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 08:53:59.815110 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803619 4562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 08:53:59.815110 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803623 4562 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 08:53:59.815110 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803626 4562 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 08:53:59.815110 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803628 4562 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 08:53:59.815110 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803631 4562 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 08:53:59.815110 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803634 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 08:53:59.815594 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803638 4562 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 08:53:59.815594 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803642 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 08:53:59.815594 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803645 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 08:53:59.815594 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803648 4562 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 08:53:59.815594 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803651 4562 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 08:53:59.815594 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803654 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 08:53:59.815594 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803657 4562 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 08:53:59.815594 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803659 4562 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 08:53:59.815594 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803662 4562 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 08:53:59.815594 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803665 4562 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 08:53:59.815594 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803667 4562 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 08:53:59.815594 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803670 4562 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 08:53:59.815594 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803673 4562 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 08:53:59.815594 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803676 4562 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 08:53:59.815594 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803679 4562 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 08:53:59.815594 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803682 4562 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 08:53:59.815594 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803685 4562 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 08:53:59.815594 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803688 4562 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 08:53:59.815594 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803690 4562 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 08:53:59.816063 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803693 4562 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 08:53:59.816063 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803696 4562 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 08:53:59.816063 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803698 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 08:53:59.816063 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803701 4562 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 08:53:59.816063 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803704 4562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 08:53:59.816063 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803706 4562 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 08:53:59.816063 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803709 4562 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 08:53:59.816063 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803711 4562 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 08:53:59.816063 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803714 4562 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 08:53:59.816063 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803716 4562 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 08:53:59.816063 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803719 4562 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 08:53:59.816063 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803722 4562 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 08:53:59.816063 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803724 4562 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 08:53:59.816063 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803727 4562 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 08:53:59.816063 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803730 4562 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 08:53:59.816063 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803733 4562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 08:53:59.816063 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803735 4562 feature_gate.go:386] unrecognized feature gate: Example Oct 30 08:53:59.816063 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803738 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 08:53:59.816063 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803740 4562 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 08:53:59.816063 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803743 4562 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 08:53:59.816583 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803745 4562 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 08:53:59.816583 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803748 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 08:53:59.816583 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803750 4562 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 08:53:59.816583 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803753 4562 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 08:53:59.816583 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803755 4562 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 08:53:59.816583 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803758 4562 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 08:53:59.816583 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803761 4562 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 08:53:59.816583 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803764 4562 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 08:53:59.816583 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803767 4562 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 08:53:59.816583 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803769 4562 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 08:53:59.816583 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803772 4562 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 08:53:59.816583 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803775 4562 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 08:53:59.816583 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803777 4562 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 08:53:59.816583 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803780 4562 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 08:53:59.816583 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803782 4562 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 08:53:59.816583 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803785 4562 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 08:53:59.816583 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803787 4562 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 08:53:59.816583 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803790 4562 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 08:53:59.816583 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803792 4562 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 08:53:59.817055 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803795 4562 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 08:53:59.817055 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803797 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 08:53:59.817055 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803800 4562 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 08:53:59.817055 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803803 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 08:53:59.817055 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803805 4562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 08:53:59.817055 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803808 4562 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 08:53:59.817055 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803810 4562 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 08:53:59.817055 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803813 4562 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 08:53:59.817055 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803815 4562 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 08:53:59.817055 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803818 4562 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 08:53:59.817055 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803821 4562 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 08:53:59.817055 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803823 4562 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 08:53:59.817055 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803826 4562 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 08:53:59.817055 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803828 4562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 08:53:59.817055 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803831 4562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 08:53:59.817055 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803833 4562 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 08:53:59.817055 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803836 4562 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 08:53:59.817055 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803839 4562 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 08:53:59.817055 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803842 4562 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 08:53:59.817604 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803846 4562 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 08:53:59.817604 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803850 4562 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 08:53:59.817604 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803854 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 08:53:59.817604 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803856 4562 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 08:53:59.817604 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803859 4562 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 08:53:59.817604 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803862 4562 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 08:53:59.817604 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803865 4562 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 08:53:59.817604 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803867 4562 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 08:53:59.817604 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803870 4562 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 08:53:59.817604 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803872 4562 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 08:53:59.817604 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803875 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 08:53:59.817604 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803877 4562 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 08:53:59.817604 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803880 4562 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 08:53:59.817604 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803882 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 08:53:59.817604 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803885 4562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 08:53:59.817604 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803888 4562 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 08:53:59.817604 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803891 4562 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 08:53:59.817604 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803894 4562 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 08:53:59.817604 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803896 4562 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 08:53:59.818136 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803899 4562 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 08:53:59.818136 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.803901 4562 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 08:53:59.818136 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.803929 4562 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 08:53:59.818136 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804342 4562 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 08:53:59.818136 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804357 4562 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 08:53:59.818136 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804363 4562 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 08:53:59.818136 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804369 4562 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 08:53:59.818136 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804374 4562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 08:53:59.818136 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804399 4562 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 08:53:59.818136 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804405 4562 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 08:53:59.818136 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804416 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 08:53:59.818136 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804422 4562 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 08:53:59.818136 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804427 4562 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 08:53:59.818136 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804431 4562 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 08:53:59.818136 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804436 4562 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 08:53:59.818136 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804440 4562 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 08:53:59.818552 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804446 4562 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 08:53:59.818552 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804451 4562 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 08:53:59.818552 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804456 4562 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 08:53:59.818552 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804461 4562 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 08:53:59.818552 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804468 4562 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 08:53:59.818552 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804475 4562 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 08:53:59.818552 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804486 4562 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 08:53:59.818552 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804491 4562 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 08:53:59.818552 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804495 4562 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 08:53:59.818552 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804500 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 08:53:59.818552 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804504 4562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 08:53:59.818552 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804509 4562 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 08:53:59.818552 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804513 4562 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 08:53:59.818552 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804520 4562 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 08:53:59.818552 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804527 4562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 08:53:59.818552 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804533 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 08:53:59.818552 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804538 4562 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 08:53:59.818552 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804544 4562 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 08:53:59.818552 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804549 4562 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 08:53:59.819020 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804560 4562 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 08:53:59.819020 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804565 4562 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 08:53:59.819020 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804570 4562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 08:53:59.819020 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804575 4562 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 08:53:59.819020 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804580 4562 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 08:53:59.819020 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804584 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 08:53:59.819020 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804589 4562 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 08:53:59.819020 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804594 4562 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 08:53:59.819020 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804598 4562 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 08:53:59.819020 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804603 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 08:53:59.819020 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804607 4562 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 08:53:59.819020 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804612 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 08:53:59.819020 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804622 4562 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 08:53:59.819020 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804627 4562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 08:53:59.819020 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804632 4562 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 08:53:59.819020 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804637 4562 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 08:53:59.819020 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804641 4562 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 08:53:59.819020 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804645 4562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 08:53:59.819020 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804650 4562 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 08:53:59.819527 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804654 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 08:53:59.819527 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804658 4562 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 08:53:59.819527 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804663 4562 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 08:53:59.819527 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804667 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 08:53:59.819527 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804671 4562 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 08:53:59.819527 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804676 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 08:53:59.819527 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804686 4562 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 08:53:59.819527 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804691 4562 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 08:53:59.819527 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804696 4562 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 08:53:59.819527 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804700 4562 feature_gate.go:386] unrecognized feature gate: Example Oct 30 08:53:59.819527 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804705 4562 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 08:53:59.819527 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804709 4562 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 08:53:59.819527 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804713 4562 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 08:53:59.819527 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804718 4562 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 08:53:59.819527 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804722 4562 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 08:53:59.819527 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804727 4562 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 08:53:59.819527 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804731 4562 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 08:53:59.819527 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804736 4562 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 08:53:59.819527 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804740 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 08:53:59.819527 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804750 4562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 08:53:59.820005 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804755 4562 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 08:53:59.820005 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804760 4562 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 08:53:59.820005 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804764 4562 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 08:53:59.820005 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804769 4562 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 08:53:59.820005 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804773 4562 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 08:53:59.820005 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804777 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 08:53:59.820005 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804782 4562 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 08:53:59.820005 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804787 4562 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 08:53:59.820005 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804793 4562 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 08:53:59.820005 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804797 4562 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 08:53:59.820005 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804802 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 08:53:59.820005 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804812 4562 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 08:53:59.820005 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804817 4562 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 08:53:59.820005 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804821 4562 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 08:53:59.820005 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804825 4562 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 08:53:59.820005 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804830 4562 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 08:53:59.820005 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804835 4562 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 08:53:59.820005 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804839 4562 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 08:53:59.820005 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804843 4562 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 08:53:59.820470 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804847 4562 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 08:53:59.820470 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804852 4562 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 08:53:59.820470 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804856 4562 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 08:53:59.820470 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804860 4562 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 08:53:59.820470 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804865 4562 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 08:53:59.820470 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804875 4562 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 08:53:59.820470 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804879 4562 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 08:53:59.820470 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804884 4562 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 08:53:59.820470 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804888 4562 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 08:53:59.820470 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804893 4562 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 08:53:59.820470 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804897 4562 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 08:53:59.820470 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804902 4562 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 08:53:59.820470 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804906 4562 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 08:53:59.820470 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804911 4562 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 08:53:59.820470 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804915 4562 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 08:53:59.820470 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.804919 4562 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 08:53:59.821063 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.804979 4562 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 08:53:59.821063 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805092 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 08:53:59.821063 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805100 4562 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 08:53:59.821063 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805107 4562 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 08:53:59.821063 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805117 4562 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 08:53:59.821063 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805121 4562 feature_gate.go:386] unrecognized feature gate: Example Oct 30 08:53:59.821063 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805127 4562 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 08:53:59.821063 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805131 4562 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 08:53:59.821063 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805139 4562 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 08:53:59.821063 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805144 4562 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 08:53:59.821063 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805148 4562 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 08:53:59.821063 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805152 4562 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 08:53:59.821063 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805156 4562 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 08:53:59.821063 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805160 4562 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 08:53:59.821063 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805165 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 08:53:59.821063 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805169 4562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 08:53:59.821679 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805173 4562 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 08:53:59.821679 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805184 4562 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 08:53:59.821679 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805188 4562 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 08:53:59.821679 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805192 4562 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 08:53:59.821679 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805196 4562 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 08:53:59.821679 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805201 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 08:53:59.821679 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805205 4562 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 08:53:59.821679 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805209 4562 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 08:53:59.821679 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805214 4562 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 08:53:59.821679 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805219 4562 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 08:53:59.821679 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805234 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 08:53:59.821679 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805239 4562 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 08:53:59.821679 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805244 4562 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 08:53:59.821679 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805251 4562 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 08:53:59.821679 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805254 4562 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 08:53:59.821679 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805257 4562 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 08:53:59.821679 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805260 4562 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 08:53:59.821679 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805263 4562 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 08:53:59.821679 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805266 4562 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 08:53:59.822157 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805269 4562 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 08:53:59.822157 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805271 4562 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 08:53:59.822157 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805274 4562 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 08:53:59.822157 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805277 4562 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 08:53:59.822157 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805281 4562 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 08:53:59.822157 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805285 4562 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 08:53:59.822157 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805315 4562 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 08:53:59.822157 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805348 4562 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 08:53:59.822157 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805370 4562 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 08:53:59.822157 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805477 4562 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 08:53:59.822157 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805486 4562 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 08:53:59.822157 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805491 4562 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 08:53:59.822157 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805495 4562 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 08:53:59.822157 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805500 4562 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 08:53:59.822157 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805504 4562 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 08:53:59.822157 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805508 4562 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 08:53:59.822157 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805512 4562 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 08:53:59.822157 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805516 4562 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 08:53:59.822157 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805523 4562 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 08:53:59.822157 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805527 4562 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 08:53:59.822867 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805531 4562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 08:53:59.822867 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805536 4562 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 08:53:59.822867 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805540 4562 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 08:53:59.822867 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805544 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 08:53:59.822867 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805549 4562 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 08:53:59.822867 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805553 4562 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 08:53:59.822867 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805557 4562 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 08:53:59.822867 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805561 4562 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 08:53:59.822867 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805566 4562 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 08:53:59.822867 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805570 4562 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 08:53:59.822867 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805574 4562 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 08:53:59.822867 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805579 4562 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 08:53:59.822867 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805583 4562 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 08:53:59.822867 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805592 4562 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 08:53:59.822867 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805599 4562 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 08:53:59.822867 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805603 4562 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 08:53:59.822867 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805607 4562 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 08:53:59.822867 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805612 4562 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 08:53:59.822867 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805618 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 08:53:59.822867 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805622 4562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 08:53:59.823487 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805627 4562 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 08:53:59.823487 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805631 4562 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 08:53:59.823487 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805638 4562 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 08:53:59.823487 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805645 4562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 08:53:59.823487 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805649 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 08:53:59.823487 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805655 4562 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 08:53:59.823487 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805661 4562 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 08:53:59.823487 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805665 4562 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 08:53:59.823487 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805670 4562 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 08:53:59.823487 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805675 4562 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 08:53:59.823487 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805682 4562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 08:53:59.823487 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805687 4562 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 08:53:59.823487 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805691 4562 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 08:53:59.823487 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805695 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 08:53:59.823487 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805700 4562 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 08:53:59.823487 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805705 4562 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 08:53:59.823487 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805708 4562 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 08:53:59.823487 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805710 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 08:53:59.823487 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805713 4562 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 08:53:59.823958 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805716 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 08:53:59.823958 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805718 4562 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 08:53:59.823958 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805721 4562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 08:53:59.823958 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805724 4562 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 08:53:59.823958 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805726 4562 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 08:53:59.823958 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805729 4562 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 08:53:59.823958 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805732 4562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 08:53:59.823958 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805735 4562 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 08:53:59.823958 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805737 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 08:53:59.823958 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805740 4562 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 08:53:59.823958 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805742 4562 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 08:53:59.823958 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805746 4562 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 08:53:59.823958 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:53:59.805749 4562 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 08:53:59.823958 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.805837 4562 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 08:53:59.823958 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.806787 4562 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 08:53:59.823958 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.806836 4562 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 08:53:59.823958 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.807016 4562 server.go:1014] "Starting client certificate rotation" Oct 30 08:53:59.823958 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.807153 4562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 08:53:59.824402 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.807202 4562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 08:48:20 +0000 UTC" deadline="2028-06-28 00:28:28.563186557 +0000 UTC" Oct 30 08:53:59.824402 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.807233 4562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14559h34m28.755955446s" Oct 30 08:53:59.824402 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.807490 4562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 08:53:59.824402 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.808522 4562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 08:53:59.824402 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.809760 4562 log.go:25] "Validated CRI v1 runtime API" Oct 30 08:53:59.824402 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.811364 4562 log.go:25] "Validated CRI v1 image API" Oct 30 08:53:59.824402 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.812064 4562 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 08:53:59.824402 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.813514 4562 fs.go:135] Filesystem UUIDs: map[23df4a2e-1e9e-44aa-bb40-ce1370554055:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 c3861628-df76-46ce-9483-cd17d3ddbd4a:/dev/nvme0n1p4] Oct 30 08:53:59.824666 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.813531 4562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/146d0980a6c4321fc991f6b14d8fc6c97e8ef968209481d013207827abdfae9b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/146d0980a6c4321fc991f6b14d8fc6c97e8ef968209481d013207827abdfae9b/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2c27bf024d4005a39e4dab9ca5956a67cc53513bcdeff79e4b17cca7f7df2898/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2c27bf024d4005a39e4dab9ca5956a67cc53513bcdeff79e4b17cca7f7df2898/userdata/shm major:0 minor:216 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3d97d7db0511b7383090a21ad240f629c2cf23dfb9d0aab848d0cfb968f73b1c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3d97d7db0511b7383090a21ad240f629c2cf23dfb9d0aab848d0cfb968f73b1c/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/47967a90d27e4dc73c73fd6ce741e8a8db648d8ce3961b2154549ef3e02a2ffa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/47967a90d27e4dc73c73fd6ce741e8a8db648d8ce3961b2154549ef3e02a2ffa/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4fdbbdeb1b6c4536eeb090fbdcd9727974b1d75f3492b1e0893d1d85c10d2841/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4fdbbdeb1b6c4536eeb090fbdcd9727974b1d75f3492b1e0893d1d85c10d2841/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/50d630405f1e6ff1b6bbb8e451da2ab8317b82f08e08cfce7f1fdf5cd0ea7728/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/50d630405f1e6ff1b6bbb8e451da2ab8317b82f08e08cfce7f1fdf5cd0ea7728/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/69fc2caf22a115675e3e0996034e0918c88e4902429d6e6f262f434487cdfd63/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/69fc2caf22a115675e3e0996034e0918c88e4902429d6e6f262f434487cdfd63/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6e0f69f96c4428a4db9835f68797d23a70e0b4ae1cb9e9ddd77552999fb86836/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6e0f69f96c4428a4db9835f68797d23a70e0b4ae1cb9e9ddd77552999fb86836/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/76206ead68e5c114f1d118341427645fd525e49d6c489cb683ff7175344afab2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/76206ead68e5c114f1d118341427645fd525e49d6c489cb683ff7175344afab2/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/76fe9ba212bdc314a14ad3f84c0a7b8705dcd55e5c1c8f582dbe89fba52cb848/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/76fe9ba212bdc314a14ad3f84c0a7b8705dcd55e5c1c8f582dbe89fba52cb848/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/98cb424b776de449e3a0f9c8a3b06b916346b73a4334818adaa121b5ca85e264/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/98cb424b776de449e3a0f9c8a3b06b916346b73a4334818adaa121b5ca85e264/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ce09d28db0ca7bdee23ac345de34a445d1b56433d9e1b22fa6df5fb18ab11d77/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ce09d28db0ca7bdee23ac345de34a445d1b56433d9e1b22fa6df5fb18ab11d77/userdata/shm major:0 minor:241 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/de2b1f8738fd9844a42cd79ccf8775beed1e975e0eca66054f0003b5c421b56a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/de2b1f8738fd9844a42cd79ccf8775beed1e975e0eca66054f0003b5c421b56a/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/de620c47e9a6a59465581463393d917cb0141b362f379b7199f6dd337451c103/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/de620c47e9a6a59465581463393d917cb0141b362f379b7199f6dd337451c103/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e49d0e65b9a49b1ead0f025a919a2e9ba5148272a2f16f6a75a9bf3d2f3e9050/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e49d0e65b9a49b1ead0f025a919a2e9ba5148272a2f16f6a75a9bf3d2f3e9050/userdata/shm major:0 minor:215 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f1508eeaa257ef4ba2b2e4666b7f69913c95ef0b0b585df918cf1d1557008e0c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f1508eeaa257ef4ba2b2e4666b7f69913c95ef0b0b585df918cf1d1557008e0c/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0278c964-1ae2-42e7-a5b3-1df265e86edc/volumes/kubernetes.io~projected/kube-api-access-rfhrk:{mountpoint:/var/lib/kubelet/pods/0278c964-1ae2-42e7-a5b3-1df265e86edc/volumes/kubernetes.io~projected/kube-api-access-rfhrk major:0 minor:238 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b64ad2b-20eb-4a29-93b9-ccc9a217135d/volumes/kubernetes.io~projected/kube-api-access-mtm2w:{mountpoint:/var/lib/kubelet/pods/1b64ad2b-20eb-4a29-93b9-ccc9a217135d/volumes/kubernetes.io~projected/kube-api-access-mtm2w major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1f51db5a-5720-4dea-ad02-f336b35d43b1/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/1f51db5a-5720-4dea-ad02-f336b35d43b1/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1f51db5a-5720-4dea-ad02-f336b35d43b1/volumes/kubernetes.io~projected/kube-api-access-5679q:{mountpoint:/var/lib/kubelet/pods/1f51db5a-5720-4dea-ad02-f336b35d43b1/volumes/kubernetes.io~projected/kube-api-access-5679q major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1f51db5a-5720-4dea-ad02-f336b35d43b1/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/1f51db5a-5720-4dea-ad02-f336b35d43b1/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2e880cd8-58f9-41e4-8795-4f5e978e18f5/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/2e880cd8-58f9-41e4-8795-4f5e978e18f5/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2e880cd8-58f9-41e4-8795-4f5e978e18f5/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/2e880cd8-58f9-41e4-8795-4f5e978e18f5/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2e880cd8-58f9-41e4-8795-4f5e978e18f5/volumes/kubernetes.io~projected/kube-api-access-vcnc5:{mountpoint:/var/lib/kubelet/pods/2e880cd8-58f9-41e4-8795-4f5e978e18f5/volumes/kubernetes.io~projected/kube-api-access-vcnc5 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3024e1a0-8ea6-4fc3-bb0f-e5997585eb86/volumes/kubernetes.io~projected/kube-api-access-v5d95:{mountpoint:/var/lib/kubelet/pods/3024e1a0-8ea6-4fc3-bb0f-e5997585eb86/volumes/kubernetes.io~projected/kube-api-access-v5d95 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4c94bc06-63d6-454f-9a56-1253c0315672/volumes/kubernetes.io~projected/kube-api-access-x8x4w:{mountpoint:/var/lib/kubelet/pods/4c94bc06-63d6-454f-9a56-1253c0315672/volumes/kubernetes.io~projected/kube-api-access-x8x4w major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54ef0e46-200b-4e7d-a007-ad959dec4486/volumes/kubernetes.io~projected/kube-api-access-vnszd:{mountpoint:/var/lib/kubelet/pods/54ef0e46-200b-4e7d-a007-ad959dec4486/volumes/kubernetes.io~projected/kube-api-access-vnszd major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e591466-fcf6-469d-9c3d-cde0767f4dde/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/5e591466-fcf6-469d-9c3d-cde0767f4dde/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2/volumes/kubernetes.io~projected/kube-api-access-tb9vs:{mountpoint:/var/lib/kubelet/pods/6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2/volumes/kubernetes.io~projected/kube-api-access-tb9vs major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/634749a3-dbc0-41f4-9e97-bf3d25f9b6a8/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/634749a3-dbc0-41f4-9e97-bf3d25f9b6a8/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6d1fb12b-1a27-4574-be55-140884eb868a/volumes/kubernetes.io~projected/kube-api-access-m9wwl:{mountpoint:/var/lib/kubelet/pods/6d1fb12b-1a27-4574-be55-140884eb868a/volumes/kubernetes.io~projected/kube-api-access-m9wwl major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6edea6bc-4807-48ff-b879-a77fc07d92de/volumes/kubernetes.io~projected/kube-api-access-87k9j:{mountpoint:/var/lib/kubelet/pods/6edea6bc-4807-48ff-b879-a77fc07d92de/volumes/kubernetes.io~projected/kube-api-access-87k9j major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6f09a608-14be-447e-8e6e-eaf0a0031554/volumes/kubernetes.io~projected/kube-api-access-nzmnn:{mountpoint:/var/lib/kubelet/pods/6f09a608-14be-447e-8e6e-eaf0a0031554/volumes/kubernetes.io~projected/kube-api-access-nzmnn major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/82a05e91-5a9d-4670-9882-c12ee332ca04/volumes/kubernetes.io~projected/kube-api-access-s8chm:{mountpoint:/var/lib/kubelet/pods/82a05e91-5a9d-4670-9882-c12ee332ca04/volumes/kubernetes.io~projected/kube-api-access-s8chm major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9c67a6b6-a3d8-49bc-a678-b203ef4201a9/volumes/kubernetes.io~projected/kube-api-access-rmxcr:{mountpoint:/var/lib/kubelet/pods/9c67a6b6-a3d8-49bc-a678-b203ef4201a9/volumes/kubernetes.io~projected/kube-api-access-rmxcr major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b79e9a37-068d-4387-a07b-84b094e6895a/volumes/kubernetes.io~projected/kube-api-access-bpl7f:{mountpoint:/var/lib/kubelet/pods/b79e9a37-068d-4387-a07b-84b094e6895a/volumes/kubernetes.io~projected/kube-api-access-bpl7f major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bab893eb-ddd1-41b0-80a7-3c4907b7a8ad/volumes/kubernetes.io~projected/kube-api-access-6dhlm:{mountpoint:/var/lib/kubelet/pods/bab893eb-ddd1-41b0-80a7-3c4907b7a8ad/volumes/kubernetes.io~projected/kube-api-access-6dhlm major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bab893eb-ddd1-41b0-80a7-3c4907b7a8ad/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/bab893eb-ddd1-41b0-80a7-3c4907b7a8ad/volumes/kubernetes.io~secret/signing-key major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/deaa55d7-ac05-4aa1-909a-4237addf17ea/volumes/kubernetes.io~projected/kube-api-access-nhzh5:{mountpoint:/var/lib/kubelet/pods/deaa55d7-ac05-4aa1-909a-4237addf17ea/volumes/kubernetes.io~projected/kube-api-access-nhzh5 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/deaa55d7-ac05-4aa1-909a-4237addf17ea/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/deaa55d7-ac05-4aa1-909a-4237addf17ea/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/9298de43f063e6b3b2cf93ef083e537cc529cc68d74d107243986f8e1c808cfb/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/c9d1abdf82f6205df095928182cdc5b2f746e98cbc1e98a9b136e1cc0f55687a/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/c3a108492cb519018f9565d2677dc0378e3de7ae698b229d3b1a0453feab5b57/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-110:{mountpoint:/var/lib/containers/storage/overlay/e88b4c9e7d32a8708e47f33900322cc356027e67a23d091661ff94d9d29aa15c/merged major:0 minor:110 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/b4cd932ec73d1a1869a9885addbeb8dd4da0e5dfd94542e405ad266ba95e2c16/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/48d292ead3eb4d9782ca1a35e0a3dde1f8bc1a6d25212de47bc794c370f09641/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/f37fbc792c8f833ae30101cccfc67574c86f0938655e37529624075826ed9485/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/c18b75646c25cb91b54ec41c25a2d5dfadc9e3bdf1803a3a7efc8ec004d0901b/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/478329cb5be543c55fa29d04447028b1951ade268c5694f9dfab6d9334c8714e/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/fd8b2f417625770614aad09202d0eaa0edfbd35c4778cf5cc18159009767ae1e/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/b978383a54e9465c80c93868c9bb0cc4507a6dbc837b9c030f01d02115a69f99/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-141:{mountpoint:/var/lib/containers/storage/overlay/b59061e8fb2620b193ff3c2be88cc57bbb160f6373d6986497448b69ca77c82d/merged major:0 minor:141 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/82624c78a5a1b29d19f9ef6183ea21b1c8a0a544c2a6824fd7970710e8bbc5ec/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/f9f5e8ce9f99a96053de0b7cf051fc36c85779de8913033e133d8476fbe19f38/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/4fa991d6ce3f2292a259e975b535550dec0db4ed665c5319ee1cccbf60087d94/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/6044dd279f02de8119991c515a2462959ae0a5d3a286f9c88903f549586ef421/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/6105ce2cc36a108a73755195bfea872f4e8e93840fa9cf765d74374b3b5bc5dc/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/68d1dc9b0c60a5e0168bd69201cd198e6b66ca9dc31ae12f57a71565b2611b43/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/3856591644ca219590fac3ebf275e74bc4fac271d05eb16850441f1f3544ac47/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/61f1aaea974c04c1b964682010395759ff575254bf0933a7cf02541788bf3a4f/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/0f50c7605606895f1e14edbd42248b5ad66e7520c0aa681402a221e6dc61384b/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-210:{mountpoint:/var/lib/containers/storage/overlay/cef334a0f9d8b0e7ac1bd9ed23bf459fa70355c07df75ba4edc70a61bf9f9d14/merged major:0 minor:210 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/2ba1e9d71d996749e592729601f334c12d01f203b7c2223d02eb9e2041012555/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-221:{mountpoint:/var/lib/containers/storage/overlay/d1cbea16ee2ac469e710e00175b139dd7e242aaa24bebcbab869b980a9ceb991/merged major:0 minor:221 fsType:overlay blockSize:0} overlay_0-227:{mountpoint:/var/lib/containers/storage/overlay/7980ba76930264c01545d5d2e7e396f08064eb8b61a2e97b586ef2aefc9c5fe1/merged major:0 minor:227 fsType:overlay blockSize:0} overlay_0-229:{mountpoint:/var/lib/containers/storage/overlay/c09fcdc83eabafddffc01dde950c9eb776a94dd779174fd951fdeb9eab21bac3/merged major:0 minor:229 fsType:overlay blockSize:0} overlay_0-239:{mountpoint:/var/lib/containers/storage/overlay/f642d30ea3dbd98f376e58827399d4d7a35488beae3dffe6c353785a503a2ed2/merged major:0 minor:239 fsType:overlay blockSize:0} overlay_0-243:{mountpoint:/var/lib/containers/storage/overlay/3ce180272ce16435c8d63ebfc736839d20a9b7ddb213675123da5a9d55aae31e/merged major:0 minor:243 fsType:overlay blockSize:0} overlay_0-245:{mountpoint:/var/lib/containers/storage/overlay/3668a164004abace22ac32a612b627155fcca117e13ed9a0821a19fede74c2dc/merged major:0 minor:245 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/711b7cb575865cb2aa206f64c13fb80610cd3182d2eeff66c2326830b9a0f4ea/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/eba642c69f324afb0caa0f7f9949855952f51098abae5db1d2d276b19ba997a8/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/f33638a1f612cb27ad5c897a09e089b380dd7956259cec2d7aff8233332e6c2a/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-45:{mountpoint:/var/lib/containers/storage/overlay/53c00b46d8cee6a5ae9747690d9a906fb2a1b2c86b18ff33954a3b5d84803a08/merged major:0 minor:45 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/afb9f2898e0041a5fb82f37f6bff08ddd70bf000ac1122efd467ab93656eff28/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/11ef617baa15a5c758602e1b6efb6f6199cf5131386534638bea48e2c9d47028/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/c45539c99c00e75eded7ce6b1cbaf9401a9927a758dbd3572e02b4686a34b9b1/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/b889cc88e86f76d0e50a2a6d7e6845bb103a02013848ccbd8405b78b4f761ceb/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-79:{mountpoint:/var/lib/containers/storage/overlay/56caadd7e8f159e962202e22c3a8111306d265dd5a1579b1bb579f10b7f05af2/merged major:0 minor:79 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/60f199d22957cde0498289f88e9b7177472bd2c29ccaf6061fcee2ca08909d79/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/3297ba83b6efa022b42df9b670ef208f3851b0d29b4bc8afe985533e7e71788f/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-91:{mountpoint:/var/lib/containers/storage/overlay/51d906528c6102f78345582d6569e5cb60bde749fc9ec8f37b36c3f3124a65d7/merged major:0 minor:91 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/4ac978274a8afe4aad0f02f708ca8d7ad6b9dbbae45dd479e3c4c17f497814b2/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/b68666e9ed5242504fcb004f10f99a90723e289bea1f8460e756379d2477c0e4/merged major:0 minor:98 fsType:overlay blockSize:0}] Oct 30 08:53:59.824818 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.819646 4562 manager.go:217] Machine: {Timestamp:2026-10-30 08:53:59.819197102 +0000 UTC m=+0.080758081 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3104405 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23c0e560d599e449473197f8f2cd77 SystemUUID:ec23c0e5-60d5-99e4-4947-3197f8f2cd77 BootID:8f978fbd-e73a-4901-aa4f-630411ad5fa4 Filesystems:[{Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/deaa55d7-ac05-4aa1-909a-4237addf17ea/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/54ef0e46-200b-4e7d-a007-ad959dec4486/volumes/kubernetes.io~projected/kube-api-access-vnszd DeviceMajor:0 DeviceMinor:214 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e49d0e65b9a49b1ead0f025a919a2e9ba5148272a2f16f6a75a9bf3d2f3e9050/userdata/shm DeviceMajor:0 DeviceMinor:215 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-227 DeviceMajor:0 DeviceMinor:227 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4c94bc06-63d6-454f-9a56-1253c0315672/volumes/kubernetes.io~projected/kube-api-access-x8x4w DeviceMajor:0 DeviceMinor:64 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/82a05e91-5a9d-4670-9882-c12ee332ca04/volumes/kubernetes.io~projected/kube-api-access-s8chm DeviceMajor:0 DeviceMinor:65 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/98cb424b776de449e3a0f9c8a3b06b916346b73a4334818adaa121b5ca85e264/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-221 DeviceMajor:0 DeviceMinor:221 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-243 DeviceMajor:0 DeviceMinor:243 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/de620c47e9a6a59465581463393d917cb0141b362f379b7199f6dd337451c103/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-110 DeviceMajor:0 DeviceMinor:110 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-45 DeviceMajor:0 DeviceMinor:45 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5e591466-fcf6-469d-9c3d-cde0767f4dde/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:236 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-239 DeviceMajor:0 DeviceMinor:239 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/634749a3-dbc0-41f4-9e97-bf3d25f9b6a8/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6e0f69f96c4428a4db9835f68797d23a70e0b4ae1cb9e9ddd77552999fb86836/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3d97d7db0511b7383090a21ad240f629c2cf23dfb9d0aab848d0cfb968f73b1c/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/50d630405f1e6ff1b6bbb8e451da2ab8317b82f08e08cfce7f1fdf5cd0ea7728/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4fdbbdeb1b6c4536eeb090fbdcd9727974b1d75f3492b1e0893d1d85c10d2841/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ce09d28db0ca7bdee23ac345de34a445d1b56433d9e1b22fa6df5fb18ab11d77/userdata/shm DeviceMajor:0 DeviceMinor:241 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/76fe9ba212bdc314a14ad3f84c0a7b8705dcd55e5c1c8f582dbe89fba52cb848/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/2e880cd8-58f9-41e4-8795-4f5e978e18f5/volumes/kubernetes.io~projected/kube-api-access-vcnc5 DeviceMajor:0 DeviceMinor:60 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-141 DeviceMajor:0 DeviceMinor:141 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f1508eeaa257ef4ba2b2e4666b7f69913c95ef0b0b585df918cf1d1557008e0c/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-210 DeviceMajor:0 DeviceMinor:210 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-79 DeviceMajor:0 DeviceMinor:79 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/69fc2caf22a115675e3e0996034e0918c88e4902429d6e6f262f434487cdfd63/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/47967a90d27e4dc73c73fd6ce741e8a8db648d8ce3961b2154549ef3e02a2ffa/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6f09a608-14be-447e-8e6e-eaf0a0031554/volumes/kubernetes.io~projected/kube-api-access-nzmnn DeviceMajor:0 DeviceMinor:67 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/76206ead68e5c114f1d118341427645fd525e49d6c489cb683ff7175344afab2/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-91 DeviceMajor:0 DeviceMinor:91 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2/volumes/kubernetes.io~projected/kube-api-access-tb9vs DeviceMajor:0 DeviceMinor:207 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-229 DeviceMajor:0 DeviceMinor:229 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0278c964-1ae2-42e7-a5b3-1df265e86edc/volumes/kubernetes.io~projected/kube-api-access-rfhrk DeviceMajor:0 DeviceMinor:238 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/1b64ad2b-20eb-4a29-93b9-ccc9a217135d/volumes/kubernetes.io~projected/kube-api-access-mtm2w DeviceMajor:0 DeviceMinor:62 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/1f51db5a-5720-4dea-ad02-f336b35d43b1/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1f51db5a-5720-4dea-ad02-f336b35d43b1/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/bab893eb-ddd1-41b0-80a7-3c4907b7a8ad/volumes/kubernetes.io~projected/kube-api-access-6dhlm DeviceMajor:0 DeviceMinor:213 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/2e880cd8-58f9-41e4-8795-4f5e978e18f5/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/1f51db5a-5720-4dea-ad02-f336b35d43b1/volumes/kubernetes.io~projected/kube-api-access-5679q DeviceMajor:0 DeviceMinor:66 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/deaa55d7-ac05-4aa1-909a-4237addf17ea/volumes/kubernetes.io~projected/kube-api-access-nhzh5 DeviceMajor:0 DeviceMinor:202 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/bab893eb-ddd1-41b0-80a7-3c4907b7a8ad/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:212 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-245 DeviceMajor:0 DeviceMinor:245 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3024e1a0-8ea6-4fc3-bb0f-e5997585eb86/volumes/kubernetes.io~projected/kube-api-access-v5d95 DeviceMajor:0 DeviceMinor:61 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2c27bf024d4005a39e4dab9ca5956a67cc53513bcdeff79e4b17cca7f7df2898/userdata/shm DeviceMajor:0 DeviceMinor:216 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/2e880cd8-58f9-41e4-8795-4f5e978e18f5/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9c67a6b6-a3d8-49bc-a678-b203ef4201a9/volumes/kubernetes.io~projected/kube-api-access-rmxcr DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/146d0980a6c4321fc991f6b14d8fc6c97e8ef968209481d013207827abdfae9b/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/b79e9a37-068d-4387-a07b-84b094e6895a/volumes/kubernetes.io~projected/kube-api-access-bpl7f DeviceMajor:0 DeviceMinor:206 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/6d1fb12b-1a27-4574-be55-140884eb868a/volumes/kubernetes.io~projected/kube-api-access-m9wwl DeviceMajor:0 DeviceMinor:63 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/de2b1f8738fd9844a42cd79ccf8775beed1e975e0eca66054f0003b5c421b56a/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6edea6bc-4807-48ff-b879-a77fc07d92de/volumes/kubernetes.io~projected/kube-api-access-87k9j DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:2c27bf024d4005a MacAddress:b6:7b:0c:2b:37:8c Speed:10000 Mtu:8901} {Name:47967a90d27e4dc MacAddress:9a:f0:d0:ec:48:c6 Speed:10000 Mtu:8901} {Name:50d630405f1e6ff MacAddress:ee:27:80:67:f1:24 Speed:10000 Mtu:8901} {Name:69fc2caf22a1156 MacAddress:aa:c8:29:03:1c:c7 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:c0:d4:1d:3f Speed:0 Mtu:9001} {Name:br-int MacAddress:42:92:e9:c7:f3:9a Speed:0 Mtu:8901} {Name:ce09d28db0ca7bd MacAddress:9a:1f:8b:6a:21:78 Speed:10000 Mtu:8901} {Name:e49d0e65b9a49b1 MacAddress:a2:4d:e6:6b:2a:bd Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:c0:d4:1d:3f Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:8a:a0:c4:ed:c0:28 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:3e:9e:95:ec:dd:ad Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 08:53:59.824818 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.819891 4562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 08:53:59.824818 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.819931 4562 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 08:53:59.824818 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.820200 4562 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 08:53:59.824818 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.820219 4562 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-244.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 08:53:59.824818 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.820354 4562 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 08:53:59.824818 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.820362 4562 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 08:53:59.824818 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.820372 4562 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 08:53:59.824818 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.820402 4562 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 08:53:59.824818 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.820482 4562 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 08:53:59.824818 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.821845 4562 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 08:53:59.824818 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.821967 4562 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 08:53:59.824818 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.822002 4562 kubelet.go:493] "Attempting to sync node with API server" Oct 30 08:53:59.824818 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.822017 4562 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 08:53:59.824818 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.822026 4562 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 08:53:59.824818 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.822034 4562 kubelet.go:405] "Adding apiserver pod source" Oct 30 08:53:59.824818 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.822042 4562 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 08:53:59.824818 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.822636 4562 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 08:53:59.824818 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.822647 4562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 08:53:59.824818 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.823117 4562 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 08:53:59.825483 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.825010 4562 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 08:53:59.825483 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.825020 4562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 08:53:59.825483 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.825105 4562 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 08:53:59.825483 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.825270 4562 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 08:53:59.825483 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.825296 4562 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 08:53:59.825483 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.825394 4562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 08:53:59.825483 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.825407 4562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 08:53:59.825483 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.825413 4562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 08:53:59.825483 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.825420 4562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 08:53:59.825483 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.825425 4562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 08:53:59.825483 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.825431 4562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 08:53:59.825483 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.825438 4562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 08:53:59.825483 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.825443 4562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 08:53:59.825483 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.825450 4562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 08:53:59.825483 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.825455 4562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 08:53:59.825483 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.825469 4562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 08:53:59.825483 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.825477 4562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 08:53:59.826050 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.825500 4562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 08:53:59.826050 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.825506 4562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 08:53:59.828903 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.828888 4562 server.go:1263] "Started kubelet" Oct 30 08:53:59.829282 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.828957 4562 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 08:53:59.829282 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.828988 4562 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 08:53:59.829282 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.829103 4562 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 08:53:59.829358 ip-10-0-143-244 systemd[1]: Started Kubernetes Kubelet. Oct 30 08:53:59.829610 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.829479 4562 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 08:53:59.830525 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.830505 4562 server.go:317] "Adding debug handlers to kubelet server" Oct 30 08:53:59.832364 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.832347 4562 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 08:53:59.832491 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.832366 4562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 08:53:59.832491 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.832425 4562 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 08:53:59.832491 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.832438 4562 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 08:53:59.832491 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.832422 4562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 08:48:21 +0000 UTC" deadline="2028-07-31 17:41:48.45684995 +0000 UTC" Oct 30 08:53:59.832491 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.832454 4562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15368h47m48.624398722s" Oct 30 08:53:59.832491 ip-10-0-143-244 kubenswrapper[4562]: E1030 08:53:59.832482 4562 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-244.ec2.internal\" not found" Oct 30 08:53:59.832774 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.832517 4562 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 08:53:59.841062 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.835198 4562 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f51db5a-5720-4dea-ad02-f336b35d43b1" volumeName="kubernetes.io/secret/1f51db5a-5720-4dea-ad02-f336b35d43b1-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 08:53:59.841062 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.835239 4562 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f09a608-14be-447e-8e6e-eaf0a0031554" volumeName="kubernetes.io/configmap/6f09a608-14be-447e-8e6e-eaf0a0031554-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 08:53:59.841062 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.835261 4562 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c94bc06-63d6-454f-9a56-1253c0315672" volumeName="kubernetes.io/projected/4c94bc06-63d6-454f-9a56-1253c0315672-kube-api-access-x8x4w" seLinuxMountContext="" Oct 30 08:53:59.841062 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.835277 4562 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3024e1a0-8ea6-4fc3-bb0f-e5997585eb86" volumeName="kubernetes.io/projected/3024e1a0-8ea6-4fc3-bb0f-e5997585eb86-kube-api-access-v5d95" seLinuxMountContext="" Oct 30 08:53:59.841062 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.835291 4562 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e591466-fcf6-469d-9c3d-cde0767f4dde" volumeName="kubernetes.io/secret/5e591466-fcf6-469d-9c3d-cde0767f4dde-original-pull-secret" seLinuxMountContext="" Oct 30 08:53:59.841062 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.835309 4562 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2" volumeName="kubernetes.io/projected/6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2-kube-api-access-tb9vs" seLinuxMountContext="" Oct 30 08:53:59.841062 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.835323 4562 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f09a608-14be-447e-8e6e-eaf0a0031554" volumeName="kubernetes.io/configmap/6f09a608-14be-447e-8e6e-eaf0a0031554-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 08:53:59.841062 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.835342 4562 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f09a608-14be-447e-8e6e-eaf0a0031554" volumeName="kubernetes.io/configmap/6f09a608-14be-447e-8e6e-eaf0a0031554-cni-binary-copy" seLinuxMountContext="" Oct 30 08:53:59.841062 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.835360 4562 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f09a608-14be-447e-8e6e-eaf0a0031554" volumeName="kubernetes.io/projected/6f09a608-14be-447e-8e6e-eaf0a0031554-kube-api-access-nzmnn" seLinuxMountContext="" Oct 30 08:53:59.841062 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.835395 4562 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bab893eb-ddd1-41b0-80a7-3c4907b7a8ad" volumeName="kubernetes.io/projected/bab893eb-ddd1-41b0-80a7-3c4907b7a8ad-kube-api-access-6dhlm" seLinuxMountContext="" Oct 30 08:53:59.841062 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.835411 4562 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6edea6bc-4807-48ff-b879-a77fc07d92de" volumeName="kubernetes.io/projected/6edea6bc-4807-48ff-b879-a77fc07d92de-kube-api-access-87k9j" seLinuxMountContext="" Oct 30 08:53:59.841062 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.835423 4562 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f51db5a-5720-4dea-ad02-f336b35d43b1" volumeName="kubernetes.io/projected/1f51db5a-5720-4dea-ad02-f336b35d43b1-kube-api-access-5679q" seLinuxMountContext="" Oct 30 08:53:59.841062 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.835442 4562 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2" volumeName="kubernetes.io/configmap/6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2-trusted-ca-bundle" seLinuxMountContext="" Oct 30 08:53:59.841062 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.835455 4562 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6d1fb12b-1a27-4574-be55-140884eb868a" volumeName="kubernetes.io/configmap/6d1fb12b-1a27-4574-be55-140884eb868a-cni-binary-copy" seLinuxMountContext="" Oct 30 08:53:59.841062 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.835860 4562 factory.go:55] Registering systemd factory Oct 30 08:53:59.841062 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.835879 4562 factory.go:223] Registration of the systemd container factory successfully Oct 30 08:53:59.841062 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.836016 4562 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="deaa55d7-ac05-4aa1-909a-4237addf17ea" volumeName="kubernetes.io/projected/deaa55d7-ac05-4aa1-909a-4237addf17ea-kube-api-access-nhzh5" seLinuxMountContext="" Oct 30 08:53:59.841062 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.836050 4562 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="deaa55d7-ac05-4aa1-909a-4237addf17ea" volumeName="kubernetes.io/secret/deaa55d7-ac05-4aa1-909a-4237addf17ea-cert" seLinuxMountContext="" Oct 30 08:53:59.841062 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.836068 4562 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0278c964-1ae2-42e7-a5b3-1df265e86edc" volumeName="kubernetes.io/projected/0278c964-1ae2-42e7-a5b3-1df265e86edc-kube-api-access-rfhrk" seLinuxMountContext="" Oct 30 08:53:59.841062 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.836126 4562 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b64ad2b-20eb-4a29-93b9-ccc9a217135d" volumeName="kubernetes.io/configmap/1b64ad2b-20eb-4a29-93b9-ccc9a217135d-serviceca" seLinuxMountContext="" Oct 30 08:53:59.841062 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.836144 4562 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54ef0e46-200b-4e7d-a007-ad959dec4486" volumeName="kubernetes.io/projected/54ef0e46-200b-4e7d-a007-ad959dec4486-kube-api-access-vnszd" seLinuxMountContext="" Oct 30 08:53:59.841062 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.836165 4562 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2" volumeName="kubernetes.io/configmap/6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2-service-ca-bundle" seLinuxMountContext="" Oct 30 08:53:59.841062 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.836181 4562 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2" volumeName="kubernetes.io/empty-dir/6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2-archives-path" seLinuxMountContext="" Oct 30 08:53:59.841062 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.836196 4562 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="82a05e91-5a9d-4670-9882-c12ee332ca04" volumeName="kubernetes.io/projected/82a05e91-5a9d-4670-9882-c12ee332ca04-kube-api-access-s8chm" seLinuxMountContext="" Oct 30 08:53:59.841062 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.836214 4562 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c67a6b6-a3d8-49bc-a678-b203ef4201a9" volumeName="kubernetes.io/empty-dir/9c67a6b6-a3d8-49bc-a678-b203ef4201a9-tmp-dir" seLinuxMountContext="" Oct 30 08:53:59.841062 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.836228 4562 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b79e9a37-068d-4387-a07b-84b094e6895a" volumeName="kubernetes.io/configmap/b79e9a37-068d-4387-a07b-84b094e6895a-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 08:53:59.841062 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.836247 4562 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6d1fb12b-1a27-4574-be55-140884eb868a" volumeName="kubernetes.io/projected/6d1fb12b-1a27-4574-be55-140884eb868a-kube-api-access-m9wwl" seLinuxMountContext="" Oct 30 08:53:59.841062 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.836259 4562 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c67a6b6-a3d8-49bc-a678-b203ef4201a9" volumeName="kubernetes.io/projected/9c67a6b6-a3d8-49bc-a678-b203ef4201a9-kube-api-access-rmxcr" seLinuxMountContext="" Oct 30 08:53:59.841062 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.836272 4562 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b79e9a37-068d-4387-a07b-84b094e6895a" volumeName="kubernetes.io/empty-dir/b79e9a37-068d-4387-a07b-84b094e6895a-data-volume" seLinuxMountContext="" Oct 30 08:53:59.841062 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.836289 4562 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f51db5a-5720-4dea-ad02-f336b35d43b1" volumeName="kubernetes.io/configmap/1f51db5a-5720-4dea-ad02-f336b35d43b1-env-overrides" seLinuxMountContext="" Oct 30 08:53:59.841062 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.836315 4562 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e880cd8-58f9-41e4-8795-4f5e978e18f5" volumeName="kubernetes.io/projected/2e880cd8-58f9-41e4-8795-4f5e978e18f5-kube-api-access-vcnc5" seLinuxMountContext="" Oct 30 08:53:59.841062 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.836327 4562 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="634749a3-dbc0-41f4-9e97-bf3d25f9b6a8" volumeName="kubernetes.io/configmap/634749a3-dbc0-41f4-9e97-bf3d25f9b6a8-konnectivity-ca" seLinuxMountContext="" Oct 30 08:53:59.841062 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.836344 4562 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6d1fb12b-1a27-4574-be55-140884eb868a" volumeName="kubernetes.io/configmap/6d1fb12b-1a27-4574-be55-140884eb868a-multus-daemon-config" seLinuxMountContext="" Oct 30 08:53:59.841062 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.836357 4562 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bab893eb-ddd1-41b0-80a7-3c4907b7a8ad" volumeName="kubernetes.io/configmap/bab893eb-ddd1-41b0-80a7-3c4907b7a8ad-signing-cabundle" seLinuxMountContext="" Oct 30 08:53:59.841062 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.836371 4562 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b64ad2b-20eb-4a29-93b9-ccc9a217135d" volumeName="kubernetes.io/projected/1b64ad2b-20eb-4a29-93b9-ccc9a217135d-kube-api-access-mtm2w" seLinuxMountContext="" Oct 30 08:53:59.841062 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.836417 4562 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e880cd8-58f9-41e4-8795-4f5e978e18f5" volumeName="kubernetes.io/empty-dir/2e880cd8-58f9-41e4-8795-4f5e978e18f5-etc-tuned" seLinuxMountContext="" Oct 30 08:53:59.841062 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.836431 4562 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e880cd8-58f9-41e4-8795-4f5e978e18f5" volumeName="kubernetes.io/empty-dir/2e880cd8-58f9-41e4-8795-4f5e978e18f5-tmp" seLinuxMountContext="" Oct 30 08:53:59.841062 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.836448 4562 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="634749a3-dbc0-41f4-9e97-bf3d25f9b6a8" volumeName="kubernetes.io/secret/634749a3-dbc0-41f4-9e97-bf3d25f9b6a8-agent-certs" seLinuxMountContext="" Oct 30 08:53:59.841062 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.836460 4562 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6edea6bc-4807-48ff-b879-a77fc07d92de" volumeName="kubernetes.io/empty-dir/6edea6bc-4807-48ff-b879-a77fc07d92de-tmp-dir" seLinuxMountContext="" Oct 30 08:53:59.841062 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.836473 4562 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6edea6bc-4807-48ff-b879-a77fc07d92de" volumeName="kubernetes.io/configmap/6edea6bc-4807-48ff-b879-a77fc07d92de-config-volume" seLinuxMountContext="" Oct 30 08:53:59.841062 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.836490 4562 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f51db5a-5720-4dea-ad02-f336b35d43b1" volumeName="kubernetes.io/configmap/1f51db5a-5720-4dea-ad02-f336b35d43b1-ovnkube-script-lib" seLinuxMountContext="" Oct 30 08:53:59.841062 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.836503 4562 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f51db5a-5720-4dea-ad02-f336b35d43b1" volumeName="kubernetes.io/configmap/1f51db5a-5720-4dea-ad02-f336b35d43b1-ovnkube-config" seLinuxMountContext="" Oct 30 08:53:59.841062 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.836520 4562 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3024e1a0-8ea6-4fc3-bb0f-e5997585eb86" volumeName="kubernetes.io/configmap/3024e1a0-8ea6-4fc3-bb0f-e5997585eb86-iptables-alerter-script" seLinuxMountContext="" Oct 30 08:53:59.841062 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.836532 4562 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b79e9a37-068d-4387-a07b-84b094e6895a" volumeName="kubernetes.io/projected/b79e9a37-068d-4387-a07b-84b094e6895a-kube-api-access-bpl7f" seLinuxMountContext="" Oct 30 08:53:59.841062 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.836551 4562 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bab893eb-ddd1-41b0-80a7-3c4907b7a8ad" volumeName="kubernetes.io/secret/bab893eb-ddd1-41b0-80a7-3c4907b7a8ad-signing-key" seLinuxMountContext="" Oct 30 08:53:59.841062 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.836563 4562 reconstruct.go:97] "Volume reconstruction finished" Oct 30 08:53:59.841062 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.836571 4562 reconciler.go:29] "Reconciler: start to sync state" Oct 30 08:53:59.841062 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.838964 4562 factory.go:153] Registering CRI-O factory Oct 30 08:53:59.841062 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.838978 4562 factory.go:223] Registration of the crio container factory successfully Oct 30 08:53:59.841062 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.839019 4562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 08:53:59.841062 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.839034 4562 factory.go:103] Registering Raw factory Oct 30 08:53:59.841062 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.839049 4562 manager.go:1199] Started watching for new ooms in manager Oct 30 08:53:59.841062 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.839625 4562 manager.go:319] Starting recovery of all containers Oct 30 08:53:59.860905 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.860869 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6q7n_1f51db5a-5720-4dea-ad02-f336b35d43b1/ovn-acl-logging/0.log" Oct 30 08:53:59.862503 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.862484 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6q7n_1f51db5a-5720-4dea-ad02-f336b35d43b1/ovn-acl-logging/0.log" Oct 30 08:53:59.862878 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.862770 4562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 08:53:59.866295 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.866268 4562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 08:53:59.866458 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.866298 4562 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 08:53:59.866458 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.866321 4562 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 08:53:59.866673 ip-10-0-143-244 kubenswrapper[4562]: E1030 08:53:59.866370 4562 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 08:53:59.871347 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.871322 4562 generic.go:356] "Generic (PLEG): container finished" podID="4bdabdb48827c33749d04f26c951dd86" containerID="eba4e2e85a1762338bc652c746c629ea5babefcbf4a7fb78490ce9c8471eef14" exitCode=0 Oct 30 08:53:59.882849 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.882822 4562 generic.go:356] "Generic (PLEG): container finished" podID="6f09a608-14be-447e-8e6e-eaf0a0031554" containerID="edb7694d185427d98a0674cdd2fcac444f6c9025917bb049e0014effea8b5b2c" exitCode=0 Oct 30 08:53:59.882950 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.882850 4562 generic.go:356] "Generic (PLEG): container finished" podID="6f09a608-14be-447e-8e6e-eaf0a0031554" containerID="91d841116d556c6ec4d9b3341cfd1f7675ce2b2da1d53365a8357011128fca52" exitCode=0 Oct 30 08:53:59.882950 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.882860 4562 generic.go:356] "Generic (PLEG): container finished" podID="6f09a608-14be-447e-8e6e-eaf0a0031554" containerID="90af3f9863d2e545aa8499e9a5902b3c59a667f6449935585f24e6e9e83f1321" exitCode=0 Oct 30 08:53:59.882950 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.882869 4562 generic.go:356] "Generic (PLEG): container finished" podID="6f09a608-14be-447e-8e6e-eaf0a0031554" containerID="c7eed2a753a92df048bef1cd2fdaa2c55d74d6a0b919fbaa4d5c6b801b0b04ab" exitCode=0 Oct 30 08:53:59.882950 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.882879 4562 generic.go:356] "Generic (PLEG): container finished" podID="6f09a608-14be-447e-8e6e-eaf0a0031554" containerID="1baa210418f0f6cf9ada0551c4e4448cc8d5811f3233d77f6434937d9c1013e9" exitCode=0 Oct 30 08:53:59.882950 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.882887 4562 generic.go:356] "Generic (PLEG): container finished" podID="6f09a608-14be-447e-8e6e-eaf0a0031554" containerID="d39760d59d325da5ae5b924fb1a3cfb60944b06ed908ff98091b141ecad3ec6e" exitCode=0 Oct 30 08:53:59.883181 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.882960 4562 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 08:53:59.883845 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.883827 4562 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 08:53:59.883939 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.883922 4562 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 08:53:59.896921 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.896901 4562 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 08:53:59.897089 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.897070 4562 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 08:53:59.897137 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.897109 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6q7n_1f51db5a-5720-4dea-ad02-f336b35d43b1/ovn-acl-logging/0.log" Oct 30 08:53:59.897504 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.897477 4562 generic.go:356] "Generic (PLEG): container finished" podID="1f51db5a-5720-4dea-ad02-f336b35d43b1" containerID="3cb37b1a8e3d1e357fede29d8044d2c67903697d6ce9fc3859603c9669368854" exitCode=1 Oct 30 08:53:59.913849 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.913830 4562 manager.go:324] Recovery completed Oct 30 08:53:59.927969 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.927948 4562 cpu_manager.go:226] "Starting" policy="none" Oct 30 08:53:59.927969 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.927961 4562 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 08:53:59.928110 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.927981 4562 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 08:53:59.928161 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.928119 4562 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 08:53:59.928161 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.928134 4562 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 08:53:59.928161 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.928156 4562 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 08:53:59.928303 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.928165 4562 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 08:53:59.928303 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.928172 4562 policy_none.go:50] "Start" Oct 30 08:53:59.928303 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.928184 4562 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 08:53:59.928303 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.928197 4562 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 08:53:59.928491 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.928374 4562 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 08:53:59.928491 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.928411 4562 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 08:53:59.928491 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.928420 4562 policy_none.go:44] "Start" Oct 30 08:53:59.932332 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.932307 4562 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 08:53:59.934696 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.934685 4562 manager.go:341] "Starting Device Plugin manager" Oct 30 08:53:59.935184 ip-10-0-143-244 kubenswrapper[4562]: E1030 08:53:59.934708 4562 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 08:53:59.935225 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.935188 4562 server.go:87] "Starting device plugin registration server" Oct 30 08:53:59.935454 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.935424 4562 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 08:53:59.935454 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.935436 4562 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 08:53:59.935570 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.935549 4562 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 08:53:59.935699 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.935655 4562 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 08:53:59.935796 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.935706 4562 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 08:53:59.935796 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.935713 4562 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 08:53:59.935890 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.935774 4562 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T08:53:59.935693457Z","UUID":"096678ae-beb7-44f7-88dd-c1bcdcc85efc","Handler":null,"Name":"","Endpoint":""} Oct 30 08:53:59.937010 ip-10-0-143-244 kubenswrapper[4562]: E1030 08:53:59.936989 4562 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 08:53:59.937210 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.937194 4562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 08:53:59.937291 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.937219 4562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 08:53:59.967261 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.967223 4562 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-244.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-244.ec2.internal"] Oct 30 08:53:59.967809 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.967766 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-244.ec2.internal" event={"ID":"4bdabdb48827c33749d04f26c951dd86","Type":"ContainerDied","Data":"eba4e2e85a1762338bc652c746c629ea5babefcbf4a7fb78490ce9c8471eef14"} Oct 30 08:53:59.967902 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.967830 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-244.ec2.internal" event={"ID":"4bdabdb48827c33749d04f26c951dd86","Type":"ContainerStarted","Data":"f1508eeaa257ef4ba2b2e4666b7f69913c95ef0b0b585df918cf1d1557008e0c"} Oct 30 08:53:59.967902 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.967845 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-244.ec2.internal" event={"ID":"e62eebd98344a1611517e7c60f1efe61","Type":"ContainerStarted","Data":"bb7c92bde4da25da139e27136c707e4d71f36fc96fae42f1c3c7b14c08b93c50"} Oct 30 08:53:59.967902 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.967859 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-244.ec2.internal" event={"ID":"e62eebd98344a1611517e7c60f1efe61","Type":"ContainerStarted","Data":"de620c47e9a6a59465581463393d917cb0141b362f379b7199f6dd337451c103"} Oct 30 08:53:59.968002 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.967846 4562 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-244.ec2.internal" Oct 30 08:53:59.968002 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.967976 4562 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-244.ec2.internal" Oct 30 08:53:59.990123 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.990104 4562 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 08:53:59.990189 ip-10-0-143-244 kubenswrapper[4562]: E1030 08:53:59.990140 4562 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-244.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-244.ec2.internal" Oct 30 08:53:59.990603 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:53:59.990591 4562 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 08:53:59.990655 ip-10-0-143-244 kubenswrapper[4562]: E1030 08:53:59.990622 4562 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-244.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-244.ec2.internal" Oct 30 08:54:00.035916 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.035898 4562 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 08:54:00.036910 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.036893 4562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.ec2.internal" event="NodeHasSufficientMemory" Oct 30 08:54:00.036988 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.036927 4562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 08:54:00.036988 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.036943 4562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-244.ec2.internal" event="NodeHasSufficientPID" Oct 30 08:54:00.036988 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.036979 4562 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-244.ec2.internal" Oct 30 08:54:00.072626 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.072603 4562 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-143-244.ec2.internal" Oct 30 08:54:00.072704 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.072655 4562 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-244.ec2.internal" Oct 30 08:54:00.085250 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.085232 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4bdabdb48827c33749d04f26c951dd86-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-244.ec2.internal\" (UID: \"4bdabdb48827c33749d04f26c951dd86\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-244.ec2.internal" Oct 30 08:54:00.085330 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.085261 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e62eebd98344a1611517e7c60f1efe61-config\") pod \"kube-apiserver-proxy-ip-10-0-143-244.ec2.internal\" (UID: \"e62eebd98344a1611517e7c60f1efe61\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-244.ec2.internal" Oct 30 08:54:00.085330 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.085284 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4bdabdb48827c33749d04f26c951dd86-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-244.ec2.internal\" (UID: \"4bdabdb48827c33749d04f26c951dd86\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-244.ec2.internal" Oct 30 08:54:00.186077 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.186023 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4bdabdb48827c33749d04f26c951dd86-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-244.ec2.internal\" (UID: \"4bdabdb48827c33749d04f26c951dd86\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-244.ec2.internal" Oct 30 08:54:00.186077 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.186055 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4bdabdb48827c33749d04f26c951dd86-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-244.ec2.internal\" (UID: \"4bdabdb48827c33749d04f26c951dd86\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-244.ec2.internal" Oct 30 08:54:00.186077 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.186072 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e62eebd98344a1611517e7c60f1efe61-config\") pod \"kube-apiserver-proxy-ip-10-0-143-244.ec2.internal\" (UID: \"e62eebd98344a1611517e7c60f1efe61\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-244.ec2.internal" Oct 30 08:54:00.186296 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.186094 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4bdabdb48827c33749d04f26c951dd86-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-244.ec2.internal\" (UID: \"4bdabdb48827c33749d04f26c951dd86\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-244.ec2.internal" Oct 30 08:54:00.186296 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.186123 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e62eebd98344a1611517e7c60f1efe61-config\") pod \"kube-apiserver-proxy-ip-10-0-143-244.ec2.internal\" (UID: \"e62eebd98344a1611517e7c60f1efe61\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-244.ec2.internal" Oct 30 08:54:00.186296 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.186139 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4bdabdb48827c33749d04f26c951dd86-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-244.ec2.internal\" (UID: \"4bdabdb48827c33749d04f26c951dd86\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-244.ec2.internal" Oct 30 08:54:00.823021 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.822988 4562 apiserver.go:52] "Watching apiserver" Oct 30 08:54:00.867258 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.867229 4562 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 08:54:00.867714 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.867694 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-nlv7n","openshift-multus/multus-rd9pd","openshift-service-ca/service-ca-5fffc97bdf-nf6sx","kube-system/konnectivity-agent-h5vr9","openshift-dns/dns-default-95kld","openshift-ingress-canary/ingress-canary-lmjss","openshift-multus/multus-additional-cni-plugins-v8qzv","openshift-cluster-node-tuning-operator/tuned-pvqcd","openshift-ovn-kubernetes/ovnkube-node-s6q7n","openshift-image-registry/node-ca-xqjlj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-244.ec2.internal","openshift-network-operator/iptables-alerter-6cncl","openshift-dns/node-resolver-lq2p2","openshift-network-diagnostics/network-check-target-mz2kl","kube-system/kube-apiserver-proxy-ip-10-0-143-244.ec2.internal","openshift-multus/network-metrics-daemon-z66mv","kube-system/global-pull-secret-syncer-vb26w","openshift-insights/insights-runtime-extractor-lc8jz","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6r2gg","openshift-insights/periodic-gathering-z7wg9-fvpzr"] Oct 30 08:54:00.868110 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.868091 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z66mv" Oct 30 08:54:00.875159 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.875135 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-95kld" Oct 30 08:54:00.875634 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.875616 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lc8jz" Oct 30 08:54:00.884264 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.884242 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 08:54:00.884602 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.884582 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-kgscj\"" Oct 30 08:54:00.884674 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.884585 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-r2tvf\"" Oct 30 08:54:00.884674 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.884638 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 08:54:00.884674 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.884649 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 08:54:00.884674 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.884663 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 08:54:00.884960 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.884939 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 08:54:00.886713 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.886695 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-gbg4x\"" Oct 30 08:54:00.886851 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.886774 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-twrgj\"" Oct 30 08:54:00.886851 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.886797 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-6v9sh\"" Oct 30 08:54:00.886851 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.886809 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 08:54:00.886851 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.886828 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 08:54:00.887004 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.886991 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 08:54:00.887035 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.887002 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 08:54:00.887537 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.887522 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 08:54:00.887592 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.887573 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 08:54:00.887621 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.887611 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 08:54:00.887661 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.887628 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-7pnl7\"" Oct 30 08:54:00.896160 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.896140 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 08:54:00.896259 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.896141 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 08:54:00.896259 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.896146 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 08:54:00.896259 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.896147 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 08:54:00.896358 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.896270 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-qhn6k\"" Oct 30 08:54:00.896521 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.896500 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pw6p6\"" Oct 30 08:54:00.897391 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.897364 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 08:54:00.897438 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.897398 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 08:54:00.897481 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.897465 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 08:54:00.897699 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.897672 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 08:54:00.897699 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.897676 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 08:54:00.897824 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.897810 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 08:54:00.897889 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.897838 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2jshw\"" Oct 30 08:54:00.897937 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.897908 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 08:54:00.898043 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.898014 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kd468\"" Oct 30 08:54:00.898222 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.898063 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 08:54:00.898222 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.898127 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 08:54:00.898222 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.898173 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-cnfjl\"" Oct 30 08:54:00.898427 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.898272 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 08:54:00.898427 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.898315 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 08:54:00.898427 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.898328 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 08:54:00.898427 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.898422 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-cb5mw\"" Oct 30 08:54:00.898597 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.898434 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 08:54:00.898597 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.898496 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 08:54:00.898714 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.898697 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 08:54:00.898950 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.898926 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 08:54:00.899064 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.898959 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 08:54:00.899298 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.899281 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 08:54:00.899362 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.899306 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 08:54:00.899362 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.899323 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 08:54:00.899362 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.899340 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 08:54:00.899566 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.899405 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wdt5m\"" Oct 30 08:54:00.899566 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.899489 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xqvv4\"" Oct 30 08:54:00.899566 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.899486 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-h5xkm\"" Oct 30 08:54:00.899566 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.899513 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 08:54:00.899566 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.899521 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 08:54:00.900157 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.900142 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 08:54:00.900287 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.900272 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 08:54:00.900354 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.900340 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-5px9g\"" Oct 30 08:54:00.900411 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.900276 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 08:54:00.900411 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.900367 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-nq2rg\"" Oct 30 08:54:00.900504 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.900416 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wqhvd\"" Oct 30 08:54:00.901205 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.901191 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 08:54:00.905642 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.905624 4562 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-244.ec2.internal" Oct 30 08:54:00.906129 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.906116 4562 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-244.ec2.internal" Oct 30 08:54:00.909921 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.909905 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 08:54:00.909971 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.909911 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 08:54:00.909971 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.909953 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 08:54:00.911991 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.911939 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 08:54:00.912343 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.912322 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 08:54:00.926750 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.926728 4562 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 08:54:00.926824 ip-10-0-143-244 kubenswrapper[4562]: E1030 08:54:00.926774 4562 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-244.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-244.ec2.internal" Oct 30 08:54:00.927052 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.927038 4562 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 08:54:00.927083 ip-10-0-143-244 kubenswrapper[4562]: E1030 08:54:00.927071 4562 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-244.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-244.ec2.internal" Oct 30 08:54:00.934162 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.934143 4562 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 08:54:00.989872 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.989851 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6f09a608-14be-447e-8e6e-eaf0a0031554-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v8qzv\" (UID: \"6f09a608-14be-447e-8e6e-eaf0a0031554\") " pod="openshift-multus/multus-additional-cni-plugins-v8qzv" Oct 30 08:54:00.989960 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.989883 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnszd\" (UniqueName: \"kubernetes.io/projected/54ef0e46-200b-4e7d-a007-ad959dec4486-kube-api-access-vnszd\") pod \"migrator-66df5d496b-nlv7n\" (UID: \"54ef0e46-200b-4e7d-a007-ad959dec4486\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nlv7n" Oct 30 08:54:00.989960 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.989909 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-host-run-netns\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:54:00.989960 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.989936 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2-service-ca-bundle\") pod \"periodic-gathering-z7wg9-fvpzr\" (UID: \"6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2\") " pod="openshift-insights/periodic-gathering-z7wg9-fvpzr" Oct 30 08:54:00.990095 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.989960 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dhlm\" (UniqueName: \"kubernetes.io/projected/bab893eb-ddd1-41b0-80a7-3c4907b7a8ad-kube-api-access-6dhlm\") pod \"service-ca-5fffc97bdf-nf6sx\" (UID: \"bab893eb-ddd1-41b0-80a7-3c4907b7a8ad\") " pod="openshift-service-ca/service-ca-5fffc97bdf-nf6sx" Oct 30 08:54:00.990095 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.990020 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b79e9a37-068d-4387-a07b-84b094e6895a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lc8jz\" (UID: \"b79e9a37-068d-4387-a07b-84b094e6895a\") " pod="openshift-insights/insights-runtime-extractor-lc8jz" Oct 30 08:54:00.990095 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.990059 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-systemd-units\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:54:00.990095 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.990086 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6f09a608-14be-447e-8e6e-eaf0a0031554-cnibin\") pod \"multus-additional-cni-plugins-v8qzv\" (UID: \"6f09a608-14be-447e-8e6e-eaf0a0031554\") " pod="openshift-multus/multus-additional-cni-plugins-v8qzv" Oct 30 08:54:00.990287 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.990116 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6f09a608-14be-447e-8e6e-eaf0a0031554-cni-binary-copy\") pod \"multus-additional-cni-plugins-v8qzv\" (UID: \"6f09a608-14be-447e-8e6e-eaf0a0031554\") " pod="openshift-multus/multus-additional-cni-plugins-v8qzv" Oct 30 08:54:00.990287 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.990145 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2e880cd8-58f9-41e4-8795-4f5e978e18f5-etc-kubernetes\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:54:00.990427 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.990278 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6f09a608-14be-447e-8e6e-eaf0a0031554-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v8qzv\" (UID: \"6f09a608-14be-447e-8e6e-eaf0a0031554\") " pod="openshift-multus/multus-additional-cni-plugins-v8qzv" Oct 30 08:54:00.990427 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.990342 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-host-var-lib-kubelet\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:54:00.990427 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.990370 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6f09a608-14be-447e-8e6e-eaf0a0031554-cni-binary-copy\") pod \"multus-additional-cni-plugins-v8qzv\" (UID: \"6f09a608-14be-447e-8e6e-eaf0a0031554\") " pod="openshift-multus/multus-additional-cni-plugins-v8qzv" Oct 30 08:54:00.990427 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.990404 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6d1fb12b-1a27-4574-be55-140884eb868a-multus-daemon-config\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:54:00.990613 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.990430 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6edea6bc-4807-48ff-b879-a77fc07d92de-config-volume\") pod \"dns-default-95kld\" (UID: \"6edea6bc-4807-48ff-b879-a77fc07d92de\") " pod="openshift-dns/dns-default-95kld" Oct 30 08:54:00.990613 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.990456 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-host-slash\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:54:00.990613 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.990484 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5e591466-fcf6-469d-9c3d-cde0767f4dde-kubelet-config\") pod \"global-pull-secret-syncer-vb26w\" (UID: \"5e591466-fcf6-469d-9c3d-cde0767f4dde\") " pod="kube-system/global-pull-secret-syncer-vb26w" Oct 30 08:54:00.990613 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.990518 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2e880cd8-58f9-41e4-8795-4f5e978e18f5-etc-sysctl-conf\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:54:00.990613 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.990599 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-host-run-k8s-cni-cncf-io\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:54:00.990845 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.990635 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2-service-ca-bundle\") pod \"periodic-gathering-z7wg9-fvpzr\" (UID: \"6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2\") " pod="openshift-insights/periodic-gathering-z7wg9-fvpzr" Oct 30 08:54:00.990845 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.990626 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-host-var-lib-cni-bin\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:54:00.990845 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.990691 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-host-var-lib-cni-multus\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:54:00.990845 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.990717 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6edea6bc-4807-48ff-b879-a77fc07d92de-config-volume\") pod \"dns-default-95kld\" (UID: \"6edea6bc-4807-48ff-b879-a77fc07d92de\") " pod="openshift-dns/dns-default-95kld" Oct 30 08:54:00.990845 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.990717 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6d1fb12b-1a27-4574-be55-140884eb868a-multus-daemon-config\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:54:00.990845 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.990741 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfhrk\" (UniqueName: \"kubernetes.io/projected/0278c964-1ae2-42e7-a5b3-1df265e86edc-kube-api-access-rfhrk\") pod \"network-check-target-mz2kl\" (UID: \"0278c964-1ae2-42e7-a5b3-1df265e86edc\") " pod="openshift-network-diagnostics/network-check-target-mz2kl" Oct 30 08:54:00.990845 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.990784 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-etc-openvswitch\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:54:00.991063 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.990887 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-host-cni-netd\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:54:00.991063 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.990922 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9c67a6b6-a3d8-49bc-a678-b203ef4201a9-hosts-file\") pod \"node-resolver-lq2p2\" (UID: \"9c67a6b6-a3d8-49bc-a678-b203ef4201a9\") " pod="openshift-dns/node-resolver-lq2p2" Oct 30 08:54:00.991063 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.990954 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmxcr\" (UniqueName: \"kubernetes.io/projected/9c67a6b6-a3d8-49bc-a678-b203ef4201a9-kube-api-access-rmxcr\") pod \"node-resolver-lq2p2\" (UID: \"9c67a6b6-a3d8-49bc-a678-b203ef4201a9\") " pod="openshift-dns/node-resolver-lq2p2" Oct 30 08:54:00.991063 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.990998 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3024e1a0-8ea6-4fc3-bb0f-e5997585eb86-iptables-alerter-script\") pod \"iptables-alerter-6cncl\" (UID: \"3024e1a0-8ea6-4fc3-bb0f-e5997585eb86\") " pod="openshift-network-operator/iptables-alerter-6cncl" Oct 30 08:54:00.991242 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.991082 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2e880cd8-58f9-41e4-8795-4f5e978e18f5-host\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:54:00.991242 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.991179 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e880cd8-58f9-41e4-8795-4f5e978e18f5-tmp\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:54:00.991242 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.991216 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-node-log\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:54:00.991423 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.991279 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e880cd8-58f9-41e4-8795-4f5e978e18f5-tmp\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:54:00.991423 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.991282 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-host-run-ovn-kubernetes\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:54:00.991423 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.991325 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1f51db5a-5720-4dea-ad02-f336b35d43b1-env-overrides\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:54:00.991423 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.991338 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3024e1a0-8ea6-4fc3-bb0f-e5997585eb86-iptables-alerter-script\") pod \"iptables-alerter-6cncl\" (UID: \"3024e1a0-8ea6-4fc3-bb0f-e5997585eb86\") " pod="openshift-network-operator/iptables-alerter-6cncl" Oct 30 08:54:00.991423 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.991360 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcnc5\" (UniqueName: \"kubernetes.io/projected/2e880cd8-58f9-41e4-8795-4f5e978e18f5-kube-api-access-vcnc5\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:54:00.991423 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.991414 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-system-cni-dir\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:54:00.991611 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.991438 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-cnibin\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:54:00.991611 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.991476 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-multus-socket-dir-parent\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:54:00.991611 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.991501 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-multus-conf-dir\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:54:00.991611 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.991527 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1f51db5a-5720-4dea-ad02-f336b35d43b1-env-overrides\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:54:00.991611 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.991526 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1f51db5a-5720-4dea-ad02-f336b35d43b1-ovn-node-metrics-cert\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:54:00.991742 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.991645 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/82a05e91-5a9d-4670-9882-c12ee332ca04-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6r2gg\" (UID: \"82a05e91-5a9d-4670-9882-c12ee332ca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6r2gg" Oct 30 08:54:00.991742 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.991672 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b79e9a37-068d-4387-a07b-84b094e6895a-data-volume\") pod \"insights-runtime-extractor-lc8jz\" (UID: \"b79e9a37-068d-4387-a07b-84b094e6895a\") " pod="openshift-insights/insights-runtime-extractor-lc8jz" Oct 30 08:54:00.991742 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.991687 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1f51db5a-5720-4dea-ad02-f336b35d43b1-ovn-node-metrics-cert\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:54:00.991742 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.991694 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-host-run-netns\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:54:00.991742 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.991728 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzmnn\" (UniqueName: \"kubernetes.io/projected/6f09a608-14be-447e-8e6e-eaf0a0031554-kube-api-access-nzmnn\") pod \"multus-additional-cni-plugins-v8qzv\" (UID: \"6f09a608-14be-447e-8e6e-eaf0a0031554\") " pod="openshift-multus/multus-additional-cni-plugins-v8qzv" Oct 30 08:54:00.991903 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.991755 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-etc-kubernetes\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:54:00.991903 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.991780 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b79e9a37-068d-4387-a07b-84b094e6895a-data-volume\") pod \"insights-runtime-extractor-lc8jz\" (UID: \"b79e9a37-068d-4387-a07b-84b094e6895a\") " pod="openshift-insights/insights-runtime-extractor-lc8jz" Oct 30 08:54:00.991903 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.991811 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9wwl\" (UniqueName: \"kubernetes.io/projected/6d1fb12b-1a27-4574-be55-140884eb868a-kube-api-access-m9wwl\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:54:00.991903 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.991836 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/bab893eb-ddd1-41b0-80a7-3c4907b7a8ad-signing-cabundle\") pod \"service-ca-5fffc97bdf-nf6sx\" (UID: \"bab893eb-ddd1-41b0-80a7-3c4907b7a8ad\") " pod="openshift-service-ca/service-ca-5fffc97bdf-nf6sx" Oct 30 08:54:00.991903 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.991884 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/82a05e91-5a9d-4670-9882-c12ee332ca04-device-dir\") pod \"aws-ebs-csi-driver-node-6r2gg\" (UID: \"82a05e91-5a9d-4670-9882-c12ee332ca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6r2gg" Oct 30 08:54:00.992032 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.991907 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/82a05e91-5a9d-4670-9882-c12ee332ca04-etc-selinux\") pod \"aws-ebs-csi-driver-node-6r2gg\" (UID: \"82a05e91-5a9d-4670-9882-c12ee332ca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6r2gg" Oct 30 08:54:00.992032 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.991931 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2e880cd8-58f9-41e4-8795-4f5e978e18f5-etc-sysctl-d\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:54:00.992032 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.991957 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-host-kubelet\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:54:00.992032 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.992002 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:54:00.992032 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.992029 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1f51db5a-5720-4dea-ad02-f336b35d43b1-ovnkube-config\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:54:00.992163 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.992055 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5679q\" (UniqueName: \"kubernetes.io/projected/1f51db5a-5720-4dea-ad02-f336b35d43b1-kube-api-access-5679q\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:54:00.992163 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.992078 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c67a6b6-a3d8-49bc-a678-b203ef4201a9-tmp-dir\") pod \"node-resolver-lq2p2\" (UID: \"9c67a6b6-a3d8-49bc-a678-b203ef4201a9\") " pod="openshift-dns/node-resolver-lq2p2" Oct 30 08:54:00.992163 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.992101 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6edea6bc-4807-48ff-b879-a77fc07d92de-metrics-tls\") pod \"dns-default-95kld\" (UID: \"6edea6bc-4807-48ff-b879-a77fc07d92de\") " pod="openshift-dns/dns-default-95kld" Oct 30 08:54:00.992163 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.992125 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/82a05e91-5a9d-4670-9882-c12ee332ca04-sys-fs\") pod \"aws-ebs-csi-driver-node-6r2gg\" (UID: \"82a05e91-5a9d-4670-9882-c12ee332ca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6r2gg" Oct 30 08:54:00.992323 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.992307 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1f51db5a-5720-4dea-ad02-f336b35d43b1-ovnkube-config\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:54:00.992353 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.992313 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/bab893eb-ddd1-41b0-80a7-3c4907b7a8ad-signing-cabundle\") pod \"service-ca-5fffc97bdf-nf6sx\" (UID: \"bab893eb-ddd1-41b0-80a7-3c4907b7a8ad\") " pod="openshift-service-ca/service-ca-5fffc97bdf-nf6sx" Oct 30 08:54:00.992536 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.992521 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c67a6b6-a3d8-49bc-a678-b203ef4201a9-tmp-dir\") pod \"node-resolver-lq2p2\" (UID: \"9c67a6b6-a3d8-49bc-a678-b203ef4201a9\") " pod="openshift-dns/node-resolver-lq2p2" Oct 30 08:54:00.992596 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.992577 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1f51db5a-5720-4dea-ad02-f336b35d43b1-ovnkube-script-lib\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:54:00.992633 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.992621 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6f09a608-14be-447e-8e6e-eaf0a0031554-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v8qzv\" (UID: \"6f09a608-14be-447e-8e6e-eaf0a0031554\") " pod="openshift-multus/multus-additional-cni-plugins-v8qzv" Oct 30 08:54:00.992750 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.992733 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6f09a608-14be-447e-8e6e-eaf0a0031554-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v8qzv\" (UID: \"6f09a608-14be-447e-8e6e-eaf0a0031554\") " pod="openshift-multus/multus-additional-cni-plugins-v8qzv" Oct 30 08:54:00.992827 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.992801 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1f51db5a-5720-4dea-ad02-f336b35d43b1-ovnkube-script-lib\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:54:00.992887 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.992815 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1b64ad2b-20eb-4a29-93b9-ccc9a217135d-serviceca\") pod \"node-ca-xqjlj\" (UID: \"1b64ad2b-20eb-4a29-93b9-ccc9a217135d\") " pod="openshift-image-registry/node-ca-xqjlj" Oct 30 08:54:00.992887 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.992864 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtm2w\" (UniqueName: \"kubernetes.io/projected/1b64ad2b-20eb-4a29-93b9-ccc9a217135d-kube-api-access-mtm2w\") pod \"node-ca-xqjlj\" (UID: \"1b64ad2b-20eb-4a29-93b9-ccc9a217135d\") " pod="openshift-image-registry/node-ca-xqjlj" Oct 30 08:54:00.992984 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.992902 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2-archives-path\") pod \"periodic-gathering-z7wg9-fvpzr\" (UID: \"6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2\") " pod="openshift-insights/periodic-gathering-z7wg9-fvpzr" Oct 30 08:54:00.992984 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.992943 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6f09a608-14be-447e-8e6e-eaf0a0031554-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v8qzv\" (UID: \"6f09a608-14be-447e-8e6e-eaf0a0031554\") " pod="openshift-multus/multus-additional-cni-plugins-v8qzv" Oct 30 08:54:00.993076 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.993013 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/bab893eb-ddd1-41b0-80a7-3c4907b7a8ad-signing-key\") pod \"service-ca-5fffc97bdf-nf6sx\" (UID: \"bab893eb-ddd1-41b0-80a7-3c4907b7a8ad\") " pod="openshift-service-ca/service-ca-5fffc97bdf-nf6sx" Oct 30 08:54:00.993076 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.993047 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/82a05e91-5a9d-4670-9882-c12ee332ca04-registration-dir\") pod \"aws-ebs-csi-driver-node-6r2gg\" (UID: \"82a05e91-5a9d-4670-9882-c12ee332ca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6r2gg" Oct 30 08:54:00.993076 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.993065 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2-archives-path\") pod \"periodic-gathering-z7wg9-fvpzr\" (UID: \"6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2\") " pod="openshift-insights/periodic-gathering-z7wg9-fvpzr" Oct 30 08:54:00.993216 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.993077 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-run-systemd\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:54:00.993216 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.993068 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1b64ad2b-20eb-4a29-93b9-ccc9a217135d-serviceca\") pod \"node-ca-xqjlj\" (UID: \"1b64ad2b-20eb-4a29-93b9-ccc9a217135d\") " pod="openshift-image-registry/node-ca-xqjlj" Oct 30 08:54:00.993216 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.993116 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6f09a608-14be-447e-8e6e-eaf0a0031554-system-cni-dir\") pod \"multus-additional-cni-plugins-v8qzv\" (UID: \"6f09a608-14be-447e-8e6e-eaf0a0031554\") " pod="openshift-multus/multus-additional-cni-plugins-v8qzv" Oct 30 08:54:00.993216 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.993196 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5d95\" (UniqueName: \"kubernetes.io/projected/3024e1a0-8ea6-4fc3-bb0f-e5997585eb86-kube-api-access-v5d95\") pod \"iptables-alerter-6cncl\" (UID: \"3024e1a0-8ea6-4fc3-bb0f-e5997585eb86\") " pod="openshift-network-operator/iptables-alerter-6cncl" Oct 30 08:54:00.993360 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.993253 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2e880cd8-58f9-41e4-8795-4f5e978e18f5-run\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:54:00.993360 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.993280 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/bab893eb-ddd1-41b0-80a7-3c4907b7a8ad-signing-key\") pod \"service-ca-5fffc97bdf-nf6sx\" (UID: \"bab893eb-ddd1-41b0-80a7-3c4907b7a8ad\") " pod="openshift-service-ca/service-ca-5fffc97bdf-nf6sx" Oct 30 08:54:00.993360 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.993334 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/634749a3-dbc0-41f4-9e97-bf3d25f9b6a8-agent-certs\") pod \"konnectivity-agent-h5vr9\" (UID: \"634749a3-dbc0-41f4-9e97-bf3d25f9b6a8\") " pod="kube-system/konnectivity-agent-h5vr9" Oct 30 08:54:00.993491 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.993362 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/634749a3-dbc0-41f4-9e97-bf3d25f9b6a8-konnectivity-ca\") pod \"konnectivity-agent-h5vr9\" (UID: \"634749a3-dbc0-41f4-9e97-bf3d25f9b6a8\") " pod="kube-system/konnectivity-agent-h5vr9" Oct 30 08:54:00.993491 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.993447 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpl7f\" (UniqueName: \"kubernetes.io/projected/b79e9a37-068d-4387-a07b-84b094e6895a-kube-api-access-bpl7f\") pod \"insights-runtime-extractor-lc8jz\" (UID: \"b79e9a37-068d-4387-a07b-84b094e6895a\") " pod="openshift-insights/insights-runtime-extractor-lc8jz" Oct 30 08:54:00.993491 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.993485 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-run-openvswitch\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:54:00.993637 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.993621 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1b64ad2b-20eb-4a29-93b9-ccc9a217135d-host\") pod \"node-ca-xqjlj\" (UID: \"1b64ad2b-20eb-4a29-93b9-ccc9a217135d\") " pod="openshift-image-registry/node-ca-xqjlj" Oct 30 08:54:00.993692 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.993667 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5e591466-fcf6-469d-9c3d-cde0767f4dde-original-pull-secret\") pod \"global-pull-secret-syncer-vb26w\" (UID: \"5e591466-fcf6-469d-9c3d-cde0767f4dde\") " pod="kube-system/global-pull-secret-syncer-vb26w" Oct 30 08:54:00.993743 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.993695 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2e880cd8-58f9-41e4-8795-4f5e978e18f5-etc-systemd\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:54:00.993818 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.993800 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/634749a3-dbc0-41f4-9e97-bf3d25f9b6a8-konnectivity-ca\") pod \"konnectivity-agent-h5vr9\" (UID: \"634749a3-dbc0-41f4-9e97-bf3d25f9b6a8\") " pod="kube-system/konnectivity-agent-h5vr9" Oct 30 08:54:00.993875 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.993801 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6d1fb12b-1a27-4574-be55-140884eb868a-cni-binary-copy\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:54:00.993875 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.993851 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-host-run-multus-certs\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:54:00.993875 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.993860 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/634749a3-dbc0-41f4-9e97-bf3d25f9b6a8-agent-certs\") pod \"konnectivity-agent-h5vr9\" (UID: \"634749a3-dbc0-41f4-9e97-bf3d25f9b6a8\") " pod="kube-system/konnectivity-agent-h5vr9" Oct 30 08:54:00.994015 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.993881 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8chm\" (UniqueName: \"kubernetes.io/projected/82a05e91-5a9d-4670-9882-c12ee332ca04-kube-api-access-s8chm\") pod \"aws-ebs-csi-driver-node-6r2gg\" (UID: \"82a05e91-5a9d-4670-9882-c12ee332ca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6r2gg" Oct 30 08:54:00.994015 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.993928 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-run-ovn\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:54:00.994015 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.993947 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6d1fb12b-1a27-4574-be55-140884eb868a-cni-binary-copy\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:54:00.994015 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.993956 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c94bc06-63d6-454f-9a56-1253c0315672-metrics-certs\") pod \"network-metrics-daemon-z66mv\" (UID: \"4c94bc06-63d6-454f-9a56-1253c0315672\") " pod="openshift-multus/network-metrics-daemon-z66mv" Oct 30 08:54:00.994015 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.993928 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5e591466-fcf6-469d-9c3d-cde0767f4dde-original-pull-secret\") pod \"global-pull-secret-syncer-vb26w\" (UID: \"5e591466-fcf6-469d-9c3d-cde0767f4dde\") " pod="kube-system/global-pull-secret-syncer-vb26w" Oct 30 08:54:00.994015 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.993984 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/deaa55d7-ac05-4aa1-909a-4237addf17ea-cert\") pod \"ingress-canary-lmjss\" (UID: \"deaa55d7-ac05-4aa1-909a-4237addf17ea\") " pod="openshift-ingress-canary/ingress-canary-lmjss" Oct 30 08:54:00.994265 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.994150 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b79e9a37-068d-4387-a07b-84b094e6895a-crio-socket\") pod \"insights-runtime-extractor-lc8jz\" (UID: \"b79e9a37-068d-4387-a07b-84b094e6895a\") " pod="openshift-insights/insights-runtime-extractor-lc8jz" Oct 30 08:54:00.994265 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.994184 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/deaa55d7-ac05-4aa1-909a-4237addf17ea-cert\") pod \"ingress-canary-lmjss\" (UID: \"deaa55d7-ac05-4aa1-909a-4237addf17ea\") " pod="openshift-ingress-canary/ingress-canary-lmjss" Oct 30 08:54:00.994265 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.994203 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3024e1a0-8ea6-4fc3-bb0f-e5997585eb86-host-slash\") pod \"iptables-alerter-6cncl\" (UID: \"3024e1a0-8ea6-4fc3-bb0f-e5997585eb86\") " pod="openshift-network-operator/iptables-alerter-6cncl" Oct 30 08:54:00.994265 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.994228 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2e880cd8-58f9-41e4-8795-4f5e978e18f5-etc-modprobe-d\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:54:00.994265 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.994254 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2e880cd8-58f9-41e4-8795-4f5e978e18f5-etc-sysconfig\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:54:00.994486 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.994275 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2e880cd8-58f9-41e4-8795-4f5e978e18f5-sys\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:54:00.994486 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.994315 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2e880cd8-58f9-41e4-8795-4f5e978e18f5-lib-modules\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:54:00.994486 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.994354 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2e880cd8-58f9-41e4-8795-4f5e978e18f5-etc-tuned\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:54:00.994486 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.994437 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6edea6bc-4807-48ff-b879-a77fc07d92de-tmp-dir\") pod \"dns-default-95kld\" (UID: \"6edea6bc-4807-48ff-b879-a77fc07d92de\") " pod="openshift-dns/dns-default-95kld" Oct 30 08:54:00.994486 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.994458 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2e880cd8-58f9-41e4-8795-4f5e978e18f5-etc-tuned\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:54:00.994681 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.994489 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6edea6bc-4807-48ff-b879-a77fc07d92de-tmp-dir\") pod \"dns-default-95kld\" (UID: \"6edea6bc-4807-48ff-b879-a77fc07d92de\") " pod="openshift-dns/dns-default-95kld" Oct 30 08:54:00.994681 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.994499 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8x4w\" (UniqueName: \"kubernetes.io/projected/4c94bc06-63d6-454f-9a56-1253c0315672-kube-api-access-x8x4w\") pod \"network-metrics-daemon-z66mv\" (UID: \"4c94bc06-63d6-454f-9a56-1253c0315672\") " pod="openshift-multus/network-metrics-daemon-z66mv" Oct 30 08:54:00.994681 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.994526 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5e591466-fcf6-469d-9c3d-cde0767f4dde-dbus\") pod \"global-pull-secret-syncer-vb26w\" (UID: \"5e591466-fcf6-469d-9c3d-cde0767f4dde\") " pod="kube-system/global-pull-secret-syncer-vb26w" Oct 30 08:54:00.994681 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.994554 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e880cd8-58f9-41e4-8795-4f5e978e18f5-var-lib-kubelet\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:54:00.994681 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.994587 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb9vs\" (UniqueName: \"kubernetes.io/projected/6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2-kube-api-access-tb9vs\") pod \"periodic-gathering-z7wg9-fvpzr\" (UID: \"6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2\") " pod="openshift-insights/periodic-gathering-z7wg9-fvpzr" Oct 30 08:54:00.994681 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.994633 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/82a05e91-5a9d-4670-9882-c12ee332ca04-socket-dir\") pod \"aws-ebs-csi-driver-node-6r2gg\" (UID: \"82a05e91-5a9d-4670-9882-c12ee332ca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6r2gg" Oct 30 08:54:00.994973 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.994730 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b79e9a37-068d-4387-a07b-84b094e6895a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lc8jz\" (UID: \"b79e9a37-068d-4387-a07b-84b094e6895a\") " pod="openshift-insights/insights-runtime-extractor-lc8jz" Oct 30 08:54:00.994973 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.994764 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-host-cni-bin\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:54:00.994973 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.994805 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-multus-cni-dir\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:54:00.994973 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.994831 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87k9j\" (UniqueName: \"kubernetes.io/projected/6edea6bc-4807-48ff-b879-a77fc07d92de-kube-api-access-87k9j\") pod \"dns-default-95kld\" (UID: \"6edea6bc-4807-48ff-b879-a77fc07d92de\") " pod="openshift-dns/dns-default-95kld" Oct 30 08:54:00.995160 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.995012 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2-trusted-ca-bundle\") pod \"periodic-gathering-z7wg9-fvpzr\" (UID: \"6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2\") " pod="openshift-insights/periodic-gathering-z7wg9-fvpzr" Oct 30 08:54:00.995160 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.995023 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b79e9a37-068d-4387-a07b-84b094e6895a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lc8jz\" (UID: \"b79e9a37-068d-4387-a07b-84b094e6895a\") " pod="openshift-insights/insights-runtime-extractor-lc8jz" Oct 30 08:54:00.995160 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.995039 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-log-socket\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:54:00.995160 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.995091 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-var-lib-openvswitch\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:54:00.995160 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.995136 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhzh5\" (UniqueName: \"kubernetes.io/projected/deaa55d7-ac05-4aa1-909a-4237addf17ea-kube-api-access-nhzh5\") pod \"ingress-canary-lmjss\" (UID: \"deaa55d7-ac05-4aa1-909a-4237addf17ea\") " pod="openshift-ingress-canary/ingress-canary-lmjss" Oct 30 08:54:00.995335 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.995172 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-hostroot\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:54:00.995670 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:00.995653 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2-trusted-ca-bundle\") pod \"periodic-gathering-z7wg9-fvpzr\" (UID: \"6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2\") " pod="openshift-insights/periodic-gathering-z7wg9-fvpzr" Oct 30 08:54:01.009819 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.009798 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnszd\" (UniqueName: \"kubernetes.io/projected/54ef0e46-200b-4e7d-a007-ad959dec4486-kube-api-access-vnszd\") pod \"migrator-66df5d496b-nlv7n\" (UID: \"54ef0e46-200b-4e7d-a007-ad959dec4486\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nlv7n" Oct 30 08:54:01.010272 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.010252 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dhlm\" (UniqueName: \"kubernetes.io/projected/bab893eb-ddd1-41b0-80a7-3c4907b7a8ad-kube-api-access-6dhlm\") pod \"service-ca-5fffc97bdf-nf6sx\" (UID: \"bab893eb-ddd1-41b0-80a7-3c4907b7a8ad\") " pod="openshift-service-ca/service-ca-5fffc97bdf-nf6sx" Oct 30 08:54:01.010915 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.010893 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtm2w\" (UniqueName: \"kubernetes.io/projected/1b64ad2b-20eb-4a29-93b9-ccc9a217135d-kube-api-access-mtm2w\") pod \"node-ca-xqjlj\" (UID: \"1b64ad2b-20eb-4a29-93b9-ccc9a217135d\") " pod="openshift-image-registry/node-ca-xqjlj" Oct 30 08:54:01.011095 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.011078 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5679q\" (UniqueName: \"kubernetes.io/projected/1f51db5a-5720-4dea-ad02-f336b35d43b1-kube-api-access-5679q\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:54:01.011268 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.011250 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcnc5\" (UniqueName: \"kubernetes.io/projected/2e880cd8-58f9-41e4-8795-4f5e978e18f5-kube-api-access-vcnc5\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:54:01.011343 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.011328 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9wwl\" (UniqueName: \"kubernetes.io/projected/6d1fb12b-1a27-4574-be55-140884eb868a-kube-api-access-m9wwl\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:54:01.012958 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.012938 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8x4w\" (UniqueName: \"kubernetes.io/projected/4c94bc06-63d6-454f-9a56-1253c0315672-kube-api-access-x8x4w\") pod \"network-metrics-daemon-z66mv\" (UID: \"4c94bc06-63d6-454f-9a56-1253c0315672\") " pod="openshift-multus/network-metrics-daemon-z66mv" Oct 30 08:54:01.013264 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.013229 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmxcr\" (UniqueName: \"kubernetes.io/projected/9c67a6b6-a3d8-49bc-a678-b203ef4201a9-kube-api-access-rmxcr\") pod \"node-resolver-lq2p2\" (UID: \"9c67a6b6-a3d8-49bc-a678-b203ef4201a9\") " pod="openshift-dns/node-resolver-lq2p2" Oct 30 08:54:01.013499 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.013480 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87k9j\" (UniqueName: \"kubernetes.io/projected/6edea6bc-4807-48ff-b879-a77fc07d92de-kube-api-access-87k9j\") pod \"dns-default-95kld\" (UID: \"6edea6bc-4807-48ff-b879-a77fc07d92de\") " pod="openshift-dns/dns-default-95kld" Oct 30 08:54:01.013580 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.013562 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb9vs\" (UniqueName: \"kubernetes.io/projected/6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2-kube-api-access-tb9vs\") pod \"periodic-gathering-z7wg9-fvpzr\" (UID: \"6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2\") " pod="openshift-insights/periodic-gathering-z7wg9-fvpzr" Oct 30 08:54:01.013797 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.013778 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpl7f\" (UniqueName: \"kubernetes.io/projected/b79e9a37-068d-4387-a07b-84b094e6895a-kube-api-access-bpl7f\") pod \"insights-runtime-extractor-lc8jz\" (UID: \"b79e9a37-068d-4387-a07b-84b094e6895a\") " pod="openshift-insights/insights-runtime-extractor-lc8jz" Oct 30 08:54:01.013831 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.013778 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5d95\" (UniqueName: \"kubernetes.io/projected/3024e1a0-8ea6-4fc3-bb0f-e5997585eb86-kube-api-access-v5d95\") pod \"iptables-alerter-6cncl\" (UID: \"3024e1a0-8ea6-4fc3-bb0f-e5997585eb86\") " pod="openshift-network-operator/iptables-alerter-6cncl" Oct 30 08:54:01.013919 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.013903 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhzh5\" (UniqueName: \"kubernetes.io/projected/deaa55d7-ac05-4aa1-909a-4237addf17ea-kube-api-access-nhzh5\") pod \"ingress-canary-lmjss\" (UID: \"deaa55d7-ac05-4aa1-909a-4237addf17ea\") " pod="openshift-ingress-canary/ingress-canary-lmjss" Oct 30 08:54:01.014141 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.014121 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfhrk\" (UniqueName: \"kubernetes.io/projected/0278c964-1ae2-42e7-a5b3-1df265e86edc-kube-api-access-rfhrk\") pod \"network-check-target-mz2kl\" (UID: \"0278c964-1ae2-42e7-a5b3-1df265e86edc\") " pod="openshift-network-diagnostics/network-check-target-mz2kl" Oct 30 08:54:01.014325 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.014309 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8chm\" (UniqueName: \"kubernetes.io/projected/82a05e91-5a9d-4670-9882-c12ee332ca04-kube-api-access-s8chm\") pod \"aws-ebs-csi-driver-node-6r2gg\" (UID: \"82a05e91-5a9d-4670-9882-c12ee332ca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6r2gg" Oct 30 08:54:01.014610 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.014590 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzmnn\" (UniqueName: \"kubernetes.io/projected/6f09a608-14be-447e-8e6e-eaf0a0031554-kube-api-access-nzmnn\") pod \"multus-additional-cni-plugins-v8qzv\" (UID: \"6f09a608-14be-447e-8e6e-eaf0a0031554\") " pod="openshift-multus/multus-additional-cni-plugins-v8qzv" Oct 30 08:54:01.096243 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.096185 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-host-slash\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:54:01.096243 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.096211 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5e591466-fcf6-469d-9c3d-cde0767f4dde-kubelet-config\") pod \"global-pull-secret-syncer-vb26w\" (UID: \"5e591466-fcf6-469d-9c3d-cde0767f4dde\") " pod="kube-system/global-pull-secret-syncer-vb26w" Oct 30 08:54:01.096243 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.096226 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2e880cd8-58f9-41e4-8795-4f5e978e18f5-etc-sysctl-conf\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:54:01.096243 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.096241 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-host-run-k8s-cni-cncf-io\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:54:01.096508 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.096255 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-host-var-lib-cni-bin\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:54:01.096508 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.096283 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-host-var-lib-cni-multus\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:54:01.096508 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.096298 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-etc-openvswitch\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:54:01.096508 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.096317 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5e591466-fcf6-469d-9c3d-cde0767f4dde-kubelet-config\") pod \"global-pull-secret-syncer-vb26w\" (UID: \"5e591466-fcf6-469d-9c3d-cde0767f4dde\") " pod="kube-system/global-pull-secret-syncer-vb26w" Oct 30 08:54:01.096508 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.096317 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-host-slash\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:54:01.096508 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.096321 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-host-cni-netd\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:54:01.096508 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.096354 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-host-cni-netd\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:54:01.096508 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.096357 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-host-var-lib-cni-bin\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:54:01.096508 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.096318 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-host-run-k8s-cni-cncf-io\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:54:01.096508 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.096367 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-etc-openvswitch\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:54:01.096508 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.096374 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9c67a6b6-a3d8-49bc-a678-b203ef4201a9-hosts-file\") pod \"node-resolver-lq2p2\" (UID: \"9c67a6b6-a3d8-49bc-a678-b203ef4201a9\") " pod="openshift-dns/node-resolver-lq2p2" Oct 30 08:54:01.096508 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.096420 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-host-var-lib-cni-multus\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:54:01.096508 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.096428 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2e880cd8-58f9-41e4-8795-4f5e978e18f5-host\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:54:01.096508 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.096449 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2e880cd8-58f9-41e4-8795-4f5e978e18f5-etc-sysctl-conf\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:54:01.096508 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.096450 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-node-log\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:54:01.096508 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.096469 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-node-log\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:54:01.096508 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.096450 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9c67a6b6-a3d8-49bc-a678-b203ef4201a9-hosts-file\") pod \"node-resolver-lq2p2\" (UID: \"9c67a6b6-a3d8-49bc-a678-b203ef4201a9\") " pod="openshift-dns/node-resolver-lq2p2" Oct 30 08:54:01.096508 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.096475 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-host-run-ovn-kubernetes\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:54:01.096508 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.096499 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2e880cd8-58f9-41e4-8795-4f5e978e18f5-host\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:54:01.096508 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.096502 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-host-run-ovn-kubernetes\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:54:01.097403 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.096525 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-system-cni-dir\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:54:01.097403 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.096549 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-cnibin\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:54:01.097403 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.096565 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-system-cni-dir\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:54:01.097403 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.096574 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-multus-socket-dir-parent\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:54:01.097403 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.096600 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-multus-conf-dir\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:54:01.097403 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.096610 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-cnibin\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:54:01.097403 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.096624 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/82a05e91-5a9d-4670-9882-c12ee332ca04-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6r2gg\" (UID: \"82a05e91-5a9d-4670-9882-c12ee332ca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6r2gg" Oct 30 08:54:01.097403 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.096636 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-multus-socket-dir-parent\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:54:01.097403 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.096649 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-host-run-netns\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:54:01.097403 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.096647 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-multus-conf-dir\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:54:01.097403 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.096670 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/82a05e91-5a9d-4670-9882-c12ee332ca04-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6r2gg\" (UID: \"82a05e91-5a9d-4670-9882-c12ee332ca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6r2gg" Oct 30 08:54:01.097403 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.096677 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-etc-kubernetes\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:54:01.097403 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.096694 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-host-run-netns\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:54:01.097403 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.096708 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/82a05e91-5a9d-4670-9882-c12ee332ca04-device-dir\") pod \"aws-ebs-csi-driver-node-6r2gg\" (UID: \"82a05e91-5a9d-4670-9882-c12ee332ca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6r2gg" Oct 30 08:54:01.097403 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.096732 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/82a05e91-5a9d-4670-9882-c12ee332ca04-etc-selinux\") pod \"aws-ebs-csi-driver-node-6r2gg\" (UID: \"82a05e91-5a9d-4670-9882-c12ee332ca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6r2gg" Oct 30 08:54:01.097403 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.096755 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2e880cd8-58f9-41e4-8795-4f5e978e18f5-etc-sysctl-d\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:54:01.097403 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.096779 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/82a05e91-5a9d-4670-9882-c12ee332ca04-device-dir\") pod \"aws-ebs-csi-driver-node-6r2gg\" (UID: \"82a05e91-5a9d-4670-9882-c12ee332ca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6r2gg" Oct 30 08:54:01.097403 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.096784 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/82a05e91-5a9d-4670-9882-c12ee332ca04-etc-selinux\") pod \"aws-ebs-csi-driver-node-6r2gg\" (UID: \"82a05e91-5a9d-4670-9882-c12ee332ca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6r2gg" Oct 30 08:54:01.097403 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.096781 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-host-kubelet\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:54:01.097403 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.096758 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-etc-kubernetes\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:54:01.097403 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.096820 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:54:01.097403 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.096840 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2e880cd8-58f9-41e4-8795-4f5e978e18f5-etc-sysctl-d\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:54:01.097403 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.096850 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6edea6bc-4807-48ff-b879-a77fc07d92de-metrics-tls\") pod \"dns-default-95kld\" (UID: \"6edea6bc-4807-48ff-b879-a77fc07d92de\") " pod="openshift-dns/dns-default-95kld" Oct 30 08:54:01.097403 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.096872 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/82a05e91-5a9d-4670-9882-c12ee332ca04-sys-fs\") pod \"aws-ebs-csi-driver-node-6r2gg\" (UID: \"82a05e91-5a9d-4670-9882-c12ee332ca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6r2gg" Oct 30 08:54:01.097403 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.096823 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-host-kubelet\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:54:01.097403 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.096897 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6f09a608-14be-447e-8e6e-eaf0a0031554-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v8qzv\" (UID: \"6f09a608-14be-447e-8e6e-eaf0a0031554\") " pod="openshift-multus/multus-additional-cni-plugins-v8qzv" Oct 30 08:54:01.097403 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.096924 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/82a05e91-5a9d-4670-9882-c12ee332ca04-registration-dir\") pod \"aws-ebs-csi-driver-node-6r2gg\" (UID: \"82a05e91-5a9d-4670-9882-c12ee332ca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6r2gg" Oct 30 08:54:01.097403 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.096930 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:54:01.097403 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.096944 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/82a05e91-5a9d-4670-9882-c12ee332ca04-sys-fs\") pod \"aws-ebs-csi-driver-node-6r2gg\" (UID: \"82a05e91-5a9d-4670-9882-c12ee332ca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6r2gg" Oct 30 08:54:01.097403 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.096947 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-run-systemd\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:54:01.097403 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.096966 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6f09a608-14be-447e-8e6e-eaf0a0031554-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v8qzv\" (UID: \"6f09a608-14be-447e-8e6e-eaf0a0031554\") " pod="openshift-multus/multus-additional-cni-plugins-v8qzv" Oct 30 08:54:01.097403 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.096978 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/82a05e91-5a9d-4670-9882-c12ee332ca04-registration-dir\") pod \"aws-ebs-csi-driver-node-6r2gg\" (UID: \"82a05e91-5a9d-4670-9882-c12ee332ca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6r2gg" Oct 30 08:54:01.097403 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.096982 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6f09a608-14be-447e-8e6e-eaf0a0031554-system-cni-dir\") pod \"multus-additional-cni-plugins-v8qzv\" (UID: \"6f09a608-14be-447e-8e6e-eaf0a0031554\") " pod="openshift-multus/multus-additional-cni-plugins-v8qzv" Oct 30 08:54:01.097403 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.097005 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2e880cd8-58f9-41e4-8795-4f5e978e18f5-run\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:54:01.097403 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.097007 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6f09a608-14be-447e-8e6e-eaf0a0031554-system-cni-dir\") pod \"multus-additional-cni-plugins-v8qzv\" (UID: \"6f09a608-14be-447e-8e6e-eaf0a0031554\") " pod="openshift-multus/multus-additional-cni-plugins-v8qzv" Oct 30 08:54:01.097403 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.097022 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-run-openvswitch\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:54:01.097403 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.097038 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1b64ad2b-20eb-4a29-93b9-ccc9a217135d-host\") pod \"node-ca-xqjlj\" (UID: \"1b64ad2b-20eb-4a29-93b9-ccc9a217135d\") " pod="openshift-image-registry/node-ca-xqjlj" Oct 30 08:54:01.097403 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.096979 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-run-systemd\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:54:01.097403 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.097067 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2e880cd8-58f9-41e4-8795-4f5e978e18f5-etc-systemd\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:54:01.097403 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.097074 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2e880cd8-58f9-41e4-8795-4f5e978e18f5-run\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:54:01.097403 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.097083 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1b64ad2b-20eb-4a29-93b9-ccc9a217135d-host\") pod \"node-ca-xqjlj\" (UID: \"1b64ad2b-20eb-4a29-93b9-ccc9a217135d\") " pod="openshift-image-registry/node-ca-xqjlj" Oct 30 08:54:01.097403 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.097085 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-run-openvswitch\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:54:01.097403 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.097092 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-host-run-multus-certs\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:54:01.097403 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.097129 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2e880cd8-58f9-41e4-8795-4f5e978e18f5-etc-systemd\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:54:01.097403 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.097132 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-host-run-multus-certs\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:54:01.097403 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.097150 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-run-ovn\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:54:01.097403 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.097175 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-run-ovn\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:54:01.097403 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.097174 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c94bc06-63d6-454f-9a56-1253c0315672-metrics-certs\") pod \"network-metrics-daemon-z66mv\" (UID: \"4c94bc06-63d6-454f-9a56-1253c0315672\") " pod="openshift-multus/network-metrics-daemon-z66mv" Oct 30 08:54:01.097403 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.097197 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b79e9a37-068d-4387-a07b-84b094e6895a-crio-socket\") pod \"insights-runtime-extractor-lc8jz\" (UID: \"b79e9a37-068d-4387-a07b-84b094e6895a\") " pod="openshift-insights/insights-runtime-extractor-lc8jz" Oct 30 08:54:01.097403 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.097221 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3024e1a0-8ea6-4fc3-bb0f-e5997585eb86-host-slash\") pod \"iptables-alerter-6cncl\" (UID: \"3024e1a0-8ea6-4fc3-bb0f-e5997585eb86\") " pod="openshift-network-operator/iptables-alerter-6cncl" Oct 30 08:54:01.097403 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.097236 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2e880cd8-58f9-41e4-8795-4f5e978e18f5-etc-modprobe-d\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:54:01.097403 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.097251 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b79e9a37-068d-4387-a07b-84b094e6895a-crio-socket\") pod \"insights-runtime-extractor-lc8jz\" (UID: \"b79e9a37-068d-4387-a07b-84b094e6895a\") " pod="openshift-insights/insights-runtime-extractor-lc8jz" Oct 30 08:54:01.097403 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.097238 4562 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 08:54:01.097403 ip-10-0-143-244 kubenswrapper[4562]: E1030 08:54:01.097253 4562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 08:54:01.097403 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.097295 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3024e1a0-8ea6-4fc3-bb0f-e5997585eb86-host-slash\") pod \"iptables-alerter-6cncl\" (UID: \"3024e1a0-8ea6-4fc3-bb0f-e5997585eb86\") " pod="openshift-network-operator/iptables-alerter-6cncl" Oct 30 08:54:01.097403 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.097318 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2e880cd8-58f9-41e4-8795-4f5e978e18f5-etc-sysconfig\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:54:01.097403 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.097267 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2e880cd8-58f9-41e4-8795-4f5e978e18f5-etc-sysconfig\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:54:01.097403 ip-10-0-143-244 kubenswrapper[4562]: E1030 08:54:01.097341 4562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c94bc06-63d6-454f-9a56-1253c0315672-metrics-certs podName:4c94bc06-63d6-454f-9a56-1253c0315672 nodeName:}" failed. No retries permitted until 2026-10-30 08:54:01.597322126 +0000 UTC m=+1.858883104 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4c94bc06-63d6-454f-9a56-1253c0315672-metrics-certs") pod "network-metrics-daemon-z66mv" (UID: "4c94bc06-63d6-454f-9a56-1253c0315672") : secret "metrics-daemon-secret" not found Oct 30 08:54:01.097403 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.097367 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2e880cd8-58f9-41e4-8795-4f5e978e18f5-sys\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:54:01.097403 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.097406 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2e880cd8-58f9-41e4-8795-4f5e978e18f5-lib-modules\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:54:01.097403 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.097433 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5e591466-fcf6-469d-9c3d-cde0767f4dde-dbus\") pod \"global-pull-secret-syncer-vb26w\" (UID: \"5e591466-fcf6-469d-9c3d-cde0767f4dde\") " pod="kube-system/global-pull-secret-syncer-vb26w" Oct 30 08:54:01.097403 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.097448 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2e880cd8-58f9-41e4-8795-4f5e978e18f5-sys\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:54:01.097403 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.097455 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e880cd8-58f9-41e4-8795-4f5e978e18f5-var-lib-kubelet\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:54:01.099478 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.097464 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2e880cd8-58f9-41e4-8795-4f5e978e18f5-etc-modprobe-d\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:54:01.099478 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.097492 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/82a05e91-5a9d-4670-9882-c12ee332ca04-socket-dir\") pod \"aws-ebs-csi-driver-node-6r2gg\" (UID: \"82a05e91-5a9d-4670-9882-c12ee332ca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6r2gg" Oct 30 08:54:01.099478 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.097518 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e880cd8-58f9-41e4-8795-4f5e978e18f5-var-lib-kubelet\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:54:01.099478 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.097523 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5e591466-fcf6-469d-9c3d-cde0767f4dde-dbus\") pod \"global-pull-secret-syncer-vb26w\" (UID: \"5e591466-fcf6-469d-9c3d-cde0767f4dde\") " pod="kube-system/global-pull-secret-syncer-vb26w" Oct 30 08:54:01.099478 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.097518 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-host-cni-bin\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:54:01.099478 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.097551 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-host-cni-bin\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:54:01.099478 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.097559 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-multus-cni-dir\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:54:01.099478 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.097570 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/82a05e91-5a9d-4670-9882-c12ee332ca04-socket-dir\") pod \"aws-ebs-csi-driver-node-6r2gg\" (UID: \"82a05e91-5a9d-4670-9882-c12ee332ca04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6r2gg" Oct 30 08:54:01.099478 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.097579 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2e880cd8-58f9-41e4-8795-4f5e978e18f5-lib-modules\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:54:01.099478 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.097583 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-log-socket\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:54:01.099478 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.097619 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-var-lib-openvswitch\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:54:01.099478 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.097624 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-log-socket\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:54:01.099478 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.097619 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-multus-cni-dir\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:54:01.099478 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.097651 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-hostroot\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:54:01.099478 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.097664 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-var-lib-openvswitch\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:54:01.099478 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.097670 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-host-run-netns\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:54:01.099478 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.097687 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-hostroot\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:54:01.099478 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.097696 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b79e9a37-068d-4387-a07b-84b094e6895a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lc8jz\" (UID: \"b79e9a37-068d-4387-a07b-84b094e6895a\") " pod="openshift-insights/insights-runtime-extractor-lc8jz" Oct 30 08:54:01.099478 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.097692 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-host-run-netns\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:54:01.099478 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.097719 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-systemd-units\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:54:01.099478 ip-10-0-143-244 kubenswrapper[4562]: E1030 08:54:01.097749 4562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 08:54:01.099478 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.097800 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6f09a608-14be-447e-8e6e-eaf0a0031554-cnibin\") pod \"multus-additional-cni-plugins-v8qzv\" (UID: \"6f09a608-14be-447e-8e6e-eaf0a0031554\") " pod="openshift-multus/multus-additional-cni-plugins-v8qzv" Oct 30 08:54:01.099478 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.097834 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2e880cd8-58f9-41e4-8795-4f5e978e18f5-etc-kubernetes\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:54:01.099478 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.097842 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1f51db5a-5720-4dea-ad02-f336b35d43b1-systemd-units\") pod \"ovnkube-node-s6q7n\" (UID: \"1f51db5a-5720-4dea-ad02-f336b35d43b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:54:01.099478 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.097866 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-host-var-lib-kubelet\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:54:01.099478 ip-10-0-143-244 kubenswrapper[4562]: E1030 08:54:01.097873 4562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b79e9a37-068d-4387-a07b-84b094e6895a-insights-runtime-extractor-tls podName:b79e9a37-068d-4387-a07b-84b094e6895a nodeName:}" failed. No retries permitted until 2026-10-30 08:54:01.59786267 +0000 UTC m=+1.859423652 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b79e9a37-068d-4387-a07b-84b094e6895a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lc8jz" (UID: "b79e9a37-068d-4387-a07b-84b094e6895a") : secret "insights-runtime-extractor-tls" not found Oct 30 08:54:01.099478 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.097889 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6f09a608-14be-447e-8e6e-eaf0a0031554-cnibin\") pod \"multus-additional-cni-plugins-v8qzv\" (UID: \"6f09a608-14be-447e-8e6e-eaf0a0031554\") " pod="openshift-multus/multus-additional-cni-plugins-v8qzv" Oct 30 08:54:01.099478 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.097918 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2e880cd8-58f9-41e4-8795-4f5e978e18f5-etc-kubernetes\") pod \"tuned-pvqcd\" (UID: \"2e880cd8-58f9-41e4-8795-4f5e978e18f5\") " pod="openshift-cluster-node-tuning-operator/tuned-pvqcd" Oct 30 08:54:01.099478 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.097924 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d1fb12b-1a27-4574-be55-140884eb868a-host-var-lib-kubelet\") pod \"multus-rd9pd\" (UID: \"6d1fb12b-1a27-4574-be55-140884eb868a\") " pod="openshift-multus/multus-rd9pd" Oct 30 08:54:01.099478 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.099426 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6edea6bc-4807-48ff-b879-a77fc07d92de-metrics-tls\") pod \"dns-default-95kld\" (UID: \"6edea6bc-4807-48ff-b879-a77fc07d92de\") " pod="openshift-dns/dns-default-95kld" Oct 30 08:54:01.175532 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.175512 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-95kld" Oct 30 08:54:01.491278 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.491246 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-95kld"] Oct 30 08:54:01.495579 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.495561 4562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 08:54:01.601947 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.601915 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c94bc06-63d6-454f-9a56-1253c0315672-metrics-certs\") pod \"network-metrics-daemon-z66mv\" (UID: \"4c94bc06-63d6-454f-9a56-1253c0315672\") " pod="openshift-multus/network-metrics-daemon-z66mv" Oct 30 08:54:01.602110 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.601958 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b79e9a37-068d-4387-a07b-84b094e6895a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lc8jz\" (UID: \"b79e9a37-068d-4387-a07b-84b094e6895a\") " pod="openshift-insights/insights-runtime-extractor-lc8jz" Oct 30 08:54:01.602110 ip-10-0-143-244 kubenswrapper[4562]: E1030 08:54:01.602063 4562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 08:54:01.602110 ip-10-0-143-244 kubenswrapper[4562]: E1030 08:54:01.602094 4562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 08:54:01.602234 ip-10-0-143-244 kubenswrapper[4562]: E1030 08:54:01.602114 4562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b79e9a37-068d-4387-a07b-84b094e6895a-insights-runtime-extractor-tls podName:b79e9a37-068d-4387-a07b-84b094e6895a nodeName:}" failed. No retries permitted until 2026-10-30 08:54:02.602099616 +0000 UTC m=+2.863660583 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b79e9a37-068d-4387-a07b-84b094e6895a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lc8jz" (UID: "b79e9a37-068d-4387-a07b-84b094e6895a") : secret "insights-runtime-extractor-tls" not found Oct 30 08:54:01.602234 ip-10-0-143-244 kubenswrapper[4562]: E1030 08:54:01.602158 4562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c94bc06-63d6-454f-9a56-1253c0315672-metrics-certs podName:4c94bc06-63d6-454f-9a56-1253c0315672 nodeName:}" failed. No retries permitted until 2026-10-30 08:54:02.602139198 +0000 UTC m=+2.863700164 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4c94bc06-63d6-454f-9a56-1253c0315672-metrics-certs") pod "network-metrics-daemon-z66mv" (UID: "4c94bc06-63d6-454f-9a56-1253c0315672") : secret "metrics-daemon-secret" not found Oct 30 08:54:01.908693 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:01.908617 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-95kld" event={"ID":"6edea6bc-4807-48ff-b879-a77fc07d92de","Type":"ContainerStarted","Data":"b159e9883fcb89f2fe21d4aec1f7a06c1da86ee3e9eac8aa20e766af10141998"} Oct 30 08:54:02.611038 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:02.610998 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c94bc06-63d6-454f-9a56-1253c0315672-metrics-certs\") pod \"network-metrics-daemon-z66mv\" (UID: \"4c94bc06-63d6-454f-9a56-1253c0315672\") " pod="openshift-multus/network-metrics-daemon-z66mv" Oct 30 08:54:02.611195 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:02.611057 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b79e9a37-068d-4387-a07b-84b094e6895a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lc8jz\" (UID: \"b79e9a37-068d-4387-a07b-84b094e6895a\") " pod="openshift-insights/insights-runtime-extractor-lc8jz" Oct 30 08:54:02.611195 ip-10-0-143-244 kubenswrapper[4562]: E1030 08:54:02.611154 4562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 08:54:02.611195 ip-10-0-143-244 kubenswrapper[4562]: E1030 08:54:02.611177 4562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 08:54:02.611370 ip-10-0-143-244 kubenswrapper[4562]: E1030 08:54:02.611231 4562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c94bc06-63d6-454f-9a56-1253c0315672-metrics-certs podName:4c94bc06-63d6-454f-9a56-1253c0315672 nodeName:}" failed. No retries permitted until 2026-10-30 08:54:04.611212267 +0000 UTC m=+4.872773252 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4c94bc06-63d6-454f-9a56-1253c0315672-metrics-certs") pod "network-metrics-daemon-z66mv" (UID: "4c94bc06-63d6-454f-9a56-1253c0315672") : secret "metrics-daemon-secret" not found Oct 30 08:54:02.611370 ip-10-0-143-244 kubenswrapper[4562]: E1030 08:54:02.611253 4562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b79e9a37-068d-4387-a07b-84b094e6895a-insights-runtime-extractor-tls podName:b79e9a37-068d-4387-a07b-84b094e6895a nodeName:}" failed. No retries permitted until 2026-10-30 08:54:04.611241378 +0000 UTC m=+4.872802347 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b79e9a37-068d-4387-a07b-84b094e6895a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lc8jz" (UID: "b79e9a37-068d-4387-a07b-84b094e6895a") : secret "insights-runtime-extractor-tls" not found Oct 30 08:54:03.915080 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:03.915046 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-95kld" event={"ID":"6edea6bc-4807-48ff-b879-a77fc07d92de","Type":"ContainerStarted","Data":"e5a07204913b72dcf63e30f221fb6e286db688464801420cb21a4d3979be33ca"} Oct 30 08:54:03.915581 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:03.915097 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-95kld" event={"ID":"6edea6bc-4807-48ff-b879-a77fc07d92de","Type":"ContainerStarted","Data":"b3f88e1211b6703141246cf787045752fcd4043bb926b5c13d2831281f41f404"} Oct 30 08:54:03.915581 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:03.915137 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-95kld" Oct 30 08:54:04.623238 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:04.623203 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b79e9a37-068d-4387-a07b-84b094e6895a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lc8jz\" (UID: \"b79e9a37-068d-4387-a07b-84b094e6895a\") " pod="openshift-insights/insights-runtime-extractor-lc8jz" Oct 30 08:54:04.623423 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:04.623262 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c94bc06-63d6-454f-9a56-1253c0315672-metrics-certs\") pod \"network-metrics-daemon-z66mv\" (UID: \"4c94bc06-63d6-454f-9a56-1253c0315672\") " pod="openshift-multus/network-metrics-daemon-z66mv" Oct 30 08:54:04.623423 ip-10-0-143-244 kubenswrapper[4562]: E1030 08:54:04.623359 4562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 08:54:04.623499 ip-10-0-143-244 kubenswrapper[4562]: E1030 08:54:04.623453 4562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b79e9a37-068d-4387-a07b-84b094e6895a-insights-runtime-extractor-tls podName:b79e9a37-068d-4387-a07b-84b094e6895a nodeName:}" failed. No retries permitted until 2026-10-30 08:54:08.623436444 +0000 UTC m=+8.884997414 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b79e9a37-068d-4387-a07b-84b094e6895a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lc8jz" (UID: "b79e9a37-068d-4387-a07b-84b094e6895a") : secret "insights-runtime-extractor-tls" not found Oct 30 08:54:04.625546 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:04.625524 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c94bc06-63d6-454f-9a56-1253c0315672-metrics-certs\") pod \"network-metrics-daemon-z66mv\" (UID: \"4c94bc06-63d6-454f-9a56-1253c0315672\") " pod="openshift-multus/network-metrics-daemon-z66mv" Oct 30 08:54:04.769083 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:04.769058 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z66mv" Oct 30 08:54:05.055834 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:05.055811 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-z66mv"] Oct 30 08:54:05.056996 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:54:05.056970 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c94bc06_63d6_454f_9a56_1253c0315672.slice/crio-522e79c3febe7e0fd6657b831932080eece586bdff4f5a0a6bfbdc2c23eb1f90 WatchSource:0}: Error finding container 522e79c3febe7e0fd6657b831932080eece586bdff4f5a0a6bfbdc2c23eb1f90: Status 404 returned error can't find the container with id 522e79c3febe7e0fd6657b831932080eece586bdff4f5a0a6bfbdc2c23eb1f90 Oct 30 08:54:05.522444 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:05.522411 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mz2kl" Oct 30 08:54:05.921779 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:05.921753 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-z66mv" event={"ID":"4c94bc06-63d6-454f-9a56-1253c0315672","Type":"ContainerStarted","Data":"522e79c3febe7e0fd6657b831932080eece586bdff4f5a0a6bfbdc2c23eb1f90"} Oct 30 08:54:06.926625 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:06.926595 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-z66mv" event={"ID":"4c94bc06-63d6-454f-9a56-1253c0315672","Type":"ContainerStarted","Data":"d8bd11cd1680017cdb475f6856c667e5bcd933786239f8beaea1ad616311d437"} Oct 30 08:54:06.927005 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:06.926642 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-z66mv" event={"ID":"4c94bc06-63d6-454f-9a56-1253c0315672","Type":"ContainerStarted","Data":"d27d22bb6d3e883009b11a8d0be571c9389942f55a5b46502ed952fc76795275"} Oct 30 08:54:07.477659 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:07.477627 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:54:08.648648 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:08.648611 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b79e9a37-068d-4387-a07b-84b094e6895a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lc8jz\" (UID: \"b79e9a37-068d-4387-a07b-84b094e6895a\") " pod="openshift-insights/insights-runtime-extractor-lc8jz" Oct 30 08:54:08.650983 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:08.650956 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b79e9a37-068d-4387-a07b-84b094e6895a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lc8jz\" (UID: \"b79e9a37-068d-4387-a07b-84b094e6895a\") " pod="openshift-insights/insights-runtime-extractor-lc8jz" Oct 30 08:54:08.675940 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:08.675904 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lc8jz" Oct 30 08:54:08.707118 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:08.707091 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:54:08.707313 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:08.707302 4562 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 08:54:08.963350 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:08.963330 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lc8jz"] Oct 30 08:54:08.965204 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:54:08.965174 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb79e9a37_068d_4387_a07b_84b094e6895a.slice/crio-c21b9d663d37881da6cd6a2eb48270dfcd620ce99769b005d17c8156754ce60e WatchSource:0}: Error finding container c21b9d663d37881da6cd6a2eb48270dfcd620ce99769b005d17c8156754ce60e: Status 404 returned error can't find the container with id c21b9d663d37881da6cd6a2eb48270dfcd620ce99769b005d17c8156754ce60e Oct 30 08:54:09.938276 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:09.938240 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lc8jz" event={"ID":"b79e9a37-068d-4387-a07b-84b094e6895a","Type":"ContainerStarted","Data":"1f256e69e57fd4d7a9d05b0cbac11a46046e6bfdde525598db926f1d8f9ec9c6"} Oct 30 08:54:09.938672 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:09.938306 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lc8jz" event={"ID":"b79e9a37-068d-4387-a07b-84b094e6895a","Type":"ContainerStarted","Data":"2dea66b0f35a051b5329c691bcebb4e809e1af7e15677bb2d6cf53f21ec158b2"} Oct 30 08:54:09.938672 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:09.938329 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lc8jz" event={"ID":"b79e9a37-068d-4387-a07b-84b094e6895a","Type":"ContainerStarted","Data":"c21b9d663d37881da6cd6a2eb48270dfcd620ce99769b005d17c8156754ce60e"} Oct 30 08:54:11.730526 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:11.730496 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-58cf89764b-plrvc"] Oct 30 08:54:11.732225 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:11.732211 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58cf89764b-plrvc" Oct 30 08:54:11.762368 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:11.762311 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 08:54:11.762573 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:11.762555 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 08:54:11.762881 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:11.762864 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-w9gxh\"" Oct 30 08:54:11.775895 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:11.775874 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 08:54:11.789222 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:11.789190 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 08:54:11.789810 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:11.789793 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-58cf89764b-plrvc"] Oct 30 08:54:11.868619 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:11.868594 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkbvd\" (UniqueName: \"kubernetes.io/projected/c209abe2-a298-4045-a241-5537b814167e-kube-api-access-bkbvd\") pod \"image-registry-58cf89764b-plrvc\" (UID: \"c209abe2-a298-4045-a241-5537b814167e\") " pod="openshift-image-registry/image-registry-58cf89764b-plrvc" Oct 30 08:54:11.868703 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:11.868629 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c209abe2-a298-4045-a241-5537b814167e-registry-tls\") pod \"image-registry-58cf89764b-plrvc\" (UID: \"c209abe2-a298-4045-a241-5537b814167e\") " pod="openshift-image-registry/image-registry-58cf89764b-plrvc" Oct 30 08:54:11.868703 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:11.868655 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c209abe2-a298-4045-a241-5537b814167e-ca-trust-extracted\") pod \"image-registry-58cf89764b-plrvc\" (UID: \"c209abe2-a298-4045-a241-5537b814167e\") " pod="openshift-image-registry/image-registry-58cf89764b-plrvc" Oct 30 08:54:11.868793 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:11.868767 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c209abe2-a298-4045-a241-5537b814167e-image-registry-private-configuration\") pod \"image-registry-58cf89764b-plrvc\" (UID: \"c209abe2-a298-4045-a241-5537b814167e\") " pod="openshift-image-registry/image-registry-58cf89764b-plrvc" Oct 30 08:54:11.868841 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:11.868812 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c209abe2-a298-4045-a241-5537b814167e-registry-certificates\") pod \"image-registry-58cf89764b-plrvc\" (UID: \"c209abe2-a298-4045-a241-5537b814167e\") " pod="openshift-image-registry/image-registry-58cf89764b-plrvc" Oct 30 08:54:11.868896 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:11.868841 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c209abe2-a298-4045-a241-5537b814167e-installation-pull-secrets\") pod \"image-registry-58cf89764b-plrvc\" (UID: \"c209abe2-a298-4045-a241-5537b814167e\") " pod="openshift-image-registry/image-registry-58cf89764b-plrvc" Oct 30 08:54:11.868960 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:11.868942 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c209abe2-a298-4045-a241-5537b814167e-trusted-ca\") pod \"image-registry-58cf89764b-plrvc\" (UID: \"c209abe2-a298-4045-a241-5537b814167e\") " pod="openshift-image-registry/image-registry-58cf89764b-plrvc" Oct 30 08:54:11.869017 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:11.868973 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c209abe2-a298-4045-a241-5537b814167e-bound-sa-token\") pod \"image-registry-58cf89764b-plrvc\" (UID: \"c209abe2-a298-4045-a241-5537b814167e\") " pod="openshift-image-registry/image-registry-58cf89764b-plrvc" Oct 30 08:54:11.951522 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:11.951493 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lc8jz" event={"ID":"b79e9a37-068d-4387-a07b-84b094e6895a","Type":"ContainerStarted","Data":"230ca8e58c6fdd324df333503753c254d2e19ebb17bd3a08095870e1ceff58c7"} Oct 30 08:54:11.969802 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:11.969779 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c209abe2-a298-4045-a241-5537b814167e-ca-trust-extracted\") pod \"image-registry-58cf89764b-plrvc\" (UID: \"c209abe2-a298-4045-a241-5537b814167e\") " pod="openshift-image-registry/image-registry-58cf89764b-plrvc" Oct 30 08:54:11.969892 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:11.969812 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c209abe2-a298-4045-a241-5537b814167e-image-registry-private-configuration\") pod \"image-registry-58cf89764b-plrvc\" (UID: \"c209abe2-a298-4045-a241-5537b814167e\") " pod="openshift-image-registry/image-registry-58cf89764b-plrvc" Oct 30 08:54:11.969892 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:11.969831 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c209abe2-a298-4045-a241-5537b814167e-registry-certificates\") pod \"image-registry-58cf89764b-plrvc\" (UID: \"c209abe2-a298-4045-a241-5537b814167e\") " pod="openshift-image-registry/image-registry-58cf89764b-plrvc" Oct 30 08:54:11.969892 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:11.969846 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c209abe2-a298-4045-a241-5537b814167e-installation-pull-secrets\") pod \"image-registry-58cf89764b-plrvc\" (UID: \"c209abe2-a298-4045-a241-5537b814167e\") " pod="openshift-image-registry/image-registry-58cf89764b-plrvc" Oct 30 08:54:11.970007 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:11.969956 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c209abe2-a298-4045-a241-5537b814167e-trusted-ca\") pod \"image-registry-58cf89764b-plrvc\" (UID: \"c209abe2-a298-4045-a241-5537b814167e\") " pod="openshift-image-registry/image-registry-58cf89764b-plrvc" Oct 30 08:54:11.970007 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:11.969980 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c209abe2-a298-4045-a241-5537b814167e-bound-sa-token\") pod \"image-registry-58cf89764b-plrvc\" (UID: \"c209abe2-a298-4045-a241-5537b814167e\") " pod="openshift-image-registry/image-registry-58cf89764b-plrvc" Oct 30 08:54:11.970309 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:11.970006 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkbvd\" (UniqueName: \"kubernetes.io/projected/c209abe2-a298-4045-a241-5537b814167e-kube-api-access-bkbvd\") pod \"image-registry-58cf89764b-plrvc\" (UID: \"c209abe2-a298-4045-a241-5537b814167e\") " pod="openshift-image-registry/image-registry-58cf89764b-plrvc" Oct 30 08:54:11.970309 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:11.970031 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c209abe2-a298-4045-a241-5537b814167e-registry-tls\") pod \"image-registry-58cf89764b-plrvc\" (UID: \"c209abe2-a298-4045-a241-5537b814167e\") " pod="openshift-image-registry/image-registry-58cf89764b-plrvc" Oct 30 08:54:11.970469 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:11.970302 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c209abe2-a298-4045-a241-5537b814167e-ca-trust-extracted\") pod \"image-registry-58cf89764b-plrvc\" (UID: \"c209abe2-a298-4045-a241-5537b814167e\") " pod="openshift-image-registry/image-registry-58cf89764b-plrvc" Oct 30 08:54:11.970919 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:11.970895 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c209abe2-a298-4045-a241-5537b814167e-registry-certificates\") pod \"image-registry-58cf89764b-plrvc\" (UID: \"c209abe2-a298-4045-a241-5537b814167e\") " pod="openshift-image-registry/image-registry-58cf89764b-plrvc" Oct 30 08:54:11.971156 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:11.971133 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c209abe2-a298-4045-a241-5537b814167e-trusted-ca\") pod \"image-registry-58cf89764b-plrvc\" (UID: \"c209abe2-a298-4045-a241-5537b814167e\") " pod="openshift-image-registry/image-registry-58cf89764b-plrvc" Oct 30 08:54:11.972352 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:11.972335 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c209abe2-a298-4045-a241-5537b814167e-image-registry-private-configuration\") pod \"image-registry-58cf89764b-plrvc\" (UID: \"c209abe2-a298-4045-a241-5537b814167e\") " pod="openshift-image-registry/image-registry-58cf89764b-plrvc" Oct 30 08:54:11.972437 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:11.972366 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c209abe2-a298-4045-a241-5537b814167e-installation-pull-secrets\") pod \"image-registry-58cf89764b-plrvc\" (UID: \"c209abe2-a298-4045-a241-5537b814167e\") " pod="openshift-image-registry/image-registry-58cf89764b-plrvc" Oct 30 08:54:11.972501 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:11.972489 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c209abe2-a298-4045-a241-5537b814167e-registry-tls\") pod \"image-registry-58cf89764b-plrvc\" (UID: \"c209abe2-a298-4045-a241-5537b814167e\") " pod="openshift-image-registry/image-registry-58cf89764b-plrvc" Oct 30 08:54:11.988923 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:11.988849 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c209abe2-a298-4045-a241-5537b814167e-bound-sa-token\") pod \"image-registry-58cf89764b-plrvc\" (UID: \"c209abe2-a298-4045-a241-5537b814167e\") " pod="openshift-image-registry/image-registry-58cf89764b-plrvc" Oct 30 08:54:11.989074 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:11.989058 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkbvd\" (UniqueName: \"kubernetes.io/projected/c209abe2-a298-4045-a241-5537b814167e-kube-api-access-bkbvd\") pod \"image-registry-58cf89764b-plrvc\" (UID: \"c209abe2-a298-4045-a241-5537b814167e\") " pod="openshift-image-registry/image-registry-58cf89764b-plrvc" Oct 30 08:54:12.041434 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:12.041408 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58cf89764b-plrvc" Oct 30 08:54:12.352846 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:12.352820 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-58cf89764b-plrvc"] Oct 30 08:54:12.354269 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:54:12.354243 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc209abe2_a298_4045_a241_5537b814167e.slice/crio-2a742df6a08ad8f8a9789418a4ac42a171ccf567d13508965d9d1b2fb7f15033 WatchSource:0}: Error finding container 2a742df6a08ad8f8a9789418a4ac42a171ccf567d13508965d9d1b2fb7f15033: Status 404 returned error can't find the container with id 2a742df6a08ad8f8a9789418a4ac42a171ccf567d13508965d9d1b2fb7f15033 Oct 30 08:54:12.956560 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:12.956527 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58cf89764b-plrvc" event={"ID":"c209abe2-a298-4045-a241-5537b814167e","Type":"ContainerStarted","Data":"a45957b18a43f2f09efb6ea318cb7758c5b631fdf7b229da469339e2dffec9c7"} Oct 30 08:54:12.956985 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:12.956582 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58cf89764b-plrvc" event={"ID":"c209abe2-a298-4045-a241-5537b814167e","Type":"ContainerStarted","Data":"2a742df6a08ad8f8a9789418a4ac42a171ccf567d13508965d9d1b2fb7f15033"} Oct 30 08:54:12.956985 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:12.956697 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-58cf89764b-plrvc" Oct 30 08:54:12.998147 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:12.998100 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-58cf89764b-plrvc" podStartSLOduration=1.9980860169999999 podStartE2EDuration="1.998086017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:54:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:54:12.997559394 +0000 UTC m=+13.259120379" watchObservedRunningTime="2026-10-30 08:54:12.998086017 +0000 UTC m=+13.259647004" Oct 30 08:54:13.920239 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:13.920209 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-95kld" Oct 30 08:54:16.298672 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:16.298640 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-dhnj7"] Oct 30 08:54:16.301965 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:16.301941 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-dhnj7" Oct 30 08:54:16.316862 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:16.316839 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-dhnj7"] Oct 30 08:54:16.329433 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:16.329410 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Oct 30 08:54:16.329545 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:16.329513 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-m8cs7\"" Oct 30 08:54:16.400860 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:16.400833 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1c3185f1-9fd0-4f9c-9ba5-6ba703f45cc6-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-dhnj7\" (UID: \"1c3185f1-9fd0-4f9c-9ba5-6ba703f45cc6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-dhnj7" Oct 30 08:54:16.501868 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:16.501837 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1c3185f1-9fd0-4f9c-9ba5-6ba703f45cc6-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-dhnj7\" (UID: \"1c3185f1-9fd0-4f9c-9ba5-6ba703f45cc6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-dhnj7" Oct 30 08:54:16.504247 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:16.504230 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1c3185f1-9fd0-4f9c-9ba5-6ba703f45cc6-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-dhnj7\" (UID: \"1c3185f1-9fd0-4f9c-9ba5-6ba703f45cc6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-dhnj7" Oct 30 08:54:16.612293 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:16.612223 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-dhnj7" Oct 30 08:54:16.740611 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:16.740582 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-h5vr9" Oct 30 08:54:16.898467 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:16.898443 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-dhnj7"] Oct 30 08:54:16.899294 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:54:16.899265 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c3185f1_9fd0_4f9c_9ba5_6ba703f45cc6.slice/crio-f48162627bb3eccae8e7690a68840273f0f47869ed5085023829cca114be2e20 WatchSource:0}: Error finding container f48162627bb3eccae8e7690a68840273f0f47869ed5085023829cca114be2e20: Status 404 returned error can't find the container with id f48162627bb3eccae8e7690a68840273f0f47869ed5085023829cca114be2e20 Oct 30 08:54:16.968988 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:16.968959 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-dhnj7" event={"ID":"1c3185f1-9fd0-4f9c-9ba5-6ba703f45cc6","Type":"ContainerStarted","Data":"f48162627bb3eccae8e7690a68840273f0f47869ed5085023829cca114be2e20"} Oct 30 08:54:17.973395 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:17.973348 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-dhnj7" event={"ID":"1c3185f1-9fd0-4f9c-9ba5-6ba703f45cc6","Type":"ContainerStarted","Data":"4f1348b10f953ad2b5615746297e2c89b44d3d89b658657b8865e40004e4a201"} Oct 30 08:54:17.973787 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:17.973540 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-dhnj7" Oct 30 08:54:17.978243 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:17.978224 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-dhnj7" Oct 30 08:54:18.012467 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:18.012426 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-dhnj7" podStartSLOduration=1.089647778 podStartE2EDuration="2.012415863s" totalImagesPullingTime="922.768085ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:54:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:54:18.011639182 +0000 UTC m=+18.273200170" watchObservedRunningTime="2026-10-30 08:54:18.012415863 +0000 UTC m=+18.273976850" Oct 30 08:54:23.108872 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:23.108843 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-xtw9h"] Oct 30 08:54:23.112338 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:23.112315 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xtw9h" Oct 30 08:54:23.130039 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:23.130014 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-jptkp\"" Oct 30 08:54:23.143170 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:23.143147 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 08:54:23.143170 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:23.143151 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 08:54:23.169030 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:23.169013 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 08:54:23.169119 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:23.169018 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 08:54:23.169172 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:23.169155 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 08:54:23.169204 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:23.169181 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 08:54:23.182395 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:23.182368 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s6q7n" Oct 30 08:54:23.249579 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:23.249550 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a9379d2d-3c28-4a90-9ee4-c4afed3b0306-node-exporter-textfile\") pod \"node-exporter-xtw9h\" (UID: \"a9379d2d-3c28-4a90-9ee4-c4afed3b0306\") " pod="openshift-monitoring/node-exporter-xtw9h" Oct 30 08:54:23.249723 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:23.249590 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a9379d2d-3c28-4a90-9ee4-c4afed3b0306-node-exporter-wtmp\") pod \"node-exporter-xtw9h\" (UID: \"a9379d2d-3c28-4a90-9ee4-c4afed3b0306\") " pod="openshift-monitoring/node-exporter-xtw9h" Oct 30 08:54:23.249723 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:23.249613 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9379d2d-3c28-4a90-9ee4-c4afed3b0306-metrics-client-ca\") pod \"node-exporter-xtw9h\" (UID: \"a9379d2d-3c28-4a90-9ee4-c4afed3b0306\") " pod="openshift-monitoring/node-exporter-xtw9h" Oct 30 08:54:23.249723 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:23.249670 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a9379d2d-3c28-4a90-9ee4-c4afed3b0306-node-exporter-accelerators-collector-config\") pod \"node-exporter-xtw9h\" (UID: \"a9379d2d-3c28-4a90-9ee4-c4afed3b0306\") " pod="openshift-monitoring/node-exporter-xtw9h" Oct 30 08:54:23.249862 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:23.249737 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a9379d2d-3c28-4a90-9ee4-c4afed3b0306-root\") pod \"node-exporter-xtw9h\" (UID: \"a9379d2d-3c28-4a90-9ee4-c4afed3b0306\") " pod="openshift-monitoring/node-exporter-xtw9h" Oct 30 08:54:23.249862 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:23.249753 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a9379d2d-3c28-4a90-9ee4-c4afed3b0306-node-exporter-tls\") pod \"node-exporter-xtw9h\" (UID: \"a9379d2d-3c28-4a90-9ee4-c4afed3b0306\") " pod="openshift-monitoring/node-exporter-xtw9h" Oct 30 08:54:23.249862 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:23.249773 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znncj\" (UniqueName: \"kubernetes.io/projected/a9379d2d-3c28-4a90-9ee4-c4afed3b0306-kube-api-access-znncj\") pod \"node-exporter-xtw9h\" (UID: \"a9379d2d-3c28-4a90-9ee4-c4afed3b0306\") " pod="openshift-monitoring/node-exporter-xtw9h" Oct 30 08:54:23.249862 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:23.249792 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a9379d2d-3c28-4a90-9ee4-c4afed3b0306-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xtw9h\" (UID: \"a9379d2d-3c28-4a90-9ee4-c4afed3b0306\") " pod="openshift-monitoring/node-exporter-xtw9h" Oct 30 08:54:23.250012 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:23.249866 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a9379d2d-3c28-4a90-9ee4-c4afed3b0306-sys\") pod \"node-exporter-xtw9h\" (UID: \"a9379d2d-3c28-4a90-9ee4-c4afed3b0306\") " pod="openshift-monitoring/node-exporter-xtw9h" Oct 30 08:54:23.351037 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:23.351009 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a9379d2d-3c28-4a90-9ee4-c4afed3b0306-sys\") pod \"node-exporter-xtw9h\" (UID: \"a9379d2d-3c28-4a90-9ee4-c4afed3b0306\") " pod="openshift-monitoring/node-exporter-xtw9h" Oct 30 08:54:23.351187 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:23.351059 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a9379d2d-3c28-4a90-9ee4-c4afed3b0306-node-exporter-textfile\") pod \"node-exporter-xtw9h\" (UID: \"a9379d2d-3c28-4a90-9ee4-c4afed3b0306\") " pod="openshift-monitoring/node-exporter-xtw9h" Oct 30 08:54:23.351187 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:23.351141 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a9379d2d-3c28-4a90-9ee4-c4afed3b0306-sys\") pod \"node-exporter-xtw9h\" (UID: \"a9379d2d-3c28-4a90-9ee4-c4afed3b0306\") " pod="openshift-monitoring/node-exporter-xtw9h" Oct 30 08:54:23.351316 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:23.351199 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a9379d2d-3c28-4a90-9ee4-c4afed3b0306-node-exporter-wtmp\") pod \"node-exporter-xtw9h\" (UID: \"a9379d2d-3c28-4a90-9ee4-c4afed3b0306\") " pod="openshift-monitoring/node-exporter-xtw9h" Oct 30 08:54:23.351316 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:23.351248 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9379d2d-3c28-4a90-9ee4-c4afed3b0306-metrics-client-ca\") pod \"node-exporter-xtw9h\" (UID: \"a9379d2d-3c28-4a90-9ee4-c4afed3b0306\") " pod="openshift-monitoring/node-exporter-xtw9h" Oct 30 08:54:23.351579 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:23.351562 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a9379d2d-3c28-4a90-9ee4-c4afed3b0306-node-exporter-accelerators-collector-config\") pod \"node-exporter-xtw9h\" (UID: \"a9379d2d-3c28-4a90-9ee4-c4afed3b0306\") " pod="openshift-monitoring/node-exporter-xtw9h" Oct 30 08:54:23.351630 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:23.351607 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a9379d2d-3c28-4a90-9ee4-c4afed3b0306-root\") pod \"node-exporter-xtw9h\" (UID: \"a9379d2d-3c28-4a90-9ee4-c4afed3b0306\") " pod="openshift-monitoring/node-exporter-xtw9h" Oct 30 08:54:23.351630 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:23.351471 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a9379d2d-3c28-4a90-9ee4-c4afed3b0306-node-exporter-textfile\") pod \"node-exporter-xtw9h\" (UID: \"a9379d2d-3c28-4a90-9ee4-c4afed3b0306\") " pod="openshift-monitoring/node-exporter-xtw9h" Oct 30 08:54:23.351630 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:23.351622 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a9379d2d-3c28-4a90-9ee4-c4afed3b0306-node-exporter-tls\") pod \"node-exporter-xtw9h\" (UID: \"a9379d2d-3c28-4a90-9ee4-c4afed3b0306\") " pod="openshift-monitoring/node-exporter-xtw9h" Oct 30 08:54:23.351775 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:23.351415 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a9379d2d-3c28-4a90-9ee4-c4afed3b0306-node-exporter-wtmp\") pod \"node-exporter-xtw9h\" (UID: \"a9379d2d-3c28-4a90-9ee4-c4afed3b0306\") " pod="openshift-monitoring/node-exporter-xtw9h" Oct 30 08:54:23.351775 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:23.351675 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znncj\" (UniqueName: \"kubernetes.io/projected/a9379d2d-3c28-4a90-9ee4-c4afed3b0306-kube-api-access-znncj\") pod \"node-exporter-xtw9h\" (UID: \"a9379d2d-3c28-4a90-9ee4-c4afed3b0306\") " pod="openshift-monitoring/node-exporter-xtw9h" Oct 30 08:54:23.351775 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:23.351704 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a9379d2d-3c28-4a90-9ee4-c4afed3b0306-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xtw9h\" (UID: \"a9379d2d-3c28-4a90-9ee4-c4afed3b0306\") " pod="openshift-monitoring/node-exporter-xtw9h" Oct 30 08:54:23.351966 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:23.351770 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9379d2d-3c28-4a90-9ee4-c4afed3b0306-metrics-client-ca\") pod \"node-exporter-xtw9h\" (UID: \"a9379d2d-3c28-4a90-9ee4-c4afed3b0306\") " pod="openshift-monitoring/node-exporter-xtw9h" Oct 30 08:54:23.351966 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:23.351833 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a9379d2d-3c28-4a90-9ee4-c4afed3b0306-root\") pod \"node-exporter-xtw9h\" (UID: \"a9379d2d-3c28-4a90-9ee4-c4afed3b0306\") " pod="openshift-monitoring/node-exporter-xtw9h" Oct 30 08:54:23.352252 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:23.352231 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a9379d2d-3c28-4a90-9ee4-c4afed3b0306-node-exporter-accelerators-collector-config\") pod \"node-exporter-xtw9h\" (UID: \"a9379d2d-3c28-4a90-9ee4-c4afed3b0306\") " pod="openshift-monitoring/node-exporter-xtw9h" Oct 30 08:54:23.353913 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:23.353882 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a9379d2d-3c28-4a90-9ee4-c4afed3b0306-node-exporter-tls\") pod \"node-exporter-xtw9h\" (UID: \"a9379d2d-3c28-4a90-9ee4-c4afed3b0306\") " pod="openshift-monitoring/node-exporter-xtw9h" Oct 30 08:54:23.354002 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:23.353925 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a9379d2d-3c28-4a90-9ee4-c4afed3b0306-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xtw9h\" (UID: \"a9379d2d-3c28-4a90-9ee4-c4afed3b0306\") " pod="openshift-monitoring/node-exporter-xtw9h" Oct 30 08:54:23.371054 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:23.371005 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znncj\" (UniqueName: \"kubernetes.io/projected/a9379d2d-3c28-4a90-9ee4-c4afed3b0306-kube-api-access-znncj\") pod \"node-exporter-xtw9h\" (UID: \"a9379d2d-3c28-4a90-9ee4-c4afed3b0306\") " pod="openshift-monitoring/node-exporter-xtw9h" Oct 30 08:54:23.428906 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:23.428877 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xtw9h" Oct 30 08:54:23.439868 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:54:23.439841 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9379d2d_3c28_4a90_9ee4_c4afed3b0306.slice/crio-7fcc0b868a067dd4d6b2cc73a81136afd88ac5c747d289d3949f666a400d6b30 WatchSource:0}: Error finding container 7fcc0b868a067dd4d6b2cc73a81136afd88ac5c747d289d3949f666a400d6b30: Status 404 returned error can't find the container with id 7fcc0b868a067dd4d6b2cc73a81136afd88ac5c747d289d3949f666a400d6b30 Oct 30 08:54:23.993676 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:23.993637 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xtw9h" event={"ID":"a9379d2d-3c28-4a90-9ee4-c4afed3b0306","Type":"ContainerStarted","Data":"7fcc0b868a067dd4d6b2cc73a81136afd88ac5c747d289d3949f666a400d6b30"} Oct 30 08:54:24.998691 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:24.998658 4562 generic.go:356] "Generic (PLEG): container finished" podID="a9379d2d-3c28-4a90-9ee4-c4afed3b0306" containerID="d43dc90b555c4025fc543da57532626f8e1713d4604aa87314bce773b938629b" exitCode=0 Oct 30 08:54:24.999058 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:24.998745 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xtw9h" event={"ID":"a9379d2d-3c28-4a90-9ee4-c4afed3b0306","Type":"ContainerDied","Data":"d43dc90b555c4025fc543da57532626f8e1713d4604aa87314bce773b938629b"} Oct 30 08:54:26.004605 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:26.004563 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xtw9h" event={"ID":"a9379d2d-3c28-4a90-9ee4-c4afed3b0306","Type":"ContainerStarted","Data":"3d5bad809f23764c7183eb775ebff48d9863d866325f713003e8fba632b9094c"} Oct 30 08:54:26.005046 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:26.004625 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xtw9h" event={"ID":"a9379d2d-3c28-4a90-9ee4-c4afed3b0306","Type":"ContainerStarted","Data":"6633077a0f3dd1007e0e5c3301a701a8bcc6b8e64090093c57694d4d3dcc6169"} Oct 30 08:54:26.074986 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:26.074938 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-xtw9h" podStartSLOduration=1.635375247 podStartE2EDuration="3.074925599s" totalImagesPullingTime="1.361389977s" totalInitContainerRuntime="78.160375ms" isStatefulPod=true podCreationTimestamp="2026-10-30 08:54:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:54:26.061874685 +0000 UTC m=+26.323435672" watchObservedRunningTime="2026-10-30 08:54:26.074925599 +0000 UTC m=+26.336486603" Oct 30 08:54:27.557377 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:27.557339 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-bbf8d8c5f-dxdml"] Oct 30 08:54:27.583037 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:27.583009 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-bbf8d8c5f-dxdml"] Oct 30 08:54:27.583172 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:27.583096 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-bbf8d8c5f-dxdml" Oct 30 08:54:27.598829 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:27.598802 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 08:54:27.611809 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:27.611790 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 08:54:27.611894 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:27.611851 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-knvh2\"" Oct 30 08:54:27.611991 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:27.611977 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 08:54:27.612045 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:27.612034 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 08:54:27.612131 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:27.612070 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-2vala520bk5ci\"" Oct 30 08:54:27.678804 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:27.678777 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e6056d7b-f80f-4579-9347-1a3029facddd-secret-metrics-server-client-certs\") pod \"metrics-server-bbf8d8c5f-dxdml\" (UID: \"e6056d7b-f80f-4579-9347-1a3029facddd\") " pod="openshift-monitoring/metrics-server-bbf8d8c5f-dxdml" Oct 30 08:54:27.678947 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:27.678807 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e6056d7b-f80f-4579-9347-1a3029facddd-metrics-server-audit-profiles\") pod \"metrics-server-bbf8d8c5f-dxdml\" (UID: \"e6056d7b-f80f-4579-9347-1a3029facddd\") " pod="openshift-monitoring/metrics-server-bbf8d8c5f-dxdml" Oct 30 08:54:27.678947 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:27.678839 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e6056d7b-f80f-4579-9347-1a3029facddd-audit-log\") pod \"metrics-server-bbf8d8c5f-dxdml\" (UID: \"e6056d7b-f80f-4579-9347-1a3029facddd\") " pod="openshift-monitoring/metrics-server-bbf8d8c5f-dxdml" Oct 30 08:54:27.679046 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:27.678955 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmtk6\" (UniqueName: \"kubernetes.io/projected/e6056d7b-f80f-4579-9347-1a3029facddd-kube-api-access-qmtk6\") pod \"metrics-server-bbf8d8c5f-dxdml\" (UID: \"e6056d7b-f80f-4579-9347-1a3029facddd\") " pod="openshift-monitoring/metrics-server-bbf8d8c5f-dxdml" Oct 30 08:54:27.679046 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:27.678978 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6056d7b-f80f-4579-9347-1a3029facddd-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-bbf8d8c5f-dxdml\" (UID: \"e6056d7b-f80f-4579-9347-1a3029facddd\") " pod="openshift-monitoring/metrics-server-bbf8d8c5f-dxdml" Oct 30 08:54:27.679046 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:27.678994 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e6056d7b-f80f-4579-9347-1a3029facddd-client-ca-bundle\") pod \"metrics-server-bbf8d8c5f-dxdml\" (UID: \"e6056d7b-f80f-4579-9347-1a3029facddd\") " pod="openshift-monitoring/metrics-server-bbf8d8c5f-dxdml" Oct 30 08:54:27.679046 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:27.679019 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e6056d7b-f80f-4579-9347-1a3029facddd-secret-metrics-server-tls\") pod \"metrics-server-bbf8d8c5f-dxdml\" (UID: \"e6056d7b-f80f-4579-9347-1a3029facddd\") " pod="openshift-monitoring/metrics-server-bbf8d8c5f-dxdml" Oct 30 08:54:27.780115 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:27.780086 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e6056d7b-f80f-4579-9347-1a3029facddd-secret-metrics-server-client-certs\") pod \"metrics-server-bbf8d8c5f-dxdml\" (UID: \"e6056d7b-f80f-4579-9347-1a3029facddd\") " pod="openshift-monitoring/metrics-server-bbf8d8c5f-dxdml" Oct 30 08:54:27.780213 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:27.780120 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e6056d7b-f80f-4579-9347-1a3029facddd-metrics-server-audit-profiles\") pod \"metrics-server-bbf8d8c5f-dxdml\" (UID: \"e6056d7b-f80f-4579-9347-1a3029facddd\") " pod="openshift-monitoring/metrics-server-bbf8d8c5f-dxdml" Oct 30 08:54:27.780213 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:27.780144 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e6056d7b-f80f-4579-9347-1a3029facddd-audit-log\") pod \"metrics-server-bbf8d8c5f-dxdml\" (UID: \"e6056d7b-f80f-4579-9347-1a3029facddd\") " pod="openshift-monitoring/metrics-server-bbf8d8c5f-dxdml" Oct 30 08:54:27.780213 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:27.780168 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmtk6\" (UniqueName: \"kubernetes.io/projected/e6056d7b-f80f-4579-9347-1a3029facddd-kube-api-access-qmtk6\") pod \"metrics-server-bbf8d8c5f-dxdml\" (UID: \"e6056d7b-f80f-4579-9347-1a3029facddd\") " pod="openshift-monitoring/metrics-server-bbf8d8c5f-dxdml" Oct 30 08:54:27.780213 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:27.780187 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6056d7b-f80f-4579-9347-1a3029facddd-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-bbf8d8c5f-dxdml\" (UID: \"e6056d7b-f80f-4579-9347-1a3029facddd\") " pod="openshift-monitoring/metrics-server-bbf8d8c5f-dxdml" Oct 30 08:54:27.780419 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:27.780293 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e6056d7b-f80f-4579-9347-1a3029facddd-client-ca-bundle\") pod \"metrics-server-bbf8d8c5f-dxdml\" (UID: \"e6056d7b-f80f-4579-9347-1a3029facddd\") " pod="openshift-monitoring/metrics-server-bbf8d8c5f-dxdml" Oct 30 08:54:27.780419 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:27.780327 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e6056d7b-f80f-4579-9347-1a3029facddd-secret-metrics-server-tls\") pod \"metrics-server-bbf8d8c5f-dxdml\" (UID: \"e6056d7b-f80f-4579-9347-1a3029facddd\") " pod="openshift-monitoring/metrics-server-bbf8d8c5f-dxdml" Oct 30 08:54:27.780670 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:27.780649 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e6056d7b-f80f-4579-9347-1a3029facddd-audit-log\") pod \"metrics-server-bbf8d8c5f-dxdml\" (UID: \"e6056d7b-f80f-4579-9347-1a3029facddd\") " pod="openshift-monitoring/metrics-server-bbf8d8c5f-dxdml" Oct 30 08:54:27.781053 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:27.781029 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6056d7b-f80f-4579-9347-1a3029facddd-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-bbf8d8c5f-dxdml\" (UID: \"e6056d7b-f80f-4579-9347-1a3029facddd\") " pod="openshift-monitoring/metrics-server-bbf8d8c5f-dxdml" Oct 30 08:54:27.781213 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:27.781194 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e6056d7b-f80f-4579-9347-1a3029facddd-metrics-server-audit-profiles\") pod \"metrics-server-bbf8d8c5f-dxdml\" (UID: \"e6056d7b-f80f-4579-9347-1a3029facddd\") " pod="openshift-monitoring/metrics-server-bbf8d8c5f-dxdml" Oct 30 08:54:27.782549 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:27.782530 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e6056d7b-f80f-4579-9347-1a3029facddd-secret-metrics-server-client-certs\") pod \"metrics-server-bbf8d8c5f-dxdml\" (UID: \"e6056d7b-f80f-4579-9347-1a3029facddd\") " pod="openshift-monitoring/metrics-server-bbf8d8c5f-dxdml" Oct 30 08:54:27.782707 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:27.782691 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e6056d7b-f80f-4579-9347-1a3029facddd-client-ca-bundle\") pod \"metrics-server-bbf8d8c5f-dxdml\" (UID: \"e6056d7b-f80f-4579-9347-1a3029facddd\") " pod="openshift-monitoring/metrics-server-bbf8d8c5f-dxdml" Oct 30 08:54:27.782845 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:27.782824 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e6056d7b-f80f-4579-9347-1a3029facddd-secret-metrics-server-tls\") pod \"metrics-server-bbf8d8c5f-dxdml\" (UID: \"e6056d7b-f80f-4579-9347-1a3029facddd\") " pod="openshift-monitoring/metrics-server-bbf8d8c5f-dxdml" Oct 30 08:54:27.799569 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:27.799542 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmtk6\" (UniqueName: \"kubernetes.io/projected/e6056d7b-f80f-4579-9347-1a3029facddd-kube-api-access-qmtk6\") pod \"metrics-server-bbf8d8c5f-dxdml\" (UID: \"e6056d7b-f80f-4579-9347-1a3029facddd\") " pod="openshift-monitoring/metrics-server-bbf8d8c5f-dxdml" Oct 30 08:54:27.905252 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:27.905177 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-bbf8d8c5f-dxdml" Oct 30 08:54:27.913948 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:27.913925 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-6vp95"] Oct 30 08:54:27.945725 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:27.945704 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-6vp95"] Oct 30 08:54:27.945836 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:27.945822 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-6vp95" Oct 30 08:54:27.961418 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:27.961393 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 08:54:27.961418 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:27.961411 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-jndgs\"" Oct 30 08:54:28.082769 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:28.082731 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq4wj\" (UniqueName: \"kubernetes.io/projected/07ca6f17-5369-4d89-94b7-1f510621475a-kube-api-access-lq4wj\") pod \"monitoring-plugin-77f4f76594-6vp95\" (UID: \"07ca6f17-5369-4d89-94b7-1f510621475a\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-6vp95" Oct 30 08:54:28.082932 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:28.082779 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/07ca6f17-5369-4d89-94b7-1f510621475a-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-6vp95\" (UID: \"07ca6f17-5369-4d89-94b7-1f510621475a\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-6vp95" Oct 30 08:54:28.184041 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:28.183972 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq4wj\" (UniqueName: \"kubernetes.io/projected/07ca6f17-5369-4d89-94b7-1f510621475a-kube-api-access-lq4wj\") pod \"monitoring-plugin-77f4f76594-6vp95\" (UID: \"07ca6f17-5369-4d89-94b7-1f510621475a\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-6vp95" Oct 30 08:54:28.184041 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:28.184011 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/07ca6f17-5369-4d89-94b7-1f510621475a-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-6vp95\" (UID: \"07ca6f17-5369-4d89-94b7-1f510621475a\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-6vp95" Oct 30 08:54:28.186750 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:28.186728 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/07ca6f17-5369-4d89-94b7-1f510621475a-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-6vp95\" (UID: \"07ca6f17-5369-4d89-94b7-1f510621475a\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-6vp95" Oct 30 08:54:28.203742 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:28.203717 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq4wj\" (UniqueName: \"kubernetes.io/projected/07ca6f17-5369-4d89-94b7-1f510621475a-kube-api-access-lq4wj\") pod \"monitoring-plugin-77f4f76594-6vp95\" (UID: \"07ca6f17-5369-4d89-94b7-1f510621475a\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-6vp95" Oct 30 08:54:28.234339 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:28.234319 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-bbf8d8c5f-dxdml"] Oct 30 08:54:28.234941 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:54:28.234919 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6056d7b_f80f_4579_9347_1a3029facddd.slice/crio-fc95599d88532590df0af100af93decaf664b4c9d8e184833d55434dd3b717e3 WatchSource:0}: Error finding container fc95599d88532590df0af100af93decaf664b4c9d8e184833d55434dd3b717e3: Status 404 returned error can't find the container with id fc95599d88532590df0af100af93decaf664b4c9d8e184833d55434dd3b717e3 Oct 30 08:54:28.280603 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:28.280582 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-6vp95" Oct 30 08:54:28.593899 ip-10-0-143-244 kubenswrapper[4562]: E1030 08:54:28.593852 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 08:54:28.597003 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:28.596974 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-6vp95"] Oct 30 08:54:28.598005 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:54:28.597980 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07ca6f17_5369_4d89_94b7_1f510621475a.slice/crio-bf33aec6fdc2a4c843a8706939f1a451eb706a9f68bc3eccf3f451a16da81e43 WatchSource:0}: Error finding container bf33aec6fdc2a4c843a8706939f1a451eb706a9f68bc3eccf3f451a16da81e43: Status 404 returned error can't find the container with id bf33aec6fdc2a4c843a8706939f1a451eb706a9f68bc3eccf3f451a16da81e43 Oct 30 08:54:29.016704 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:29.016670 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-bbf8d8c5f-dxdml" event={"ID":"e6056d7b-f80f-4579-9347-1a3029facddd","Type":"ContainerStarted","Data":"fc95599d88532590df0af100af93decaf664b4c9d8e184833d55434dd3b717e3"} Oct 30 08:54:29.018741 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:29.018709 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-6vp95" event={"ID":"07ca6f17-5369-4d89-94b7-1f510621475a","Type":"ContainerStarted","Data":"bf33aec6fdc2a4c843a8706939f1a451eb706a9f68bc3eccf3f451a16da81e43"} Oct 30 08:54:31.028333 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:31.028302 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-6vp95" event={"ID":"07ca6f17-5369-4d89-94b7-1f510621475a","Type":"ContainerStarted","Data":"cfa360658e32f94dc01b2301a2108f0d9a04b4d709527caf90810339a0740dff"} Oct 30 08:54:31.028788 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:31.028516 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-6vp95" Oct 30 08:54:31.030099 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:31.030069 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-bbf8d8c5f-dxdml" event={"ID":"e6056d7b-f80f-4579-9347-1a3029facddd","Type":"ContainerStarted","Data":"05866fd1ff99c4207cd6ba1d49d6fa28603f7e047fea0a37fd9596c035ce9101"} Oct 30 08:54:31.033040 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:31.033021 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-6vp95" Oct 30 08:54:31.071933 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:31.071893 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-6vp95" podStartSLOduration=2.512551941 podStartE2EDuration="4.071882485s" totalImagesPullingTime="1.559330544s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:54:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:54:31.071313493 +0000 UTC m=+31.332874478" watchObservedRunningTime="2026-10-30 08:54:31.071882485 +0000 UTC m=+31.333443476" Oct 30 08:54:31.156946 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:31.156898 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-bbf8d8c5f-dxdml" podStartSLOduration=2.382814391 podStartE2EDuration="4.15688458s" totalImagesPullingTime="1.774070189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:54:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:54:31.156317929 +0000 UTC m=+31.417878945" watchObservedRunningTime="2026-10-30 08:54:31.15688458 +0000 UTC m=+31.418445567" Oct 30 08:54:33.606367 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:33.606331 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-nvfch"] Oct 30 08:54:33.608773 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:33.608757 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-nvfch" Oct 30 08:54:33.623815 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:33.623793 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/36743cf2-58e7-4e0f-a6d3-df4733e65937-tmp\") pod \"downloads-6dd789c8b7-nvfch\" (UID: \"36743cf2-58e7-4e0f-a6d3-df4733e65937\") " pod="openshift-console/downloads-6dd789c8b7-nvfch" Oct 30 08:54:33.623948 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:33.623865 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5f92\" (UniqueName: \"kubernetes.io/projected/36743cf2-58e7-4e0f-a6d3-df4733e65937-kube-api-access-n5f92\") pod \"downloads-6dd789c8b7-nvfch\" (UID: \"36743cf2-58e7-4e0f-a6d3-df4733e65937\") " pod="openshift-console/downloads-6dd789c8b7-nvfch" Oct 30 08:54:33.624241 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:33.624226 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 08:54:33.624308 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:33.624227 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 08:54:33.624308 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:33.624262 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-v4qtk\"" Oct 30 08:54:33.638534 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:33.638509 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-nvfch"] Oct 30 08:54:33.724498 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:33.724464 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/36743cf2-58e7-4e0f-a6d3-df4733e65937-tmp\") pod \"downloads-6dd789c8b7-nvfch\" (UID: \"36743cf2-58e7-4e0f-a6d3-df4733e65937\") " pod="openshift-console/downloads-6dd789c8b7-nvfch" Oct 30 08:54:33.724666 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:33.724527 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5f92\" (UniqueName: \"kubernetes.io/projected/36743cf2-58e7-4e0f-a6d3-df4733e65937-kube-api-access-n5f92\") pod \"downloads-6dd789c8b7-nvfch\" (UID: \"36743cf2-58e7-4e0f-a6d3-df4733e65937\") " pod="openshift-console/downloads-6dd789c8b7-nvfch" Oct 30 08:54:33.724837 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:33.724819 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/36743cf2-58e7-4e0f-a6d3-df4733e65937-tmp\") pod \"downloads-6dd789c8b7-nvfch\" (UID: \"36743cf2-58e7-4e0f-a6d3-df4733e65937\") " pod="openshift-console/downloads-6dd789c8b7-nvfch" Oct 30 08:54:33.745019 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:33.744991 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5f92\" (UniqueName: \"kubernetes.io/projected/36743cf2-58e7-4e0f-a6d3-df4733e65937-kube-api-access-n5f92\") pod \"downloads-6dd789c8b7-nvfch\" (UID: \"36743cf2-58e7-4e0f-a6d3-df4733e65937\") " pod="openshift-console/downloads-6dd789c8b7-nvfch" Oct 30 08:54:33.918065 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:33.917978 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-nvfch" Oct 30 08:54:33.964655 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:33.964631 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-58cf89764b-plrvc" Oct 30 08:54:34.221750 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:34.221724 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-nvfch"] Oct 30 08:54:34.222691 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:54:34.222664 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36743cf2_58e7_4e0f_a6d3_df4733e65937.slice/crio-a2d7b6d04bfb3ade126c0caf6fb44327ec2a66a0c7678df1392c41e330755dca WatchSource:0}: Error finding container a2d7b6d04bfb3ade126c0caf6fb44327ec2a66a0c7678df1392c41e330755dca: Status 404 returned error can't find the container with id a2d7b6d04bfb3ade126c0caf6fb44327ec2a66a0c7678df1392c41e330755dca Oct 30 08:54:35.045163 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:35.045125 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-nvfch" event={"ID":"36743cf2-58e7-4e0f-a6d3-df4733e65937","Type":"ContainerStarted","Data":"a2d7b6d04bfb3ade126c0caf6fb44327ec2a66a0c7678df1392c41e330755dca"} Oct 30 08:54:35.525975 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:35.525943 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mz2kl" Oct 30 08:54:43.336692 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:43.336603 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-77bc87bfbf-cw68b"] Oct 30 08:54:43.342254 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:43.342226 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77bc87bfbf-cw68b" Oct 30 08:54:43.356559 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:43.356534 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77bc87bfbf-cw68b"] Oct 30 08:54:43.379337 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:43.379312 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 08:54:43.379573 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:43.379551 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 08:54:43.379651 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:43.379614 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 08:54:43.381088 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:43.381073 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 08:54:43.381860 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:43.381842 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 08:54:43.382010 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:43.381995 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-qw5g4\"" Oct 30 08:54:43.399833 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:43.399809 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/210800be-29a2-40f4-a7de-f2c4ef9a8acd-service-ca\") pod \"console-77bc87bfbf-cw68b\" (UID: \"210800be-29a2-40f4-a7de-f2c4ef9a8acd\") " pod="openshift-console/console-77bc87bfbf-cw68b" Oct 30 08:54:43.399958 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:43.399852 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/210800be-29a2-40f4-a7de-f2c4ef9a8acd-console-config\") pod \"console-77bc87bfbf-cw68b\" (UID: \"210800be-29a2-40f4-a7de-f2c4ef9a8acd\") " pod="openshift-console/console-77bc87bfbf-cw68b" Oct 30 08:54:43.400029 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:43.399957 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/210800be-29a2-40f4-a7de-f2c4ef9a8acd-oauth-serving-cert\") pod \"console-77bc87bfbf-cw68b\" (UID: \"210800be-29a2-40f4-a7de-f2c4ef9a8acd\") " pod="openshift-console/console-77bc87bfbf-cw68b" Oct 30 08:54:43.400029 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:43.400003 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/210800be-29a2-40f4-a7de-f2c4ef9a8acd-console-oauth-config\") pod \"console-77bc87bfbf-cw68b\" (UID: \"210800be-29a2-40f4-a7de-f2c4ef9a8acd\") " pod="openshift-console/console-77bc87bfbf-cw68b" Oct 30 08:54:43.400142 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:43.400043 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/210800be-29a2-40f4-a7de-f2c4ef9a8acd-console-serving-cert\") pod \"console-77bc87bfbf-cw68b\" (UID: \"210800be-29a2-40f4-a7de-f2c4ef9a8acd\") " pod="openshift-console/console-77bc87bfbf-cw68b" Oct 30 08:54:43.400199 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:43.400138 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj5v6\" (UniqueName: \"kubernetes.io/projected/210800be-29a2-40f4-a7de-f2c4ef9a8acd-kube-api-access-kj5v6\") pod \"console-77bc87bfbf-cw68b\" (UID: \"210800be-29a2-40f4-a7de-f2c4ef9a8acd\") " pod="openshift-console/console-77bc87bfbf-cw68b" Oct 30 08:54:43.400199 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:43.400174 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/210800be-29a2-40f4-a7de-f2c4ef9a8acd-tmp\") pod \"console-77bc87bfbf-cw68b\" (UID: \"210800be-29a2-40f4-a7de-f2c4ef9a8acd\") " pod="openshift-console/console-77bc87bfbf-cw68b" Oct 30 08:54:43.501309 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:43.501272 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj5v6\" (UniqueName: \"kubernetes.io/projected/210800be-29a2-40f4-a7de-f2c4ef9a8acd-kube-api-access-kj5v6\") pod \"console-77bc87bfbf-cw68b\" (UID: \"210800be-29a2-40f4-a7de-f2c4ef9a8acd\") " pod="openshift-console/console-77bc87bfbf-cw68b" Oct 30 08:54:43.501309 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:43.501321 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/210800be-29a2-40f4-a7de-f2c4ef9a8acd-tmp\") pod \"console-77bc87bfbf-cw68b\" (UID: \"210800be-29a2-40f4-a7de-f2c4ef9a8acd\") " pod="openshift-console/console-77bc87bfbf-cw68b" Oct 30 08:54:43.501569 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:43.501365 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/210800be-29a2-40f4-a7de-f2c4ef9a8acd-service-ca\") pod \"console-77bc87bfbf-cw68b\" (UID: \"210800be-29a2-40f4-a7de-f2c4ef9a8acd\") " pod="openshift-console/console-77bc87bfbf-cw68b" Oct 30 08:54:43.501569 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:43.501426 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/210800be-29a2-40f4-a7de-f2c4ef9a8acd-console-config\") pod \"console-77bc87bfbf-cw68b\" (UID: \"210800be-29a2-40f4-a7de-f2c4ef9a8acd\") " pod="openshift-console/console-77bc87bfbf-cw68b" Oct 30 08:54:43.501569 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:43.501460 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/210800be-29a2-40f4-a7de-f2c4ef9a8acd-oauth-serving-cert\") pod \"console-77bc87bfbf-cw68b\" (UID: \"210800be-29a2-40f4-a7de-f2c4ef9a8acd\") " pod="openshift-console/console-77bc87bfbf-cw68b" Oct 30 08:54:43.501569 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:43.501491 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/210800be-29a2-40f4-a7de-f2c4ef9a8acd-console-oauth-config\") pod \"console-77bc87bfbf-cw68b\" (UID: \"210800be-29a2-40f4-a7de-f2c4ef9a8acd\") " pod="openshift-console/console-77bc87bfbf-cw68b" Oct 30 08:54:43.501569 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:43.501522 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/210800be-29a2-40f4-a7de-f2c4ef9a8acd-console-serving-cert\") pod \"console-77bc87bfbf-cw68b\" (UID: \"210800be-29a2-40f4-a7de-f2c4ef9a8acd\") " pod="openshift-console/console-77bc87bfbf-cw68b" Oct 30 08:54:43.501857 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:43.501831 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/210800be-29a2-40f4-a7de-f2c4ef9a8acd-tmp\") pod \"console-77bc87bfbf-cw68b\" (UID: \"210800be-29a2-40f4-a7de-f2c4ef9a8acd\") " pod="openshift-console/console-77bc87bfbf-cw68b" Oct 30 08:54:43.502417 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:43.502349 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/210800be-29a2-40f4-a7de-f2c4ef9a8acd-service-ca\") pod \"console-77bc87bfbf-cw68b\" (UID: \"210800be-29a2-40f4-a7de-f2c4ef9a8acd\") " pod="openshift-console/console-77bc87bfbf-cw68b" Oct 30 08:54:43.502544 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:43.502484 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/210800be-29a2-40f4-a7de-f2c4ef9a8acd-oauth-serving-cert\") pod \"console-77bc87bfbf-cw68b\" (UID: \"210800be-29a2-40f4-a7de-f2c4ef9a8acd\") " pod="openshift-console/console-77bc87bfbf-cw68b" Oct 30 08:54:43.502614 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:43.502592 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/210800be-29a2-40f4-a7de-f2c4ef9a8acd-console-config\") pod \"console-77bc87bfbf-cw68b\" (UID: \"210800be-29a2-40f4-a7de-f2c4ef9a8acd\") " pod="openshift-console/console-77bc87bfbf-cw68b" Oct 30 08:54:43.504545 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:43.504500 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/210800be-29a2-40f4-a7de-f2c4ef9a8acd-console-serving-cert\") pod \"console-77bc87bfbf-cw68b\" (UID: \"210800be-29a2-40f4-a7de-f2c4ef9a8acd\") " pod="openshift-console/console-77bc87bfbf-cw68b" Oct 30 08:54:43.504668 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:43.504588 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/210800be-29a2-40f4-a7de-f2c4ef9a8acd-console-oauth-config\") pod \"console-77bc87bfbf-cw68b\" (UID: \"210800be-29a2-40f4-a7de-f2c4ef9a8acd\") " pod="openshift-console/console-77bc87bfbf-cw68b" Oct 30 08:54:43.521914 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:43.521892 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj5v6\" (UniqueName: \"kubernetes.io/projected/210800be-29a2-40f4-a7de-f2c4ef9a8acd-kube-api-access-kj5v6\") pod \"console-77bc87bfbf-cw68b\" (UID: \"210800be-29a2-40f4-a7de-f2c4ef9a8acd\") " pod="openshift-console/console-77bc87bfbf-cw68b" Oct 30 08:54:43.656670 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:43.656590 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77bc87bfbf-cw68b" Oct 30 08:54:43.954276 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:43.954247 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77bc87bfbf-cw68b"] Oct 30 08:54:43.955161 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:54:43.955133 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod210800be_29a2_40f4_a7de_f2c4ef9a8acd.slice/crio-96d03a10656b8ef7d3629b519a14b2bcc170460ac5b4f421c7d03a90cc539c13 WatchSource:0}: Error finding container 96d03a10656b8ef7d3629b519a14b2bcc170460ac5b4f421c7d03a90cc539c13: Status 404 returned error can't find the container with id 96d03a10656b8ef7d3629b519a14b2bcc170460ac5b4f421c7d03a90cc539c13 Oct 30 08:54:44.079365 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:44.079323 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77bc87bfbf-cw68b" event={"ID":"210800be-29a2-40f4-a7de-f2c4ef9a8acd","Type":"ContainerStarted","Data":"96d03a10656b8ef7d3629b519a14b2bcc170460ac5b4f421c7d03a90cc539c13"} Oct 30 08:54:47.093516 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:47.093476 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77bc87bfbf-cw68b" event={"ID":"210800be-29a2-40f4-a7de-f2c4ef9a8acd","Type":"ContainerStarted","Data":"cb605ca507fd44095e1db3cfb6060910eae2f06a5456db9b589c1507fa600301"} Oct 30 08:54:47.906164 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:47.906123 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-bbf8d8c5f-dxdml" Oct 30 08:54:47.906353 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:47.906206 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-bbf8d8c5f-dxdml" Oct 30 08:54:50.106967 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:50.106921 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-nvfch" event={"ID":"36743cf2-58e7-4e0f-a6d3-df4733e65937","Type":"ContainerStarted","Data":"a878745cef60a6cbfc6ae71dd475c334f1ac9250b37001712ec8ffc55ffc61f9"} Oct 30 08:54:50.107479 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:50.107140 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-nvfch" Oct 30 08:54:50.123494 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:50.123465 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-nvfch" Oct 30 08:54:50.173805 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:50.173742 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-nvfch" podStartSLOduration=1.8709252360000002 podStartE2EDuration="17.173725494s" totalImagesPullingTime="15.302800258s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:54:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:54:50.160531248 +0000 UTC m=+50.422092239" watchObservedRunningTime="2026-10-30 08:54:50.173725494 +0000 UTC m=+50.435286483" Oct 30 08:54:50.174162 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:50.174122 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-77bc87bfbf-cw68b" podStartSLOduration=4.214703161 podStartE2EDuration="7.174114774s" totalImagesPullingTime="2.959411613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:54:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:54:47.135641762 +0000 UTC m=+47.397202758" watchObservedRunningTime="2026-10-30 08:54:50.174114774 +0000 UTC m=+50.435675762" Oct 30 08:54:53.351426 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:53.351336 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-cd4b79b88-hprw9"] Oct 30 08:54:53.377524 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:53.377495 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cd4b79b88-hprw9" Oct 30 08:54:53.377688 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:53.377668 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cd4b79b88-hprw9"] Oct 30 08:54:53.434209 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:53.434174 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 08:54:53.492159 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:53.492127 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3af8589f-5f3d-4eb7-a997-2483276c2d1f-console-serving-cert\") pod \"console-cd4b79b88-hprw9\" (UID: \"3af8589f-5f3d-4eb7-a997-2483276c2d1f\") " pod="openshift-console/console-cd4b79b88-hprw9" Oct 30 08:54:53.492321 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:53.492181 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3af8589f-5f3d-4eb7-a997-2483276c2d1f-console-oauth-config\") pod \"console-cd4b79b88-hprw9\" (UID: \"3af8589f-5f3d-4eb7-a997-2483276c2d1f\") " pod="openshift-console/console-cd4b79b88-hprw9" Oct 30 08:54:53.492321 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:53.492204 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3af8589f-5f3d-4eb7-a997-2483276c2d1f-service-ca\") pod \"console-cd4b79b88-hprw9\" (UID: \"3af8589f-5f3d-4eb7-a997-2483276c2d1f\") " pod="openshift-console/console-cd4b79b88-hprw9" Oct 30 08:54:53.492321 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:53.492284 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3af8589f-5f3d-4eb7-a997-2483276c2d1f-trusted-ca-bundle\") pod \"console-cd4b79b88-hprw9\" (UID: \"3af8589f-5f3d-4eb7-a997-2483276c2d1f\") " pod="openshift-console/console-cd4b79b88-hprw9" Oct 30 08:54:53.492492 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:53.492355 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnklm\" (UniqueName: \"kubernetes.io/projected/3af8589f-5f3d-4eb7-a997-2483276c2d1f-kube-api-access-lnklm\") pod \"console-cd4b79b88-hprw9\" (UID: \"3af8589f-5f3d-4eb7-a997-2483276c2d1f\") " pod="openshift-console/console-cd4b79b88-hprw9" Oct 30 08:54:53.492492 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:53.492393 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3af8589f-5f3d-4eb7-a997-2483276c2d1f-oauth-serving-cert\") pod \"console-cd4b79b88-hprw9\" (UID: \"3af8589f-5f3d-4eb7-a997-2483276c2d1f\") " pod="openshift-console/console-cd4b79b88-hprw9" Oct 30 08:54:53.492492 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:53.492444 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3af8589f-5f3d-4eb7-a997-2483276c2d1f-tmp\") pod \"console-cd4b79b88-hprw9\" (UID: \"3af8589f-5f3d-4eb7-a997-2483276c2d1f\") " pod="openshift-console/console-cd4b79b88-hprw9" Oct 30 08:54:53.492492 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:53.492475 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3af8589f-5f3d-4eb7-a997-2483276c2d1f-console-config\") pod \"console-cd4b79b88-hprw9\" (UID: \"3af8589f-5f3d-4eb7-a997-2483276c2d1f\") " pod="openshift-console/console-cd4b79b88-hprw9" Oct 30 08:54:53.593263 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:53.593229 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnklm\" (UniqueName: \"kubernetes.io/projected/3af8589f-5f3d-4eb7-a997-2483276c2d1f-kube-api-access-lnklm\") pod \"console-cd4b79b88-hprw9\" (UID: \"3af8589f-5f3d-4eb7-a997-2483276c2d1f\") " pod="openshift-console/console-cd4b79b88-hprw9" Oct 30 08:54:53.593488 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:53.593269 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3af8589f-5f3d-4eb7-a997-2483276c2d1f-oauth-serving-cert\") pod \"console-cd4b79b88-hprw9\" (UID: \"3af8589f-5f3d-4eb7-a997-2483276c2d1f\") " pod="openshift-console/console-cd4b79b88-hprw9" Oct 30 08:54:53.593488 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:53.593301 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3af8589f-5f3d-4eb7-a997-2483276c2d1f-tmp\") pod \"console-cd4b79b88-hprw9\" (UID: \"3af8589f-5f3d-4eb7-a997-2483276c2d1f\") " pod="openshift-console/console-cd4b79b88-hprw9" Oct 30 08:54:53.593488 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:53.593335 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3af8589f-5f3d-4eb7-a997-2483276c2d1f-console-config\") pod \"console-cd4b79b88-hprw9\" (UID: \"3af8589f-5f3d-4eb7-a997-2483276c2d1f\") " pod="openshift-console/console-cd4b79b88-hprw9" Oct 30 08:54:53.593488 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:53.593434 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3af8589f-5f3d-4eb7-a997-2483276c2d1f-console-serving-cert\") pod \"console-cd4b79b88-hprw9\" (UID: \"3af8589f-5f3d-4eb7-a997-2483276c2d1f\") " pod="openshift-console/console-cd4b79b88-hprw9" Oct 30 08:54:53.593488 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:53.593473 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3af8589f-5f3d-4eb7-a997-2483276c2d1f-console-oauth-config\") pod \"console-cd4b79b88-hprw9\" (UID: \"3af8589f-5f3d-4eb7-a997-2483276c2d1f\") " pod="openshift-console/console-cd4b79b88-hprw9" Oct 30 08:54:53.593762 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:53.593494 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3af8589f-5f3d-4eb7-a997-2483276c2d1f-service-ca\") pod \"console-cd4b79b88-hprw9\" (UID: \"3af8589f-5f3d-4eb7-a997-2483276c2d1f\") " pod="openshift-console/console-cd4b79b88-hprw9" Oct 30 08:54:53.593762 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:53.593518 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3af8589f-5f3d-4eb7-a997-2483276c2d1f-trusted-ca-bundle\") pod \"console-cd4b79b88-hprw9\" (UID: \"3af8589f-5f3d-4eb7-a997-2483276c2d1f\") " pod="openshift-console/console-cd4b79b88-hprw9" Oct 30 08:54:53.593762 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:53.593720 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3af8589f-5f3d-4eb7-a997-2483276c2d1f-tmp\") pod \"console-cd4b79b88-hprw9\" (UID: \"3af8589f-5f3d-4eb7-a997-2483276c2d1f\") " pod="openshift-console/console-cd4b79b88-hprw9" Oct 30 08:54:53.594160 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:53.594133 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3af8589f-5f3d-4eb7-a997-2483276c2d1f-oauth-serving-cert\") pod \"console-cd4b79b88-hprw9\" (UID: \"3af8589f-5f3d-4eb7-a997-2483276c2d1f\") " pod="openshift-console/console-cd4b79b88-hprw9" Oct 30 08:54:53.594283 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:53.594211 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3af8589f-5f3d-4eb7-a997-2483276c2d1f-console-config\") pod \"console-cd4b79b88-hprw9\" (UID: \"3af8589f-5f3d-4eb7-a997-2483276c2d1f\") " pod="openshift-console/console-cd4b79b88-hprw9" Oct 30 08:54:53.594878 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:53.594851 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3af8589f-5f3d-4eb7-a997-2483276c2d1f-trusted-ca-bundle\") pod \"console-cd4b79b88-hprw9\" (UID: \"3af8589f-5f3d-4eb7-a997-2483276c2d1f\") " pod="openshift-console/console-cd4b79b88-hprw9" Oct 30 08:54:53.595324 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:53.595299 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3af8589f-5f3d-4eb7-a997-2483276c2d1f-service-ca\") pod \"console-cd4b79b88-hprw9\" (UID: \"3af8589f-5f3d-4eb7-a997-2483276c2d1f\") " pod="openshift-console/console-cd4b79b88-hprw9" Oct 30 08:54:53.596552 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:53.596527 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3af8589f-5f3d-4eb7-a997-2483276c2d1f-console-serving-cert\") pod \"console-cd4b79b88-hprw9\" (UID: \"3af8589f-5f3d-4eb7-a997-2483276c2d1f\") " pod="openshift-console/console-cd4b79b88-hprw9" Oct 30 08:54:53.596552 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:53.596539 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3af8589f-5f3d-4eb7-a997-2483276c2d1f-console-oauth-config\") pod \"console-cd4b79b88-hprw9\" (UID: \"3af8589f-5f3d-4eb7-a997-2483276c2d1f\") " pod="openshift-console/console-cd4b79b88-hprw9" Oct 30 08:54:53.614213 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:53.614157 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnklm\" (UniqueName: \"kubernetes.io/projected/3af8589f-5f3d-4eb7-a997-2483276c2d1f-kube-api-access-lnklm\") pod \"console-cd4b79b88-hprw9\" (UID: \"3af8589f-5f3d-4eb7-a997-2483276c2d1f\") " pod="openshift-console/console-cd4b79b88-hprw9" Oct 30 08:54:53.656936 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:53.656889 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-77bc87bfbf-cw68b" Oct 30 08:54:53.657102 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:53.656966 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-77bc87bfbf-cw68b" Oct 30 08:54:53.663008 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:53.662979 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-77bc87bfbf-cw68b" Oct 30 08:54:53.706133 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:53.706105 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cd4b79b88-hprw9" Oct 30 08:54:54.029249 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:54.029218 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cd4b79b88-hprw9"] Oct 30 08:54:54.030234 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:54:54.030192 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3af8589f_5f3d_4eb7_a997_2483276c2d1f.slice/crio-66bcdd547ae506455a2c00618dd647cdcb06e61c57a6a3601390662a141e0ad3 WatchSource:0}: Error finding container 66bcdd547ae506455a2c00618dd647cdcb06e61c57a6a3601390662a141e0ad3: Status 404 returned error can't find the container with id 66bcdd547ae506455a2c00618dd647cdcb06e61c57a6a3601390662a141e0ad3 Oct 30 08:54:54.125849 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:54.125818 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cd4b79b88-hprw9" event={"ID":"3af8589f-5f3d-4eb7-a997-2483276c2d1f","Type":"ContainerStarted","Data":"66bcdd547ae506455a2c00618dd647cdcb06e61c57a6a3601390662a141e0ad3"} Oct 30 08:54:54.130126 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:54.130102 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-77bc87bfbf-cw68b" Oct 30 08:54:55.131253 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:55.131210 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cd4b79b88-hprw9" event={"ID":"3af8589f-5f3d-4eb7-a997-2483276c2d1f","Type":"ContainerStarted","Data":"e7bd9f263a4ca04ac679a760b1c9c0d6e7e9878d6605c52b4ad308415d61a199"} Oct 30 08:54:55.187013 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:54:55.186961 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-cd4b79b88-hprw9" podStartSLOduration=2.186942285 podStartE2EDuration="2.186942285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:54:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:54:55.172676857 +0000 UTC m=+55.434237845" watchObservedRunningTime="2026-10-30 08:54:55.186942285 +0000 UTC m=+55.448503270" Oct 30 08:54:58.594539 ip-10-0-143-244 kubenswrapper[4562]: E1030 08:54:58.594501 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 08:55:03.706517 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:03.706481 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-cd4b79b88-hprw9" Oct 30 08:55:03.707019 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:03.706552 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-cd4b79b88-hprw9" Oct 30 08:55:03.710939 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:03.710915 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-cd4b79b88-hprw9" Oct 30 08:55:04.166587 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:04.166506 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-cd4b79b88-hprw9" Oct 30 08:55:04.234895 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:04.234859 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77bc87bfbf-cw68b"] Oct 30 08:55:07.911223 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:07.911190 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-bbf8d8c5f-dxdml" Oct 30 08:55:07.914992 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:07.914970 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-bbf8d8c5f-dxdml" Oct 30 08:55:10.720800 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:10.720767 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-56bcdf8bfd-z2fmv"] Oct 30 08:55:10.728087 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:10.728062 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56bcdf8bfd-z2fmv" Oct 30 08:55:10.745883 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:10.745856 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56bcdf8bfd-z2fmv"] Oct 30 08:55:10.832761 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:10.832728 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/87ebdbba-18b3-43d3-a509-93382374d35f-console-serving-cert\") pod \"console-56bcdf8bfd-z2fmv\" (UID: \"87ebdbba-18b3-43d3-a509-93382374d35f\") " pod="openshift-console/console-56bcdf8bfd-z2fmv" Oct 30 08:55:10.832924 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:10.832774 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tcxn\" (UniqueName: \"kubernetes.io/projected/87ebdbba-18b3-43d3-a509-93382374d35f-kube-api-access-8tcxn\") pod \"console-56bcdf8bfd-z2fmv\" (UID: \"87ebdbba-18b3-43d3-a509-93382374d35f\") " pod="openshift-console/console-56bcdf8bfd-z2fmv" Oct 30 08:55:10.832924 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:10.832802 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/87ebdbba-18b3-43d3-a509-93382374d35f-tmp\") pod \"console-56bcdf8bfd-z2fmv\" (UID: \"87ebdbba-18b3-43d3-a509-93382374d35f\") " pod="openshift-console/console-56bcdf8bfd-z2fmv" Oct 30 08:55:10.832924 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:10.832878 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87ebdbba-18b3-43d3-a509-93382374d35f-trusted-ca-bundle\") pod \"console-56bcdf8bfd-z2fmv\" (UID: \"87ebdbba-18b3-43d3-a509-93382374d35f\") " pod="openshift-console/console-56bcdf8bfd-z2fmv" Oct 30 08:55:10.833040 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:10.832968 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/87ebdbba-18b3-43d3-a509-93382374d35f-service-ca\") pod \"console-56bcdf8bfd-z2fmv\" (UID: \"87ebdbba-18b3-43d3-a509-93382374d35f\") " pod="openshift-console/console-56bcdf8bfd-z2fmv" Oct 30 08:55:10.833040 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:10.832999 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/87ebdbba-18b3-43d3-a509-93382374d35f-console-oauth-config\") pod \"console-56bcdf8bfd-z2fmv\" (UID: \"87ebdbba-18b3-43d3-a509-93382374d35f\") " pod="openshift-console/console-56bcdf8bfd-z2fmv" Oct 30 08:55:10.833113 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:10.833067 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/87ebdbba-18b3-43d3-a509-93382374d35f-oauth-serving-cert\") pod \"console-56bcdf8bfd-z2fmv\" (UID: \"87ebdbba-18b3-43d3-a509-93382374d35f\") " pod="openshift-console/console-56bcdf8bfd-z2fmv" Oct 30 08:55:10.833161 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:10.833117 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/87ebdbba-18b3-43d3-a509-93382374d35f-console-config\") pod \"console-56bcdf8bfd-z2fmv\" (UID: \"87ebdbba-18b3-43d3-a509-93382374d35f\") " pod="openshift-console/console-56bcdf8bfd-z2fmv" Oct 30 08:55:10.933789 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:10.933757 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/87ebdbba-18b3-43d3-a509-93382374d35f-console-oauth-config\") pod \"console-56bcdf8bfd-z2fmv\" (UID: \"87ebdbba-18b3-43d3-a509-93382374d35f\") " pod="openshift-console/console-56bcdf8bfd-z2fmv" Oct 30 08:55:10.933970 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:10.933810 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/87ebdbba-18b3-43d3-a509-93382374d35f-oauth-serving-cert\") pod \"console-56bcdf8bfd-z2fmv\" (UID: \"87ebdbba-18b3-43d3-a509-93382374d35f\") " pod="openshift-console/console-56bcdf8bfd-z2fmv" Oct 30 08:55:10.933970 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:10.933834 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/87ebdbba-18b3-43d3-a509-93382374d35f-console-config\") pod \"console-56bcdf8bfd-z2fmv\" (UID: \"87ebdbba-18b3-43d3-a509-93382374d35f\") " pod="openshift-console/console-56bcdf8bfd-z2fmv" Oct 30 08:55:10.934083 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:10.933969 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/87ebdbba-18b3-43d3-a509-93382374d35f-console-serving-cert\") pod \"console-56bcdf8bfd-z2fmv\" (UID: \"87ebdbba-18b3-43d3-a509-93382374d35f\") " pod="openshift-console/console-56bcdf8bfd-z2fmv" Oct 30 08:55:10.934083 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:10.934007 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tcxn\" (UniqueName: \"kubernetes.io/projected/87ebdbba-18b3-43d3-a509-93382374d35f-kube-api-access-8tcxn\") pod \"console-56bcdf8bfd-z2fmv\" (UID: \"87ebdbba-18b3-43d3-a509-93382374d35f\") " pod="openshift-console/console-56bcdf8bfd-z2fmv" Oct 30 08:55:10.934183 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:10.934133 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/87ebdbba-18b3-43d3-a509-93382374d35f-tmp\") pod \"console-56bcdf8bfd-z2fmv\" (UID: \"87ebdbba-18b3-43d3-a509-93382374d35f\") " pod="openshift-console/console-56bcdf8bfd-z2fmv" Oct 30 08:55:10.934183 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:10.934174 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87ebdbba-18b3-43d3-a509-93382374d35f-trusted-ca-bundle\") pod \"console-56bcdf8bfd-z2fmv\" (UID: \"87ebdbba-18b3-43d3-a509-93382374d35f\") " pod="openshift-console/console-56bcdf8bfd-z2fmv" Oct 30 08:55:10.934284 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:10.934215 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/87ebdbba-18b3-43d3-a509-93382374d35f-service-ca\") pod \"console-56bcdf8bfd-z2fmv\" (UID: \"87ebdbba-18b3-43d3-a509-93382374d35f\") " pod="openshift-console/console-56bcdf8bfd-z2fmv" Oct 30 08:55:10.934682 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:10.934656 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/87ebdbba-18b3-43d3-a509-93382374d35f-tmp\") pod \"console-56bcdf8bfd-z2fmv\" (UID: \"87ebdbba-18b3-43d3-a509-93382374d35f\") " pod="openshift-console/console-56bcdf8bfd-z2fmv" Oct 30 08:55:10.934819 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:10.934749 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/87ebdbba-18b3-43d3-a509-93382374d35f-oauth-serving-cert\") pod \"console-56bcdf8bfd-z2fmv\" (UID: \"87ebdbba-18b3-43d3-a509-93382374d35f\") " pod="openshift-console/console-56bcdf8bfd-z2fmv" Oct 30 08:55:10.934819 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:10.934771 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/87ebdbba-18b3-43d3-a509-93382374d35f-console-config\") pod \"console-56bcdf8bfd-z2fmv\" (UID: \"87ebdbba-18b3-43d3-a509-93382374d35f\") " pod="openshift-console/console-56bcdf8bfd-z2fmv" Oct 30 08:55:10.934976 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:10.934958 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/87ebdbba-18b3-43d3-a509-93382374d35f-service-ca\") pod \"console-56bcdf8bfd-z2fmv\" (UID: \"87ebdbba-18b3-43d3-a509-93382374d35f\") " pod="openshift-console/console-56bcdf8bfd-z2fmv" Oct 30 08:55:10.935473 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:10.935448 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87ebdbba-18b3-43d3-a509-93382374d35f-trusted-ca-bundle\") pod \"console-56bcdf8bfd-z2fmv\" (UID: \"87ebdbba-18b3-43d3-a509-93382374d35f\") " pod="openshift-console/console-56bcdf8bfd-z2fmv" Oct 30 08:55:10.936442 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:10.936421 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/87ebdbba-18b3-43d3-a509-93382374d35f-console-oauth-config\") pod \"console-56bcdf8bfd-z2fmv\" (UID: \"87ebdbba-18b3-43d3-a509-93382374d35f\") " pod="openshift-console/console-56bcdf8bfd-z2fmv" Oct 30 08:55:10.936524 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:10.936451 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/87ebdbba-18b3-43d3-a509-93382374d35f-console-serving-cert\") pod \"console-56bcdf8bfd-z2fmv\" (UID: \"87ebdbba-18b3-43d3-a509-93382374d35f\") " pod="openshift-console/console-56bcdf8bfd-z2fmv" Oct 30 08:55:10.953787 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:10.953764 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tcxn\" (UniqueName: \"kubernetes.io/projected/87ebdbba-18b3-43d3-a509-93382374d35f-kube-api-access-8tcxn\") pod \"console-56bcdf8bfd-z2fmv\" (UID: \"87ebdbba-18b3-43d3-a509-93382374d35f\") " pod="openshift-console/console-56bcdf8bfd-z2fmv" Oct 30 08:55:11.037426 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:11.037355 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56bcdf8bfd-z2fmv" Oct 30 08:55:11.328504 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:11.328483 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56bcdf8bfd-z2fmv"] Oct 30 08:55:11.329191 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:55:11.329160 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87ebdbba_18b3_43d3_a509_93382374d35f.slice/crio-6af2ee5b076f09debf7f33598dcafde38a54647d5fe9919106e448992e505c6f WatchSource:0}: Error finding container 6af2ee5b076f09debf7f33598dcafde38a54647d5fe9919106e448992e505c6f: Status 404 returned error can't find the container with id 6af2ee5b076f09debf7f33598dcafde38a54647d5fe9919106e448992e505c6f Oct 30 08:55:12.188946 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:12.188908 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56bcdf8bfd-z2fmv" event={"ID":"87ebdbba-18b3-43d3-a509-93382374d35f","Type":"ContainerStarted","Data":"74370a503ba00b6e6959624909b5ee3ca28b94fcf1b835833b40adb226ad30f3"} Oct 30 08:55:12.189449 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:12.188959 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56bcdf8bfd-z2fmv" event={"ID":"87ebdbba-18b3-43d3-a509-93382374d35f","Type":"ContainerStarted","Data":"6af2ee5b076f09debf7f33598dcafde38a54647d5fe9919106e448992e505c6f"} Oct 30 08:55:12.242875 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:12.242816 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-56bcdf8bfd-z2fmv" podStartSLOduration=2.242802825 podStartE2EDuration="2.242802825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:55:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:55:12.241743082 +0000 UTC m=+72.503304070" watchObservedRunningTime="2026-10-30 08:55:12.242802825 +0000 UTC m=+72.504363813" Oct 30 08:55:18.212828 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:18.212794 4562 generic.go:356] "Generic (PLEG): container finished" podID="6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2" containerID="859c0d2a7fbfc11982a8900f9a8d63d376ba7f82a1a0e8ce7b0955524c93589f" exitCode=0 Oct 30 08:55:18.213236 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:18.212863 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-z7wg9-fvpzr" event={"ID":"6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2","Type":"ContainerDied","Data":"859c0d2a7fbfc11982a8900f9a8d63d376ba7f82a1a0e8ce7b0955524c93589f"} Oct 30 08:55:19.352017 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:19.351982 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-z7wg9-fvpzr" Oct 30 08:55:19.510077 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:19.509997 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2-kube-api-access-tb9vs\" (UniqueName: \"kubernetes.io/projected/6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2-kube-api-access-tb9vs\") pod \"6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2\" (UID: \"6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2\") " Oct 30 08:55:19.510077 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:19.510055 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2-archives-path\") pod \"6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2\" (UID: \"6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2\") " Oct 30 08:55:19.510305 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:19.510199 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2-service-ca-bundle\") pod \"6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2\" (UID: \"6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2\") " Oct 30 08:55:19.510305 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:19.510235 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2-trusted-ca-bundle\") pod \"6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2\" (UID: \"6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2\") " Oct 30 08:55:19.510440 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:19.510319 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2-archives-path" pod "6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2" (UID: "6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:55:19.510548 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:19.510529 4562 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2-archives-path\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 08:55:19.510586 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:19.510541 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2-service-ca-bundle" pod "6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2" (UID: "6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:55:19.510586 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:19.510567 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2-trusted-ca-bundle" pod "6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2" (UID: "6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:55:19.512237 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:19.512218 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2-kube-api-access-tb9vs" pod "6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2" (UID: "6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2"). InnerVolumeSpecName "kube-api-access-tb9vs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:55:19.611707 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:19.611678 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tb9vs\" (UniqueName: \"kubernetes.io/projected/6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2-kube-api-access-tb9vs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 08:55:19.611707 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:19.611704 4562 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2-service-ca-bundle\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 08:55:19.611707 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:19.611713 4562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2-trusted-ca-bundle\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 08:55:20.220578 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:20.220544 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-z7wg9-fvpzr" event={"ID":"6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2","Type":"ContainerDied","Data":"50d630405f1e6ff1b6bbb8e451da2ab8317b82f08e08cfce7f1fdf5cd0ea7728"} Oct 30 08:55:20.220578 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:20.220564 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-z7wg9-fvpzr" Oct 30 08:55:20.220767 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:20.220593 4562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50d630405f1e6ff1b6bbb8e451da2ab8317b82f08e08cfce7f1fdf5cd0ea7728" Oct 30 08:55:21.037886 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:21.037852 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-56bcdf8bfd-z2fmv" Oct 30 08:55:21.038346 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:21.037907 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-56bcdf8bfd-z2fmv" Oct 30 08:55:21.042660 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:21.042639 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-56bcdf8bfd-z2fmv" Oct 30 08:55:21.227855 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:21.227826 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-56bcdf8bfd-z2fmv" Oct 30 08:55:21.293598 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:21.293526 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cd4b79b88-hprw9"] Oct 30 08:55:26.510973 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:26.510941 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-k8knv-9jwhj"] Oct 30 08:55:26.511338 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:26.511193 4562 cpu_manager.go:414] "containerMap: removing container" podUID="6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2" containerName="insights-gathering" Oct 30 08:55:26.511338 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:26.511203 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2" containerName="insights-gathering" Oct 30 08:55:26.511338 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:26.511258 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="6296ad27-59fd-4726-8ca9-0ccdbf0fb4e2" containerName="insights-gathering" Oct 30 08:55:26.542699 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:26.542676 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-k8knv-9jwhj"] Oct 30 08:55:26.542835 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:26.542761 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-k8knv-9jwhj" Oct 30 08:55:26.571762 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:26.571728 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 08:55:26.571873 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:26.571808 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-nq2rg\"" Oct 30 08:55:26.599108 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:26.599082 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 08:55:26.667766 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:26.667739 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf34fc37-dd3b-44c0-9d55-91ed73b00e7c-service-ca-bundle\") pod \"periodic-gathering-k8knv-9jwhj\" (UID: \"cf34fc37-dd3b-44c0-9d55-91ed73b00e7c\") " pod="openshift-insights/periodic-gathering-k8knv-9jwhj" Oct 30 08:55:26.667881 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:26.667770 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf34fc37-dd3b-44c0-9d55-91ed73b00e7c-trusted-ca-bundle\") pod \"periodic-gathering-k8knv-9jwhj\" (UID: \"cf34fc37-dd3b-44c0-9d55-91ed73b00e7c\") " pod="openshift-insights/periodic-gathering-k8knv-9jwhj" Oct 30 08:55:26.667881 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:26.667789 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w75rb\" (UniqueName: \"kubernetes.io/projected/cf34fc37-dd3b-44c0-9d55-91ed73b00e7c-kube-api-access-w75rb\") pod \"periodic-gathering-k8knv-9jwhj\" (UID: \"cf34fc37-dd3b-44c0-9d55-91ed73b00e7c\") " pod="openshift-insights/periodic-gathering-k8knv-9jwhj" Oct 30 08:55:26.667966 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:26.667951 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/cf34fc37-dd3b-44c0-9d55-91ed73b00e7c-archives-path\") pod \"periodic-gathering-k8knv-9jwhj\" (UID: \"cf34fc37-dd3b-44c0-9d55-91ed73b00e7c\") " pod="openshift-insights/periodic-gathering-k8knv-9jwhj" Oct 30 08:55:26.769078 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:26.769019 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/cf34fc37-dd3b-44c0-9d55-91ed73b00e7c-archives-path\") pod \"periodic-gathering-k8knv-9jwhj\" (UID: \"cf34fc37-dd3b-44c0-9d55-91ed73b00e7c\") " pod="openshift-insights/periodic-gathering-k8knv-9jwhj" Oct 30 08:55:26.769078 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:26.769065 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf34fc37-dd3b-44c0-9d55-91ed73b00e7c-service-ca-bundle\") pod \"periodic-gathering-k8knv-9jwhj\" (UID: \"cf34fc37-dd3b-44c0-9d55-91ed73b00e7c\") " pod="openshift-insights/periodic-gathering-k8knv-9jwhj" Oct 30 08:55:26.769254 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:26.769088 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf34fc37-dd3b-44c0-9d55-91ed73b00e7c-trusted-ca-bundle\") pod \"periodic-gathering-k8knv-9jwhj\" (UID: \"cf34fc37-dd3b-44c0-9d55-91ed73b00e7c\") " pod="openshift-insights/periodic-gathering-k8knv-9jwhj" Oct 30 08:55:26.769254 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:26.769110 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w75rb\" (UniqueName: \"kubernetes.io/projected/cf34fc37-dd3b-44c0-9d55-91ed73b00e7c-kube-api-access-w75rb\") pod \"periodic-gathering-k8knv-9jwhj\" (UID: \"cf34fc37-dd3b-44c0-9d55-91ed73b00e7c\") " pod="openshift-insights/periodic-gathering-k8knv-9jwhj" Oct 30 08:55:26.769508 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:26.769483 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/cf34fc37-dd3b-44c0-9d55-91ed73b00e7c-archives-path\") pod \"periodic-gathering-k8knv-9jwhj\" (UID: \"cf34fc37-dd3b-44c0-9d55-91ed73b00e7c\") " pod="openshift-insights/periodic-gathering-k8knv-9jwhj" Oct 30 08:55:26.769739 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:26.769715 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf34fc37-dd3b-44c0-9d55-91ed73b00e7c-service-ca-bundle\") pod \"periodic-gathering-k8knv-9jwhj\" (UID: \"cf34fc37-dd3b-44c0-9d55-91ed73b00e7c\") " pod="openshift-insights/periodic-gathering-k8knv-9jwhj" Oct 30 08:55:26.769958 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:26.769936 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf34fc37-dd3b-44c0-9d55-91ed73b00e7c-trusted-ca-bundle\") pod \"periodic-gathering-k8knv-9jwhj\" (UID: \"cf34fc37-dd3b-44c0-9d55-91ed73b00e7c\") " pod="openshift-insights/periodic-gathering-k8knv-9jwhj" Oct 30 08:55:26.789453 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:26.789426 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w75rb\" (UniqueName: \"kubernetes.io/projected/cf34fc37-dd3b-44c0-9d55-91ed73b00e7c-kube-api-access-w75rb\") pod \"periodic-gathering-k8knv-9jwhj\" (UID: \"cf34fc37-dd3b-44c0-9d55-91ed73b00e7c\") " pod="openshift-insights/periodic-gathering-k8knv-9jwhj" Oct 30 08:55:26.865494 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:26.865472 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-k8knv-9jwhj" Oct 30 08:55:27.192101 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:27.192073 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-k8knv-9jwhj"] Oct 30 08:55:27.192391 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:55:27.192352 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf34fc37_dd3b_44c0_9d55_91ed73b00e7c.slice/crio-c762e5557b27816afb001ddd48dbff991ab26ceafdeca21db70f16a7165e02bc WatchSource:0}: Error finding container c762e5557b27816afb001ddd48dbff991ab26ceafdeca21db70f16a7165e02bc: Status 404 returned error can't find the container with id c762e5557b27816afb001ddd48dbff991ab26ceafdeca21db70f16a7165e02bc Oct 30 08:55:27.245192 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:27.245170 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-k8knv-9jwhj" event={"ID":"cf34fc37-dd3b-44c0-9d55-91ed73b00e7c","Type":"ContainerStarted","Data":"c762e5557b27816afb001ddd48dbff991ab26ceafdeca21db70f16a7165e02bc"} Oct 30 08:55:28.250064 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:28.250025 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-k8knv-9jwhj" event={"ID":"cf34fc37-dd3b-44c0-9d55-91ed73b00e7c","Type":"ContainerStarted","Data":"c3b3ab220ef67d7aab911b8a29ce53297830b7729fa57cbd0a92972d136edee5"} Oct 30 08:55:28.289290 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:28.289245 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-k8knv-9jwhj" podStartSLOduration=2.289233047 podStartE2EDuration="2.289233047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:55:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:55:28.288892224 +0000 UTC m=+88.550453212" watchObservedRunningTime="2026-10-30 08:55:28.289233047 +0000 UTC m=+88.550794034" Oct 30 08:55:28.594006 ip-10-0-143-244 kubenswrapper[4562]: E1030 08:55:28.593927 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 08:55:29.258257 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:29.258222 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-77bc87bfbf-cw68b" podUID="210800be-29a2-40f4-a7de-f2c4ef9a8acd" containerName="console" containerID="cri-o://cb605ca507fd44095e1db3cfb6060910eae2f06a5456db9b589c1507fa600301" gracePeriod=15 Oct 30 08:55:29.533601 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:29.533546 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77bc87bfbf-cw68b" Oct 30 08:55:29.588824 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:29.588800 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/210800be-29a2-40f4-a7de-f2c4ef9a8acd-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/210800be-29a2-40f4-a7de-f2c4ef9a8acd-oauth-serving-cert\") pod \"210800be-29a2-40f4-a7de-f2c4ef9a8acd\" (UID: \"210800be-29a2-40f4-a7de-f2c4ef9a8acd\") " Oct 30 08:55:29.588954 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:29.588840 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/210800be-29a2-40f4-a7de-f2c4ef9a8acd-tmp\" (UniqueName: \"kubernetes.io/empty-dir/210800be-29a2-40f4-a7de-f2c4ef9a8acd-tmp\") pod \"210800be-29a2-40f4-a7de-f2c4ef9a8acd\" (UID: \"210800be-29a2-40f4-a7de-f2c4ef9a8acd\") " Oct 30 08:55:29.588954 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:29.588883 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/210800be-29a2-40f4-a7de-f2c4ef9a8acd-kube-api-access-kj5v6\" (UniqueName: \"kubernetes.io/projected/210800be-29a2-40f4-a7de-f2c4ef9a8acd-kube-api-access-kj5v6\") pod \"210800be-29a2-40f4-a7de-f2c4ef9a8acd\" (UID: \"210800be-29a2-40f4-a7de-f2c4ef9a8acd\") " Oct 30 08:55:29.588954 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:29.588910 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/210800be-29a2-40f4-a7de-f2c4ef9a8acd-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/210800be-29a2-40f4-a7de-f2c4ef9a8acd-console-serving-cert\") pod \"210800be-29a2-40f4-a7de-f2c4ef9a8acd\" (UID: \"210800be-29a2-40f4-a7de-f2c4ef9a8acd\") " Oct 30 08:55:29.588954 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:29.588941 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/210800be-29a2-40f4-a7de-f2c4ef9a8acd-service-ca\" (UniqueName: \"kubernetes.io/configmap/210800be-29a2-40f4-a7de-f2c4ef9a8acd-service-ca\") pod \"210800be-29a2-40f4-a7de-f2c4ef9a8acd\" (UID: \"210800be-29a2-40f4-a7de-f2c4ef9a8acd\") " Oct 30 08:55:29.589184 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:29.588976 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/210800be-29a2-40f4-a7de-f2c4ef9a8acd-console-config\" (UniqueName: \"kubernetes.io/configmap/210800be-29a2-40f4-a7de-f2c4ef9a8acd-console-config\") pod \"210800be-29a2-40f4-a7de-f2c4ef9a8acd\" (UID: \"210800be-29a2-40f4-a7de-f2c4ef9a8acd\") " Oct 30 08:55:29.589184 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:29.589006 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/210800be-29a2-40f4-a7de-f2c4ef9a8acd-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/210800be-29a2-40f4-a7de-f2c4ef9a8acd-console-oauth-config\") pod \"210800be-29a2-40f4-a7de-f2c4ef9a8acd\" (UID: \"210800be-29a2-40f4-a7de-f2c4ef9a8acd\") " Oct 30 08:55:29.589316 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:29.589294 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/210800be-29a2-40f4-a7de-f2c4ef9a8acd-tmp" pod "210800be-29a2-40f4-a7de-f2c4ef9a8acd" (UID: "210800be-29a2-40f4-a7de-f2c4ef9a8acd"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:55:29.589402 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:29.589327 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/210800be-29a2-40f4-a7de-f2c4ef9a8acd-oauth-serving-cert" pod "210800be-29a2-40f4-a7de-f2c4ef9a8acd" (UID: "210800be-29a2-40f4-a7de-f2c4ef9a8acd"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:55:29.589469 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:29.589437 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/210800be-29a2-40f4-a7de-f2c4ef9a8acd-service-ca" pod "210800be-29a2-40f4-a7de-f2c4ef9a8acd" (UID: "210800be-29a2-40f4-a7de-f2c4ef9a8acd"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:55:29.589546 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:29.589525 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/210800be-29a2-40f4-a7de-f2c4ef9a8acd-console-config" pod "210800be-29a2-40f4-a7de-f2c4ef9a8acd" (UID: "210800be-29a2-40f4-a7de-f2c4ef9a8acd"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:55:29.591190 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:29.591170 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/210800be-29a2-40f4-a7de-f2c4ef9a8acd-console-serving-cert" pod "210800be-29a2-40f4-a7de-f2c4ef9a8acd" (UID: "210800be-29a2-40f4-a7de-f2c4ef9a8acd"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:55:29.591320 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:29.591296 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/210800be-29a2-40f4-a7de-f2c4ef9a8acd-kube-api-access-kj5v6" pod "210800be-29a2-40f4-a7de-f2c4ef9a8acd" (UID: "210800be-29a2-40f4-a7de-f2c4ef9a8acd"). InnerVolumeSpecName "kube-api-access-kj5v6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:55:29.591578 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:29.591557 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/210800be-29a2-40f4-a7de-f2c4ef9a8acd-console-oauth-config" pod "210800be-29a2-40f4-a7de-f2c4ef9a8acd" (UID: "210800be-29a2-40f4-a7de-f2c4ef9a8acd"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:55:29.689955 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:29.689925 4562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/210800be-29a2-40f4-a7de-f2c4ef9a8acd-service-ca\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 08:55:29.689955 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:29.689950 4562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/210800be-29a2-40f4-a7de-f2c4ef9a8acd-console-config\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 08:55:29.690087 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:29.689960 4562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/210800be-29a2-40f4-a7de-f2c4ef9a8acd-console-oauth-config\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 08:55:29.690087 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:29.689970 4562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/210800be-29a2-40f4-a7de-f2c4ef9a8acd-oauth-serving-cert\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 08:55:29.690087 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:29.689980 4562 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/210800be-29a2-40f4-a7de-f2c4ef9a8acd-tmp\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 08:55:29.690087 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:29.689988 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kj5v6\" (UniqueName: \"kubernetes.io/projected/210800be-29a2-40f4-a7de-f2c4ef9a8acd-kube-api-access-kj5v6\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 08:55:29.690087 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:29.689998 4562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/210800be-29a2-40f4-a7de-f2c4ef9a8acd-console-serving-cert\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 08:55:30.257670 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:30.257634 4562 generic.go:356] "Generic (PLEG): container finished" podID="210800be-29a2-40f4-a7de-f2c4ef9a8acd" containerID="cb605ca507fd44095e1db3cfb6060910eae2f06a5456db9b589c1507fa600301" exitCode=0 Oct 30 08:55:30.257835 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:30.257682 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77bc87bfbf-cw68b" Oct 30 08:55:30.257835 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:30.257715 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77bc87bfbf-cw68b" event={"ID":"210800be-29a2-40f4-a7de-f2c4ef9a8acd","Type":"ContainerDied","Data":"cb605ca507fd44095e1db3cfb6060910eae2f06a5456db9b589c1507fa600301"} Oct 30 08:55:30.257835 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:30.257760 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77bc87bfbf-cw68b" event={"ID":"210800be-29a2-40f4-a7de-f2c4ef9a8acd","Type":"ContainerDied","Data":"96d03a10656b8ef7d3629b519a14b2bcc170460ac5b4f421c7d03a90cc539c13"} Oct 30 08:55:30.257835 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:30.257779 4562 scope.go:122] "RemoveContainer" containerID="cb605ca507fd44095e1db3cfb6060910eae2f06a5456db9b589c1507fa600301" Oct 30 08:55:30.265450 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:30.265430 4562 scope.go:122] "RemoveContainer" containerID="cb605ca507fd44095e1db3cfb6060910eae2f06a5456db9b589c1507fa600301" Oct 30 08:55:30.265752 ip-10-0-143-244 kubenswrapper[4562]: E1030 08:55:30.265673 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb605ca507fd44095e1db3cfb6060910eae2f06a5456db9b589c1507fa600301\": container with ID starting with cb605ca507fd44095e1db3cfb6060910eae2f06a5456db9b589c1507fa600301 not found: ID does not exist" containerID="cb605ca507fd44095e1db3cfb6060910eae2f06a5456db9b589c1507fa600301" Oct 30 08:55:30.265752 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:30.265697 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb605ca507fd44095e1db3cfb6060910eae2f06a5456db9b589c1507fa600301"} err="failed to get container status \"cb605ca507fd44095e1db3cfb6060910eae2f06a5456db9b589c1507fa600301\": rpc error: code = NotFound desc = could not find container \"cb605ca507fd44095e1db3cfb6060910eae2f06a5456db9b589c1507fa600301\": container with ID starting with cb605ca507fd44095e1db3cfb6060910eae2f06a5456db9b589c1507fa600301 not found: ID does not exist" Oct 30 08:55:30.309679 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:30.309654 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77bc87bfbf-cw68b"] Oct 30 08:55:30.322734 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:30.322714 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-77bc87bfbf-cw68b"] Oct 30 08:55:31.874791 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:31.874754 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="210800be-29a2-40f4-a7de-f2c4ef9a8acd" path="/var/lib/kubelet/pods/210800be-29a2-40f4-a7de-f2c4ef9a8acd/volumes" Oct 30 08:55:46.313249 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:46.313194 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-cd4b79b88-hprw9" podUID="3af8589f-5f3d-4eb7-a997-2483276c2d1f" containerName="console" containerID="cri-o://e7bd9f263a4ca04ac679a760b1c9c0d6e7e9878d6605c52b4ad308415d61a199" gracePeriod=15 Oct 30 08:55:46.579025 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:46.579000 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cd4b79b88-hprw9" Oct 30 08:55:46.610828 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:46.610801 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3af8589f-5f3d-4eb7-a997-2483276c2d1f-service-ca\" (UniqueName: \"kubernetes.io/configmap/3af8589f-5f3d-4eb7-a997-2483276c2d1f-service-ca\") pod \"3af8589f-5f3d-4eb7-a997-2483276c2d1f\" (UID: \"3af8589f-5f3d-4eb7-a997-2483276c2d1f\") " Oct 30 08:55:46.610963 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:46.610852 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3af8589f-5f3d-4eb7-a997-2483276c2d1f-console-config\" (UniqueName: \"kubernetes.io/configmap/3af8589f-5f3d-4eb7-a997-2483276c2d1f-console-config\") pod \"3af8589f-5f3d-4eb7-a997-2483276c2d1f\" (UID: \"3af8589f-5f3d-4eb7-a997-2483276c2d1f\") " Oct 30 08:55:46.610963 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:46.610880 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3af8589f-5f3d-4eb7-a997-2483276c2d1f-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3af8589f-5f3d-4eb7-a997-2483276c2d1f-console-serving-cert\") pod \"3af8589f-5f3d-4eb7-a997-2483276c2d1f\" (UID: \"3af8589f-5f3d-4eb7-a997-2483276c2d1f\") " Oct 30 08:55:46.610963 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:46.610901 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3af8589f-5f3d-4eb7-a997-2483276c2d1f-tmp\" (UniqueName: \"kubernetes.io/empty-dir/3af8589f-5f3d-4eb7-a997-2483276c2d1f-tmp\") pod \"3af8589f-5f3d-4eb7-a997-2483276c2d1f\" (UID: \"3af8589f-5f3d-4eb7-a997-2483276c2d1f\") " Oct 30 08:55:46.610963 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:46.610948 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3af8589f-5f3d-4eb7-a997-2483276c2d1f-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3af8589f-5f3d-4eb7-a997-2483276c2d1f-trusted-ca-bundle\") pod \"3af8589f-5f3d-4eb7-a997-2483276c2d1f\" (UID: \"3af8589f-5f3d-4eb7-a997-2483276c2d1f\") " Oct 30 08:55:46.611157 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:46.610974 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3af8589f-5f3d-4eb7-a997-2483276c2d1f-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3af8589f-5f3d-4eb7-a997-2483276c2d1f-console-oauth-config\") pod \"3af8589f-5f3d-4eb7-a997-2483276c2d1f\" (UID: \"3af8589f-5f3d-4eb7-a997-2483276c2d1f\") " Oct 30 08:55:46.611157 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:46.611009 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3af8589f-5f3d-4eb7-a997-2483276c2d1f-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3af8589f-5f3d-4eb7-a997-2483276c2d1f-oauth-serving-cert\") pod \"3af8589f-5f3d-4eb7-a997-2483276c2d1f\" (UID: \"3af8589f-5f3d-4eb7-a997-2483276c2d1f\") " Oct 30 08:55:46.611157 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:46.611047 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3af8589f-5f3d-4eb7-a997-2483276c2d1f-kube-api-access-lnklm\" (UniqueName: \"kubernetes.io/projected/3af8589f-5f3d-4eb7-a997-2483276c2d1f-kube-api-access-lnklm\") pod \"3af8589f-5f3d-4eb7-a997-2483276c2d1f\" (UID: \"3af8589f-5f3d-4eb7-a997-2483276c2d1f\") " Oct 30 08:55:46.611157 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:46.611116 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3af8589f-5f3d-4eb7-a997-2483276c2d1f-service-ca" pod "3af8589f-5f3d-4eb7-a997-2483276c2d1f" (UID: "3af8589f-5f3d-4eb7-a997-2483276c2d1f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:55:46.611356 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:46.611234 4562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3af8589f-5f3d-4eb7-a997-2483276c2d1f-service-ca\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 08:55:46.611356 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:46.611311 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3af8589f-5f3d-4eb7-a997-2483276c2d1f-console-config" pod "3af8589f-5f3d-4eb7-a997-2483276c2d1f" (UID: "3af8589f-5f3d-4eb7-a997-2483276c2d1f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:55:46.611785 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:46.611753 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3af8589f-5f3d-4eb7-a997-2483276c2d1f-oauth-serving-cert" pod "3af8589f-5f3d-4eb7-a997-2483276c2d1f" (UID: "3af8589f-5f3d-4eb7-a997-2483276c2d1f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:55:46.611883 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:46.611758 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3af8589f-5f3d-4eb7-a997-2483276c2d1f-trusted-ca-bundle" pod "3af8589f-5f3d-4eb7-a997-2483276c2d1f" (UID: "3af8589f-5f3d-4eb7-a997-2483276c2d1f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:55:46.611883 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:46.611802 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3af8589f-5f3d-4eb7-a997-2483276c2d1f-tmp" pod "3af8589f-5f3d-4eb7-a997-2483276c2d1f" (UID: "3af8589f-5f3d-4eb7-a997-2483276c2d1f"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:55:46.613440 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:46.613407 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3af8589f-5f3d-4eb7-a997-2483276c2d1f-kube-api-access-lnklm" pod "3af8589f-5f3d-4eb7-a997-2483276c2d1f" (UID: "3af8589f-5f3d-4eb7-a997-2483276c2d1f"). InnerVolumeSpecName "kube-api-access-lnklm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:55:46.613723 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:46.613687 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3af8589f-5f3d-4eb7-a997-2483276c2d1f-console-serving-cert" pod "3af8589f-5f3d-4eb7-a997-2483276c2d1f" (UID: "3af8589f-5f3d-4eb7-a997-2483276c2d1f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:55:46.613812 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:46.613782 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3af8589f-5f3d-4eb7-a997-2483276c2d1f-console-oauth-config" pod "3af8589f-5f3d-4eb7-a997-2483276c2d1f" (UID: "3af8589f-5f3d-4eb7-a997-2483276c2d1f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:55:46.711754 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:46.711723 4562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3af8589f-5f3d-4eb7-a997-2483276c2d1f-trusted-ca-bundle\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 08:55:46.711754 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:46.711750 4562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3af8589f-5f3d-4eb7-a997-2483276c2d1f-console-oauth-config\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 08:55:46.711754 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:46.711760 4562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3af8589f-5f3d-4eb7-a997-2483276c2d1f-oauth-serving-cert\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 08:55:46.711997 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:46.711769 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnklm\" (UniqueName: \"kubernetes.io/projected/3af8589f-5f3d-4eb7-a997-2483276c2d1f-kube-api-access-lnklm\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 08:55:46.711997 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:46.711778 4562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3af8589f-5f3d-4eb7-a997-2483276c2d1f-console-config\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 08:55:46.711997 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:46.711786 4562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3af8589f-5f3d-4eb7-a997-2483276c2d1f-console-serving-cert\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 08:55:46.711997 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:46.711795 4562 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3af8589f-5f3d-4eb7-a997-2483276c2d1f-tmp\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 08:55:47.319784 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:47.319748 4562 generic.go:356] "Generic (PLEG): container finished" podID="3af8589f-5f3d-4eb7-a997-2483276c2d1f" containerID="e7bd9f263a4ca04ac679a760b1c9c0d6e7e9878d6605c52b4ad308415d61a199" exitCode=0 Oct 30 08:55:47.320215 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:47.319791 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cd4b79b88-hprw9" Oct 30 08:55:47.320215 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:47.319816 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cd4b79b88-hprw9" event={"ID":"3af8589f-5f3d-4eb7-a997-2483276c2d1f","Type":"ContainerDied","Data":"e7bd9f263a4ca04ac679a760b1c9c0d6e7e9878d6605c52b4ad308415d61a199"} Oct 30 08:55:47.320215 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:47.319856 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cd4b79b88-hprw9" event={"ID":"3af8589f-5f3d-4eb7-a997-2483276c2d1f","Type":"ContainerDied","Data":"66bcdd547ae506455a2c00618dd647cdcb06e61c57a6a3601390662a141e0ad3"} Oct 30 08:55:47.320215 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:47.319875 4562 scope.go:122] "RemoveContainer" containerID="e7bd9f263a4ca04ac679a760b1c9c0d6e7e9878d6605c52b4ad308415d61a199" Oct 30 08:55:47.327926 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:47.327908 4562 scope.go:122] "RemoveContainer" containerID="e7bd9f263a4ca04ac679a760b1c9c0d6e7e9878d6605c52b4ad308415d61a199" Oct 30 08:55:47.328170 ip-10-0-143-244 kubenswrapper[4562]: E1030 08:55:47.328148 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7bd9f263a4ca04ac679a760b1c9c0d6e7e9878d6605c52b4ad308415d61a199\": container with ID starting with e7bd9f263a4ca04ac679a760b1c9c0d6e7e9878d6605c52b4ad308415d61a199 not found: ID does not exist" containerID="e7bd9f263a4ca04ac679a760b1c9c0d6e7e9878d6605c52b4ad308415d61a199" Oct 30 08:55:47.328213 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:47.328180 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7bd9f263a4ca04ac679a760b1c9c0d6e7e9878d6605c52b4ad308415d61a199"} err="failed to get container status \"e7bd9f263a4ca04ac679a760b1c9c0d6e7e9878d6605c52b4ad308415d61a199\": rpc error: code = NotFound desc = could not find container \"e7bd9f263a4ca04ac679a760b1c9c0d6e7e9878d6605c52b4ad308415d61a199\": container with ID starting with e7bd9f263a4ca04ac679a760b1c9c0d6e7e9878d6605c52b4ad308415d61a199 not found: ID does not exist" Oct 30 08:55:47.376942 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:47.376916 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cd4b79b88-hprw9"] Oct 30 08:55:47.390184 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:47.390159 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-cd4b79b88-hprw9"] Oct 30 08:55:47.871600 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:55:47.871570 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3af8589f-5f3d-4eb7-a997-2483276c2d1f" path="/var/lib/kubelet/pods/3af8589f-5f3d-4eb7-a997-2483276c2d1f/volumes" Oct 30 08:55:58.594406 ip-10-0-143-244 kubenswrapper[4562]: E1030 08:55:58.594357 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 08:56:28.593667 ip-10-0-143-244 kubenswrapper[4562]: E1030 08:56:28.593583 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 08:56:50.526097 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:56:50.526060 4562 generic.go:356] "Generic (PLEG): container finished" podID="cf34fc37-dd3b-44c0-9d55-91ed73b00e7c" containerID="c3b3ab220ef67d7aab911b8a29ce53297830b7729fa57cbd0a92972d136edee5" exitCode=0 Oct 30 08:56:50.526477 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:56:50.526116 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-k8knv-9jwhj" event={"ID":"cf34fc37-dd3b-44c0-9d55-91ed73b00e7c","Type":"ContainerDied","Data":"c3b3ab220ef67d7aab911b8a29ce53297830b7729fa57cbd0a92972d136edee5"} Oct 30 08:56:51.654946 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:56:51.654918 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-k8knv-9jwhj" Oct 30 08:56:51.778496 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:56:51.778465 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf34fc37-dd3b-44c0-9d55-91ed73b00e7c-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/cf34fc37-dd3b-44c0-9d55-91ed73b00e7c-archives-path\") pod \"cf34fc37-dd3b-44c0-9d55-91ed73b00e7c\" (UID: \"cf34fc37-dd3b-44c0-9d55-91ed73b00e7c\") " Oct 30 08:56:51.778684 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:56:51.778503 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf34fc37-dd3b-44c0-9d55-91ed73b00e7c-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf34fc37-dd3b-44c0-9d55-91ed73b00e7c-trusted-ca-bundle\") pod \"cf34fc37-dd3b-44c0-9d55-91ed73b00e7c\" (UID: \"cf34fc37-dd3b-44c0-9d55-91ed73b00e7c\") " Oct 30 08:56:51.778684 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:56:51.778574 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf34fc37-dd3b-44c0-9d55-91ed73b00e7c-kube-api-access-w75rb\" (UniqueName: \"kubernetes.io/projected/cf34fc37-dd3b-44c0-9d55-91ed73b00e7c-kube-api-access-w75rb\") pod \"cf34fc37-dd3b-44c0-9d55-91ed73b00e7c\" (UID: \"cf34fc37-dd3b-44c0-9d55-91ed73b00e7c\") " Oct 30 08:56:51.778684 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:56:51.778595 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf34fc37-dd3b-44c0-9d55-91ed73b00e7c-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf34fc37-dd3b-44c0-9d55-91ed73b00e7c-service-ca-bundle\") pod \"cf34fc37-dd3b-44c0-9d55-91ed73b00e7c\" (UID: \"cf34fc37-dd3b-44c0-9d55-91ed73b00e7c\") " Oct 30 08:56:51.778975 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:56:51.778883 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf34fc37-dd3b-44c0-9d55-91ed73b00e7c-archives-path" pod "cf34fc37-dd3b-44c0-9d55-91ed73b00e7c" (UID: "cf34fc37-dd3b-44c0-9d55-91ed73b00e7c"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:56:51.779025 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:56:51.779005 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf34fc37-dd3b-44c0-9d55-91ed73b00e7c-trusted-ca-bundle" pod "cf34fc37-dd3b-44c0-9d55-91ed73b00e7c" (UID: "cf34fc37-dd3b-44c0-9d55-91ed73b00e7c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:56:51.779086 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:56:51.779065 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf34fc37-dd3b-44c0-9d55-91ed73b00e7c-service-ca-bundle" pod "cf34fc37-dd3b-44c0-9d55-91ed73b00e7c" (UID: "cf34fc37-dd3b-44c0-9d55-91ed73b00e7c"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:56:51.780855 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:56:51.780802 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf34fc37-dd3b-44c0-9d55-91ed73b00e7c-kube-api-access-w75rb" pod "cf34fc37-dd3b-44c0-9d55-91ed73b00e7c" (UID: "cf34fc37-dd3b-44c0-9d55-91ed73b00e7c"). InnerVolumeSpecName "kube-api-access-w75rb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:56:51.879130 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:56:51.879109 4562 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/cf34fc37-dd3b-44c0-9d55-91ed73b00e7c-archives-path\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 08:56:51.879130 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:56:51.879131 4562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf34fc37-dd3b-44c0-9d55-91ed73b00e7c-trusted-ca-bundle\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 08:56:51.879263 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:56:51.879141 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w75rb\" (UniqueName: \"kubernetes.io/projected/cf34fc37-dd3b-44c0-9d55-91ed73b00e7c-kube-api-access-w75rb\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 08:56:51.879263 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:56:51.879150 4562 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf34fc37-dd3b-44c0-9d55-91ed73b00e7c-service-ca-bundle\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 08:56:52.533954 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:56:52.533912 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-k8knv-9jwhj" event={"ID":"cf34fc37-dd3b-44c0-9d55-91ed73b00e7c","Type":"ContainerDied","Data":"c762e5557b27816afb001ddd48dbff991ab26ceafdeca21db70f16a7165e02bc"} Oct 30 08:56:52.533954 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:56:52.533950 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-k8knv-9jwhj" Oct 30 08:56:52.534155 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:56:52.533963 4562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c762e5557b27816afb001ddd48dbff991ab26ceafdeca21db70f16a7165e02bc" Oct 30 08:56:58.594246 ip-10-0-143-244 kubenswrapper[4562]: E1030 08:56:58.594196 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 08:57:28.593145 ip-10-0-143-244 kubenswrapper[4562]: E1030 08:57:28.593110 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 08:57:42.960427 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:57:42.960396 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-x7j7f"] Oct 30 08:57:42.960815 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:57:42.960640 4562 cpu_manager.go:414] "containerMap: removing container" podUID="210800be-29a2-40f4-a7de-f2c4ef9a8acd" containerName="console" Oct 30 08:57:42.960815 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:57:42.960650 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="210800be-29a2-40f4-a7de-f2c4ef9a8acd" containerName="console" Oct 30 08:57:42.960815 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:57:42.960665 4562 cpu_manager.go:414] "containerMap: removing container" podUID="3af8589f-5f3d-4eb7-a997-2483276c2d1f" containerName="console" Oct 30 08:57:42.960815 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:57:42.960670 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3af8589f-5f3d-4eb7-a997-2483276c2d1f" containerName="console" Oct 30 08:57:42.960815 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:57:42.960686 4562 cpu_manager.go:414] "containerMap: removing container" podUID="cf34fc37-dd3b-44c0-9d55-91ed73b00e7c" containerName="insights-gathering" Oct 30 08:57:42.960815 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:57:42.960692 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf34fc37-dd3b-44c0-9d55-91ed73b00e7c" containerName="insights-gathering" Oct 30 08:57:42.960815 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:57:42.960732 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="3af8589f-5f3d-4eb7-a997-2483276c2d1f" containerName="console" Oct 30 08:57:42.960815 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:57:42.960740 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf34fc37-dd3b-44c0-9d55-91ed73b00e7c" containerName="insights-gathering" Oct 30 08:57:42.960815 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:57:42.960747 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="210800be-29a2-40f4-a7de-f2c4ef9a8acd" containerName="console" Oct 30 08:57:42.962409 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:57:42.962372 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x7j7f" Oct 30 08:57:42.980943 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:57:42.980922 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 08:57:42.984846 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:57:42.984819 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 08:57:42.984961 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:57:42.984896 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 08:57:42.994271 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:57:42.994253 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 08:57:42.994501 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:57:42.994487 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 08:57:42.997986 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:57:42.997967 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-h44vk\"" Oct 30 08:57:42.998311 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:57:42.998297 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 08:57:42.999989 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:57:42.999969 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-x7j7f"] Oct 30 08:57:43.030738 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:57:43.030712 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a12daa38-7df2-4784-be09-c966271bf213-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-x7j7f\" (UID: \"a12daa38-7df2-4784-be09-c966271bf213\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x7j7f" Oct 30 08:57:43.030835 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:57:43.030746 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a12daa38-7df2-4784-be09-c966271bf213-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-x7j7f\" (UID: \"a12daa38-7df2-4784-be09-c966271bf213\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x7j7f" Oct 30 08:57:43.030835 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:57:43.030764 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a12daa38-7df2-4784-be09-c966271bf213-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-x7j7f\" (UID: \"a12daa38-7df2-4784-be09-c966271bf213\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x7j7f" Oct 30 08:57:43.030835 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:57:43.030779 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztg9s\" (UniqueName: \"kubernetes.io/projected/a12daa38-7df2-4784-be09-c966271bf213-kube-api-access-ztg9s\") pod \"istiod-openshift-gateway-7876bfcb74-x7j7f\" (UID: \"a12daa38-7df2-4784-be09-c966271bf213\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x7j7f" Oct 30 08:57:43.030948 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:57:43.030866 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a12daa38-7df2-4784-be09-c966271bf213-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-x7j7f\" (UID: \"a12daa38-7df2-4784-be09-c966271bf213\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x7j7f" Oct 30 08:57:43.030948 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:57:43.030911 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a12daa38-7df2-4784-be09-c966271bf213-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-x7j7f\" (UID: \"a12daa38-7df2-4784-be09-c966271bf213\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x7j7f" Oct 30 08:57:43.030948 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:57:43.030933 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a12daa38-7df2-4784-be09-c966271bf213-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-x7j7f\" (UID: \"a12daa38-7df2-4784-be09-c966271bf213\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x7j7f" Oct 30 08:57:43.131689 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:57:43.131655 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a12daa38-7df2-4784-be09-c966271bf213-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-x7j7f\" (UID: \"a12daa38-7df2-4784-be09-c966271bf213\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x7j7f" Oct 30 08:57:43.131824 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:57:43.131714 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a12daa38-7df2-4784-be09-c966271bf213-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-x7j7f\" (UID: \"a12daa38-7df2-4784-be09-c966271bf213\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x7j7f" Oct 30 08:57:43.131824 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:57:43.131742 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a12daa38-7df2-4784-be09-c966271bf213-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-x7j7f\" (UID: \"a12daa38-7df2-4784-be09-c966271bf213\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x7j7f" Oct 30 08:57:43.131824 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:57:43.131756 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a12daa38-7df2-4784-be09-c966271bf213-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-x7j7f\" (UID: \"a12daa38-7df2-4784-be09-c966271bf213\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x7j7f" Oct 30 08:57:43.131824 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:57:43.131771 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztg9s\" (UniqueName: \"kubernetes.io/projected/a12daa38-7df2-4784-be09-c966271bf213-kube-api-access-ztg9s\") pod \"istiod-openshift-gateway-7876bfcb74-x7j7f\" (UID: \"a12daa38-7df2-4784-be09-c966271bf213\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x7j7f" Oct 30 08:57:43.131824 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:57:43.131788 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a12daa38-7df2-4784-be09-c966271bf213-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-x7j7f\" (UID: \"a12daa38-7df2-4784-be09-c966271bf213\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x7j7f" Oct 30 08:57:43.131824 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:57:43.131813 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a12daa38-7df2-4784-be09-c966271bf213-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-x7j7f\" (UID: \"a12daa38-7df2-4784-be09-c966271bf213\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x7j7f" Oct 30 08:57:43.132515 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:57:43.132472 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a12daa38-7df2-4784-be09-c966271bf213-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-x7j7f\" (UID: \"a12daa38-7df2-4784-be09-c966271bf213\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x7j7f" Oct 30 08:57:43.134146 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:57:43.134124 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a12daa38-7df2-4784-be09-c966271bf213-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-x7j7f\" (UID: \"a12daa38-7df2-4784-be09-c966271bf213\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x7j7f" Oct 30 08:57:43.134240 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:57:43.134178 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a12daa38-7df2-4784-be09-c966271bf213-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-x7j7f\" (UID: \"a12daa38-7df2-4784-be09-c966271bf213\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x7j7f" Oct 30 08:57:43.134361 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:57:43.134343 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a12daa38-7df2-4784-be09-c966271bf213-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-x7j7f\" (UID: \"a12daa38-7df2-4784-be09-c966271bf213\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x7j7f" Oct 30 08:57:43.134573 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:57:43.134554 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a12daa38-7df2-4784-be09-c966271bf213-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-x7j7f\" (UID: \"a12daa38-7df2-4784-be09-c966271bf213\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x7j7f" Oct 30 08:57:43.151741 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:57:43.151720 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a12daa38-7df2-4784-be09-c966271bf213-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-x7j7f\" (UID: \"a12daa38-7df2-4784-be09-c966271bf213\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x7j7f" Oct 30 08:57:43.152445 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:57:43.152425 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztg9s\" (UniqueName: \"kubernetes.io/projected/a12daa38-7df2-4784-be09-c966271bf213-kube-api-access-ztg9s\") pod \"istiod-openshift-gateway-7876bfcb74-x7j7f\" (UID: \"a12daa38-7df2-4784-be09-c966271bf213\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x7j7f" Oct 30 08:57:43.272646 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:57:43.272541 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x7j7f" Oct 30 08:57:43.573692 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:57:43.573658 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-x7j7f"] Oct 30 08:57:43.574217 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:57:43.574187 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda12daa38_7df2_4784_be09_c966271bf213.slice/crio-78dec6dec4fa7bc26a19c2f7875d069884f681f5e80d537f6e7d1813b092d2b0 WatchSource:0}: Error finding container 78dec6dec4fa7bc26a19c2f7875d069884f681f5e80d537f6e7d1813b092d2b0: Status 404 returned error can't find the container with id 78dec6dec4fa7bc26a19c2f7875d069884f681f5e80d537f6e7d1813b092d2b0 Oct 30 08:57:43.706691 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:57:43.706659 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x7j7f" event={"ID":"a12daa38-7df2-4784-be09-c966271bf213","Type":"ContainerStarted","Data":"78dec6dec4fa7bc26a19c2f7875d069884f681f5e80d537f6e7d1813b092d2b0"} Oct 30 08:57:46.266902 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:57:46.266874 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 08:57:46.267144 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:57:46.266934 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 08:57:46.719637 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:57:46.719605 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x7j7f" event={"ID":"a12daa38-7df2-4784-be09-c966271bf213","Type":"ContainerStarted","Data":"567d2277c48021511dc524c60345e1580d612ce7b9554b8f973fa2163f3b5d6d"} Oct 30 08:57:46.719797 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:57:46.719720 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x7j7f" Oct 30 08:57:46.775586 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:57:46.775538 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x7j7f" podStartSLOduration=2.085466361 podStartE2EDuration="4.775525777s" totalImagesPullingTime="2.690059416s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:57:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:57:46.774684013 +0000 UTC m=+227.036245000" watchObservedRunningTime="2026-10-30 08:57:46.775525777 +0000 UTC m=+227.037086764" Oct 30 08:57:47.724637 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:57:47.724602 4562 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-x7j7f container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 08:57:47.724995 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:57:47.724660 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x7j7f" podUID="a12daa38-7df2-4784-be09-c966271bf213" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 08:57:50.724211 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:57:50.724182 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x7j7f" Oct 30 08:57:58.594122 ip-10-0-143-244 kubenswrapper[4562]: E1030 08:57:58.594087 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 08:58:23.471709 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:58:23.471674 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-5ttnd"] Oct 30 08:58:23.475077 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:58:23.475063 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-5ttnd" Oct 30 08:58:23.484924 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:58:23.484904 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-5ttnd"] Oct 30 08:58:23.585563 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:58:23.585533 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 08:58:23.585729 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:58:23.585638 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 08:58:23.585774 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:58:23.585720 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 08:58:23.585774 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:58:23.585748 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-95xdf\"" Oct 30 08:58:23.585774 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:58:23.585757 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 08:58:23.585861 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:58:23.585773 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 08:58:23.627291 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:58:23.627257 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/634844c2-e03e-499f-9dea-cd606cd6b67b-webhook-cert\") pod \"opendatahub-operator-controller-manager-69546f6877-5ttnd\" (UID: \"634844c2-e03e-499f-9dea-cd606cd6b67b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-5ttnd" Oct 30 08:58:23.627455 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:58:23.627297 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/634844c2-e03e-499f-9dea-cd606cd6b67b-manifests\") pod \"opendatahub-operator-controller-manager-69546f6877-5ttnd\" (UID: \"634844c2-e03e-499f-9dea-cd606cd6b67b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-5ttnd" Oct 30 08:58:23.627455 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:58:23.627320 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/634844c2-e03e-499f-9dea-cd606cd6b67b-metrics-cert\") pod \"opendatahub-operator-controller-manager-69546f6877-5ttnd\" (UID: \"634844c2-e03e-499f-9dea-cd606cd6b67b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-5ttnd" Oct 30 08:58:23.627455 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:58:23.627376 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/634844c2-e03e-499f-9dea-cd606cd6b67b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-69546f6877-5ttnd\" (UID: \"634844c2-e03e-499f-9dea-cd606cd6b67b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-5ttnd" Oct 30 08:58:23.627560 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:58:23.627482 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzmmh\" (UniqueName: \"kubernetes.io/projected/634844c2-e03e-499f-9dea-cd606cd6b67b-kube-api-access-nzmmh\") pod \"opendatahub-operator-controller-manager-69546f6877-5ttnd\" (UID: \"634844c2-e03e-499f-9dea-cd606cd6b67b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-5ttnd" Oct 30 08:58:23.627560 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:58:23.627512 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/634844c2-e03e-499f-9dea-cd606cd6b67b-tmp\") pod \"opendatahub-operator-controller-manager-69546f6877-5ttnd\" (UID: \"634844c2-e03e-499f-9dea-cd606cd6b67b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-5ttnd" Oct 30 08:58:23.727851 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:58:23.727774 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/634844c2-e03e-499f-9dea-cd606cd6b67b-webhook-cert\") pod \"opendatahub-operator-controller-manager-69546f6877-5ttnd\" (UID: \"634844c2-e03e-499f-9dea-cd606cd6b67b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-5ttnd" Oct 30 08:58:23.727851 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:58:23.727807 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/634844c2-e03e-499f-9dea-cd606cd6b67b-manifests\") pod \"opendatahub-operator-controller-manager-69546f6877-5ttnd\" (UID: \"634844c2-e03e-499f-9dea-cd606cd6b67b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-5ttnd" Oct 30 08:58:23.727851 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:58:23.727829 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/634844c2-e03e-499f-9dea-cd606cd6b67b-metrics-cert\") pod \"opendatahub-operator-controller-manager-69546f6877-5ttnd\" (UID: \"634844c2-e03e-499f-9dea-cd606cd6b67b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-5ttnd" Oct 30 08:58:23.727851 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:58:23.727848 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/634844c2-e03e-499f-9dea-cd606cd6b67b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-69546f6877-5ttnd\" (UID: \"634844c2-e03e-499f-9dea-cd606cd6b67b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-5ttnd" Oct 30 08:58:23.728126 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:58:23.727890 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzmmh\" (UniqueName: \"kubernetes.io/projected/634844c2-e03e-499f-9dea-cd606cd6b67b-kube-api-access-nzmmh\") pod \"opendatahub-operator-controller-manager-69546f6877-5ttnd\" (UID: \"634844c2-e03e-499f-9dea-cd606cd6b67b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-5ttnd" Oct 30 08:58:23.728126 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:58:23.727914 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/634844c2-e03e-499f-9dea-cd606cd6b67b-tmp\") pod \"opendatahub-operator-controller-manager-69546f6877-5ttnd\" (UID: \"634844c2-e03e-499f-9dea-cd606cd6b67b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-5ttnd" Oct 30 08:58:23.728276 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:58:23.728247 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/634844c2-e03e-499f-9dea-cd606cd6b67b-manifests\") pod \"opendatahub-operator-controller-manager-69546f6877-5ttnd\" (UID: \"634844c2-e03e-499f-9dea-cd606cd6b67b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-5ttnd" Oct 30 08:58:23.730327 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:58:23.730302 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/634844c2-e03e-499f-9dea-cd606cd6b67b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-69546f6877-5ttnd\" (UID: \"634844c2-e03e-499f-9dea-cd606cd6b67b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-5ttnd" Oct 30 08:58:23.730456 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:58:23.730325 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/634844c2-e03e-499f-9dea-cd606cd6b67b-webhook-cert\") pod \"opendatahub-operator-controller-manager-69546f6877-5ttnd\" (UID: \"634844c2-e03e-499f-9dea-cd606cd6b67b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-5ttnd" Oct 30 08:58:23.730537 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:58:23.730515 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/634844c2-e03e-499f-9dea-cd606cd6b67b-tmp\") pod \"opendatahub-operator-controller-manager-69546f6877-5ttnd\" (UID: \"634844c2-e03e-499f-9dea-cd606cd6b67b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-5ttnd" Oct 30 08:58:23.730839 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:58:23.730818 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/634844c2-e03e-499f-9dea-cd606cd6b67b-metrics-cert\") pod \"opendatahub-operator-controller-manager-69546f6877-5ttnd\" (UID: \"634844c2-e03e-499f-9dea-cd606cd6b67b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-5ttnd" Oct 30 08:58:23.748866 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:58:23.748839 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzmmh\" (UniqueName: \"kubernetes.io/projected/634844c2-e03e-499f-9dea-cd606cd6b67b-kube-api-access-nzmmh\") pod \"opendatahub-operator-controller-manager-69546f6877-5ttnd\" (UID: \"634844c2-e03e-499f-9dea-cd606cd6b67b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-5ttnd" Oct 30 08:58:23.789651 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:58:23.789626 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-5ttnd" Oct 30 08:58:24.159446 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:58:24.159415 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod634844c2_e03e_499f_9dea_cd606cd6b67b.slice/crio-1156e8020c03233f0bde0883b4fdb27badf406068ea32842961b6e4562066110 WatchSource:0}: Error finding container 1156e8020c03233f0bde0883b4fdb27badf406068ea32842961b6e4562066110: Status 404 returned error can't find the container with id 1156e8020c03233f0bde0883b4fdb27badf406068ea32842961b6e4562066110 Oct 30 08:58:24.159870 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:58:24.159845 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-5ttnd"] Oct 30 08:58:24.852620 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:58:24.852571 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-5ttnd" event={"ID":"634844c2-e03e-499f-9dea-cd606cd6b67b","Type":"ContainerStarted","Data":"1156e8020c03233f0bde0883b4fdb27badf406068ea32842961b6e4562066110"} Oct 30 08:58:27.869654 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:58:27.869624 4562 generic.go:356] "Generic (PLEG): container finished" podID="634844c2-e03e-499f-9dea-cd606cd6b67b" containerID="3f6ebe67f31b086ccc1a3679e6590c6f224e5f714159a8a78a23a761045cdf4c" exitCode=0 Oct 30 08:58:27.872518 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:58:27.872496 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-5ttnd" event={"ID":"634844c2-e03e-499f-9dea-cd606cd6b67b","Type":"ContainerDied","Data":"3f6ebe67f31b086ccc1a3679e6590c6f224e5f714159a8a78a23a761045cdf4c"} Oct 30 08:58:28.594752 ip-10-0-143-244 kubenswrapper[4562]: E1030 08:58:28.594666 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 08:58:28.875870 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:58:28.875785 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-5ttnd" event={"ID":"634844c2-e03e-499f-9dea-cd606cd6b67b","Type":"ContainerStarted","Data":"9d2df27c3a0cdfc41eba1c23494e3b3741fd760bbf9e778c76c6e71dd782584e"} Oct 30 08:58:28.876264 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:58:28.875921 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-5ttnd" Oct 30 08:58:28.971006 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:58:28.970941 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-5ttnd" podStartSLOduration=2.511000132 podStartE2EDuration="5.970921779s" totalImagesPullingTime="2.990718182s" totalInitContainerRuntime="469.203465ms" isStatefulPod=false podCreationTimestamp="2026-10-30 08:58:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:58:28.956679559 +0000 UTC m=+269.218240572" watchObservedRunningTime="2026-10-30 08:58:28.970921779 +0000 UTC m=+269.232482765" Oct 30 08:58:39.881589 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:58:39.881559 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69546f6877-5ttnd" Oct 30 08:58:58.098022 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:58:58.097988 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-767d5cbf5-vx9wj"] Oct 30 08:58:58.101237 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:58:58.101221 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-767d5cbf5-vx9wj" Oct 30 08:58:58.118585 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:58:58.118560 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-767d5cbf5-vx9wj"] Oct 30 08:58:58.189707 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:58:58.189665 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/993358dc-1271-474e-b372-5e30c394340d-tls-certs\") pod \"kube-auth-proxy-767d5cbf5-vx9wj\" (UID: \"993358dc-1271-474e-b372-5e30c394340d\") " pod="openshift-ingress/kube-auth-proxy-767d5cbf5-vx9wj" Oct 30 08:58:58.189879 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:58:58.189837 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x97w\" (UniqueName: \"kubernetes.io/projected/993358dc-1271-474e-b372-5e30c394340d-kube-api-access-8x97w\") pod \"kube-auth-proxy-767d5cbf5-vx9wj\" (UID: \"993358dc-1271-474e-b372-5e30c394340d\") " pod="openshift-ingress/kube-auth-proxy-767d5cbf5-vx9wj" Oct 30 08:58:58.189948 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:58:58.189884 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/993358dc-1271-474e-b372-5e30c394340d-tmp\") pod \"kube-auth-proxy-767d5cbf5-vx9wj\" (UID: \"993358dc-1271-474e-b372-5e30c394340d\") " pod="openshift-ingress/kube-auth-proxy-767d5cbf5-vx9wj" Oct 30 08:58:58.199065 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:58:58.199031 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 08:58:58.199232 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:58:58.199205 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 08:58:58.199350 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:58:58.199212 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-h299b\"" Oct 30 08:58:58.290356 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:58:58.290321 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/993358dc-1271-474e-b372-5e30c394340d-tmp\") pod \"kube-auth-proxy-767d5cbf5-vx9wj\" (UID: \"993358dc-1271-474e-b372-5e30c394340d\") " pod="openshift-ingress/kube-auth-proxy-767d5cbf5-vx9wj" Oct 30 08:58:58.290551 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:58:58.290410 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/993358dc-1271-474e-b372-5e30c394340d-tls-certs\") pod \"kube-auth-proxy-767d5cbf5-vx9wj\" (UID: \"993358dc-1271-474e-b372-5e30c394340d\") " pod="openshift-ingress/kube-auth-proxy-767d5cbf5-vx9wj" Oct 30 08:58:58.290551 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:58:58.290495 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x97w\" (UniqueName: \"kubernetes.io/projected/993358dc-1271-474e-b372-5e30c394340d-kube-api-access-8x97w\") pod \"kube-auth-proxy-767d5cbf5-vx9wj\" (UID: \"993358dc-1271-474e-b372-5e30c394340d\") " pod="openshift-ingress/kube-auth-proxy-767d5cbf5-vx9wj" Oct 30 08:58:58.292597 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:58:58.292570 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/993358dc-1271-474e-b372-5e30c394340d-tmp\") pod \"kube-auth-proxy-767d5cbf5-vx9wj\" (UID: \"993358dc-1271-474e-b372-5e30c394340d\") " pod="openshift-ingress/kube-auth-proxy-767d5cbf5-vx9wj" Oct 30 08:58:58.292856 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:58:58.292838 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/993358dc-1271-474e-b372-5e30c394340d-tls-certs\") pod \"kube-auth-proxy-767d5cbf5-vx9wj\" (UID: \"993358dc-1271-474e-b372-5e30c394340d\") " pod="openshift-ingress/kube-auth-proxy-767d5cbf5-vx9wj" Oct 30 08:58:58.314454 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:58:58.314420 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x97w\" (UniqueName: \"kubernetes.io/projected/993358dc-1271-474e-b372-5e30c394340d-kube-api-access-8x97w\") pod \"kube-auth-proxy-767d5cbf5-vx9wj\" (UID: \"993358dc-1271-474e-b372-5e30c394340d\") " pod="openshift-ingress/kube-auth-proxy-767d5cbf5-vx9wj" Oct 30 08:58:58.413918 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:58:58.413839 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-767d5cbf5-vx9wj" Oct 30 08:58:58.597970 ip-10-0-143-244 kubenswrapper[4562]: E1030 08:58:58.597930 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 08:58:58.707232 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:58:58.707201 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-767d5cbf5-vx9wj"] Oct 30 08:58:58.707770 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:58:58.707744 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod993358dc_1271_474e_b372_5e30c394340d.slice/crio-c3117060f8fd08106fceaca064967fcb3701dd36e180b7863df4404086fc3597 WatchSource:0}: Error finding container c3117060f8fd08106fceaca064967fcb3701dd36e180b7863df4404086fc3597: Status 404 returned error can't find the container with id c3117060f8fd08106fceaca064967fcb3701dd36e180b7863df4404086fc3597 Oct 30 08:58:58.981866 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:58:58.981790 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-767d5cbf5-vx9wj" event={"ID":"993358dc-1271-474e-b372-5e30c394340d","Type":"ContainerStarted","Data":"c3117060f8fd08106fceaca064967fcb3701dd36e180b7863df4404086fc3597"} Oct 30 08:58:59.888578 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:58:59.888543 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6q7n_1f51db5a-5720-4dea-ad02-f336b35d43b1/ovn-acl-logging/0.log" Oct 30 08:58:59.893361 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:58:59.893338 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6q7n_1f51db5a-5720-4dea-ad02-f336b35d43b1/ovn-acl-logging/0.log" Oct 30 08:59:01.111243 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:01.111210 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 08:59:01.995658 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:01.995624 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-767d5cbf5-vx9wj" event={"ID":"993358dc-1271-474e-b372-5e30c394340d","Type":"ContainerStarted","Data":"d6f25064101f55758701728b62f68d7981377626ab01fad76f47e38c50d0224d"} Oct 30 08:59:02.041269 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:02.041219 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-767d5cbf5-vx9wj" podStartSLOduration=1.6551781490000002 podStartE2EDuration="4.041204751s" totalImagesPullingTime="2.386026602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:58:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:59:02.040517894 +0000 UTC m=+302.302078882" watchObservedRunningTime="2026-10-30 08:59:02.041204751 +0000 UTC m=+302.302765741" Oct 30 08:59:28.593882 ip-10-0-143-244 kubenswrapper[4562]: E1030 08:59:28.593804 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 08:59:55.923427 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:55.923393 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nkb2b"] Oct 30 08:59:55.925666 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:55.925650 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nkb2b" Oct 30 08:59:55.950293 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:55.950257 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nkb2b"] Oct 30 08:59:55.962686 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:55.962662 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bc88a93-dfe9-4dad-b1a7-76d39693168c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nkb2b\" (UID: \"3bc88a93-dfe9-4dad-b1a7-76d39693168c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nkb2b" Oct 30 08:59:55.962817 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:55.962699 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3bc88a93-dfe9-4dad-b1a7-76d39693168c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nkb2b\" (UID: \"3bc88a93-dfe9-4dad-b1a7-76d39693168c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nkb2b" Oct 30 08:59:55.962817 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:55.962727 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brk42\" (UniqueName: \"kubernetes.io/projected/3bc88a93-dfe9-4dad-b1a7-76d39693168c-kube-api-access-brk42\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nkb2b\" (UID: \"3bc88a93-dfe9-4dad-b1a7-76d39693168c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nkb2b" Oct 30 08:59:55.962902 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:55.962823 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bc88a93-dfe9-4dad-b1a7-76d39693168c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nkb2b\" (UID: \"3bc88a93-dfe9-4dad-b1a7-76d39693168c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nkb2b" Oct 30 08:59:55.962902 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:55.962850 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3bc88a93-dfe9-4dad-b1a7-76d39693168c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nkb2b\" (UID: \"3bc88a93-dfe9-4dad-b1a7-76d39693168c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nkb2b" Oct 30 08:59:55.962902 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:55.962880 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3bc88a93-dfe9-4dad-b1a7-76d39693168c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nkb2b\" (UID: \"3bc88a93-dfe9-4dad-b1a7-76d39693168c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nkb2b" Oct 30 08:59:55.963003 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:55.962902 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3bc88a93-dfe9-4dad-b1a7-76d39693168c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nkb2b\" (UID: \"3bc88a93-dfe9-4dad-b1a7-76d39693168c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nkb2b" Oct 30 08:59:56.063813 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:56.063773 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bc88a93-dfe9-4dad-b1a7-76d39693168c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nkb2b\" (UID: \"3bc88a93-dfe9-4dad-b1a7-76d39693168c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nkb2b" Oct 30 08:59:56.063984 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:56.063825 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3bc88a93-dfe9-4dad-b1a7-76d39693168c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nkb2b\" (UID: \"3bc88a93-dfe9-4dad-b1a7-76d39693168c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nkb2b" Oct 30 08:59:56.063984 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:56.063850 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brk42\" (UniqueName: \"kubernetes.io/projected/3bc88a93-dfe9-4dad-b1a7-76d39693168c-kube-api-access-brk42\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nkb2b\" (UID: \"3bc88a93-dfe9-4dad-b1a7-76d39693168c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nkb2b" Oct 30 08:59:56.063984 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:56.063881 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bc88a93-dfe9-4dad-b1a7-76d39693168c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nkb2b\" (UID: \"3bc88a93-dfe9-4dad-b1a7-76d39693168c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nkb2b" Oct 30 08:59:56.063984 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:56.063933 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3bc88a93-dfe9-4dad-b1a7-76d39693168c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nkb2b\" (UID: \"3bc88a93-dfe9-4dad-b1a7-76d39693168c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nkb2b" Oct 30 08:59:56.063984 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:56.063976 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3bc88a93-dfe9-4dad-b1a7-76d39693168c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nkb2b\" (UID: \"3bc88a93-dfe9-4dad-b1a7-76d39693168c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nkb2b" Oct 30 08:59:56.064253 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:56.064005 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3bc88a93-dfe9-4dad-b1a7-76d39693168c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nkb2b\" (UID: \"3bc88a93-dfe9-4dad-b1a7-76d39693168c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nkb2b" Oct 30 08:59:56.064808 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:56.064778 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3bc88a93-dfe9-4dad-b1a7-76d39693168c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nkb2b\" (UID: \"3bc88a93-dfe9-4dad-b1a7-76d39693168c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nkb2b" Oct 30 08:59:56.066285 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:56.066253 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bc88a93-dfe9-4dad-b1a7-76d39693168c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nkb2b\" (UID: \"3bc88a93-dfe9-4dad-b1a7-76d39693168c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nkb2b" Oct 30 08:59:56.066406 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:56.066307 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3bc88a93-dfe9-4dad-b1a7-76d39693168c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nkb2b\" (UID: \"3bc88a93-dfe9-4dad-b1a7-76d39693168c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nkb2b" Oct 30 08:59:56.066474 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:56.066453 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3bc88a93-dfe9-4dad-b1a7-76d39693168c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nkb2b\" (UID: \"3bc88a93-dfe9-4dad-b1a7-76d39693168c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nkb2b" Oct 30 08:59:56.066510 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:56.066465 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3bc88a93-dfe9-4dad-b1a7-76d39693168c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nkb2b\" (UID: \"3bc88a93-dfe9-4dad-b1a7-76d39693168c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nkb2b" Oct 30 08:59:56.083777 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:56.083751 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bc88a93-dfe9-4dad-b1a7-76d39693168c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nkb2b\" (UID: \"3bc88a93-dfe9-4dad-b1a7-76d39693168c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nkb2b" Oct 30 08:59:56.084123 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:56.084105 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brk42\" (UniqueName: \"kubernetes.io/projected/3bc88a93-dfe9-4dad-b1a7-76d39693168c-kube-api-access-brk42\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nkb2b\" (UID: \"3bc88a93-dfe9-4dad-b1a7-76d39693168c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nkb2b" Oct 30 08:59:56.235411 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:56.235370 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nkb2b" Oct 30 08:59:56.539097 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:56.539072 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nkb2b"] Oct 30 08:59:56.540060 ip-10-0-143-244 kubenswrapper[4562]: W1030 08:59:56.540034 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bc88a93_dfe9_4dad_b1a7_76d39693168c.slice/crio-f28b8a3250dd70b956ab2c3121f3f8c6ba78e94fca139082f66e0a8af8fc3eab WatchSource:0}: Error finding container f28b8a3250dd70b956ab2c3121f3f8c6ba78e94fca139082f66e0a8af8fc3eab: Status 404 returned error can't find the container with id f28b8a3250dd70b956ab2c3121f3f8c6ba78e94fca139082f66e0a8af8fc3eab Oct 30 08:59:56.542048 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:56.542018 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 08:59:56.542107 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:56.542076 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 08:59:57.206466 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:57.206425 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nkb2b" event={"ID":"3bc88a93-dfe9-4dad-b1a7-76d39693168c","Type":"ContainerStarted","Data":"33aa9ea5f355359388e0cbb125f09149e4d7879848ec892d3c967d1d045eadef"} Oct 30 08:59:57.206941 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:57.206615 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nkb2b" Oct 30 08:59:57.206941 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:57.206640 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nkb2b" event={"ID":"3bc88a93-dfe9-4dad-b1a7-76d39693168c","Type":"ContainerStarted","Data":"f28b8a3250dd70b956ab2c3121f3f8c6ba78e94fca139082f66e0a8af8fc3eab"} Oct 30 08:59:57.208324 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:57.208295 4562 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-nkb2b container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 08:59:57.208485 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:57.208350 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nkb2b" podUID="3bc88a93-dfe9-4dad-b1a7-76d39693168c" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 08:59:57.250440 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:57.250364 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nkb2b" podStartSLOduration=2.250349829 podStartE2EDuration="2.250349829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:59:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:59:57.249529132 +0000 UTC m=+357.511090147" watchObservedRunningTime="2026-10-30 08:59:57.250349829 +0000 UTC m=+357.511910818" Oct 30 08:59:58.210908 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:58.210881 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nkb2b" Oct 30 08:59:58.294121 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:58.294093 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-x7j7f"] Oct 30 08:59:58.294297 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:58.294265 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x7j7f" podUID="a12daa38-7df2-4784-be09-c966271bf213" containerName="discovery" containerID="cri-o://567d2277c48021511dc524c60345e1580d612ce7b9554b8f973fa2163f3b5d6d" gracePeriod=30 Oct 30 08:59:58.531546 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:58.531519 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x7j7f" Oct 30 08:59:58.586843 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:58.586816 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a12daa38-7df2-4784-be09-c966271bf213-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a12daa38-7df2-4784-be09-c966271bf213-local-certs\") pod \"a12daa38-7df2-4784-be09-c966271bf213\" (UID: \"a12daa38-7df2-4784-be09-c966271bf213\") " Oct 30 08:59:58.587015 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:58.586866 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a12daa38-7df2-4784-be09-c966271bf213-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a12daa38-7df2-4784-be09-c966271bf213-istio-csr-dns-cert\") pod \"a12daa38-7df2-4784-be09-c966271bf213\" (UID: \"a12daa38-7df2-4784-be09-c966271bf213\") " Oct 30 08:59:58.587015 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:58.586903 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a12daa38-7df2-4784-be09-c966271bf213-istio-token\" (UniqueName: \"kubernetes.io/projected/a12daa38-7df2-4784-be09-c966271bf213-istio-token\") pod \"a12daa38-7df2-4784-be09-c966271bf213\" (UID: \"a12daa38-7df2-4784-be09-c966271bf213\") " Oct 30 08:59:58.587015 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:58.586924 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a12daa38-7df2-4784-be09-c966271bf213-kube-api-access-ztg9s\" (UniqueName: \"kubernetes.io/projected/a12daa38-7df2-4784-be09-c966271bf213-kube-api-access-ztg9s\") pod \"a12daa38-7df2-4784-be09-c966271bf213\" (UID: \"a12daa38-7df2-4784-be09-c966271bf213\") " Oct 30 08:59:58.587015 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:58.586960 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a12daa38-7df2-4784-be09-c966271bf213-cacerts\" (UniqueName: \"kubernetes.io/secret/a12daa38-7df2-4784-be09-c966271bf213-cacerts\") pod \"a12daa38-7df2-4784-be09-c966271bf213\" (UID: \"a12daa38-7df2-4784-be09-c966271bf213\") " Oct 30 08:59:58.587015 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:58.586996 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a12daa38-7df2-4784-be09-c966271bf213-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a12daa38-7df2-4784-be09-c966271bf213-istio-csr-ca-configmap\") pod \"a12daa38-7df2-4784-be09-c966271bf213\" (UID: \"a12daa38-7df2-4784-be09-c966271bf213\") " Oct 30 08:59:58.587300 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:58.587173 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a12daa38-7df2-4784-be09-c966271bf213-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a12daa38-7df2-4784-be09-c966271bf213-istio-kubeconfig\") pod \"a12daa38-7df2-4784-be09-c966271bf213\" (UID: \"a12daa38-7df2-4784-be09-c966271bf213\") " Oct 30 08:59:58.587664 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:58.587633 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a12daa38-7df2-4784-be09-c966271bf213-istio-csr-ca-configmap" pod "a12daa38-7df2-4784-be09-c966271bf213" (UID: "a12daa38-7df2-4784-be09-c966271bf213"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:59:58.589395 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:58.589343 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a12daa38-7df2-4784-be09-c966271bf213-local-certs" pod "a12daa38-7df2-4784-be09-c966271bf213" (UID: "a12daa38-7df2-4784-be09-c966271bf213"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:59:58.590340 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:58.590314 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a12daa38-7df2-4784-be09-c966271bf213-istio-token" pod "a12daa38-7df2-4784-be09-c966271bf213" (UID: "a12daa38-7df2-4784-be09-c966271bf213"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:59:58.590458 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:58.590359 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a12daa38-7df2-4784-be09-c966271bf213-istio-csr-dns-cert" pod "a12daa38-7df2-4784-be09-c966271bf213" (UID: "a12daa38-7df2-4784-be09-c966271bf213"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:59:58.590525 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:58.590480 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a12daa38-7df2-4784-be09-c966271bf213-kube-api-access-ztg9s" pod "a12daa38-7df2-4784-be09-c966271bf213" (UID: "a12daa38-7df2-4784-be09-c966271bf213"). InnerVolumeSpecName "kube-api-access-ztg9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:59:58.590595 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:58.590516 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a12daa38-7df2-4784-be09-c966271bf213-cacerts" pod "a12daa38-7df2-4784-be09-c966271bf213" (UID: "a12daa38-7df2-4784-be09-c966271bf213"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:59:58.590905 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:58.590880 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a12daa38-7df2-4784-be09-c966271bf213-istio-kubeconfig" pod "a12daa38-7df2-4784-be09-c966271bf213" (UID: "a12daa38-7df2-4784-be09-c966271bf213"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:59:58.596439 ip-10-0-143-244 kubenswrapper[4562]: E1030 08:59:58.596413 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 08:59:58.688736 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:58.688706 4562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a12daa38-7df2-4784-be09-c966271bf213-istio-token\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 08:59:58.688736 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:58.688731 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztg9s\" (UniqueName: \"kubernetes.io/projected/a12daa38-7df2-4784-be09-c966271bf213-kube-api-access-ztg9s\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 08:59:58.688736 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:58.688742 4562 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a12daa38-7df2-4784-be09-c966271bf213-cacerts\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 08:59:58.688929 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:58.688753 4562 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a12daa38-7df2-4784-be09-c966271bf213-istio-csr-ca-configmap\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 08:59:58.688929 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:58.688764 4562 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a12daa38-7df2-4784-be09-c966271bf213-istio-kubeconfig\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 08:59:58.688929 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:58.688772 4562 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a12daa38-7df2-4784-be09-c966271bf213-local-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 08:59:58.688929 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:58.688780 4562 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a12daa38-7df2-4784-be09-c966271bf213-istio-csr-dns-cert\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 08:59:59.214799 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:59.214763 4562 generic.go:356] "Generic (PLEG): container finished" podID="a12daa38-7df2-4784-be09-c966271bf213" containerID="567d2277c48021511dc524c60345e1580d612ce7b9554b8f973fa2163f3b5d6d" exitCode=0 Oct 30 08:59:59.215256 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:59.214810 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x7j7f" Oct 30 08:59:59.215256 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:59.214862 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x7j7f" event={"ID":"a12daa38-7df2-4784-be09-c966271bf213","Type":"ContainerDied","Data":"567d2277c48021511dc524c60345e1580d612ce7b9554b8f973fa2163f3b5d6d"} Oct 30 08:59:59.215256 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:59.214914 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-x7j7f" event={"ID":"a12daa38-7df2-4784-be09-c966271bf213","Type":"ContainerDied","Data":"78dec6dec4fa7bc26a19c2f7875d069884f681f5e80d537f6e7d1813b092d2b0"} Oct 30 08:59:59.215256 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:59.214936 4562 scope.go:122] "RemoveContainer" containerID="567d2277c48021511dc524c60345e1580d612ce7b9554b8f973fa2163f3b5d6d" Oct 30 08:59:59.224100 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:59.224083 4562 scope.go:122] "RemoveContainer" containerID="567d2277c48021511dc524c60345e1580d612ce7b9554b8f973fa2163f3b5d6d" Oct 30 08:59:59.224355 ip-10-0-143-244 kubenswrapper[4562]: E1030 08:59:59.224336 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"567d2277c48021511dc524c60345e1580d612ce7b9554b8f973fa2163f3b5d6d\": container with ID starting with 567d2277c48021511dc524c60345e1580d612ce7b9554b8f973fa2163f3b5d6d not found: ID does not exist" containerID="567d2277c48021511dc524c60345e1580d612ce7b9554b8f973fa2163f3b5d6d" Oct 30 08:59:59.224421 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:59.224362 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"567d2277c48021511dc524c60345e1580d612ce7b9554b8f973fa2163f3b5d6d"} err="failed to get container status \"567d2277c48021511dc524c60345e1580d612ce7b9554b8f973fa2163f3b5d6d\": rpc error: code = NotFound desc = could not find container \"567d2277c48021511dc524c60345e1580d612ce7b9554b8f973fa2163f3b5d6d\": container with ID starting with 567d2277c48021511dc524c60345e1580d612ce7b9554b8f973fa2163f3b5d6d not found: ID does not exist" Oct 30 08:59:59.282965 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:59.282943 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-x7j7f"] Oct 30 08:59:59.296199 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:59.296177 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-x7j7f"] Oct 30 08:59:59.872568 ip-10-0-143-244 kubenswrapper[4562]: I1030 08:59:59.872536 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a12daa38-7df2-4784-be09-c966271bf213" path="/var/lib/kubelet/pods/a12daa38-7df2-4784-be09-c966271bf213/volumes" Oct 30 09:00:28.593882 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:00:28.593843 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:00:58.594401 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:00:58.594342 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:01:28.593683 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:01:28.593648 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:01:58.594226 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:01:58.594186 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:02:28.593947 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:02:28.593864 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:02:58.596533 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:02:58.596489 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:03:28.594461 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:03:28.594419 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:03:58.594165 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:03:58.594089 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:03:59.919479 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:03:59.919449 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6q7n_1f51db5a-5720-4dea-ad02-f336b35d43b1/ovn-acl-logging/0.log" Oct 30 09:03:59.922867 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:03:59.922845 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6q7n_1f51db5a-5720-4dea-ad02-f336b35d43b1/ovn-acl-logging/0.log" Oct 30 09:04:28.593243 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:04:28.593207 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:04:58.593601 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:04:58.593555 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:05:28.594071 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:05:28.593992 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:05:58.593677 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:05:58.593640 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:06:28.594203 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:06:28.594166 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:06:58.594548 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:06:58.594469 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:07:28.592910 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:07:28.592872 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:07:58.593521 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:07:58.593484 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:08:28.594171 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:08:28.594132 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:08:58.594120 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:08:58.594081 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:08:59.940869 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:08:59.940844 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6q7n_1f51db5a-5720-4dea-ad02-f336b35d43b1/ovn-acl-logging/0.log" Oct 30 09:08:59.945571 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:08:59.945549 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6q7n_1f51db5a-5720-4dea-ad02-f336b35d43b1/ovn-acl-logging/0.log" Oct 30 09:09:28.593747 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:09:28.593712 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:09:37.345265 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:37.345236 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-wdbpk"] Oct 30 09:09:37.345781 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:37.345505 4562 cpu_manager.go:414] "containerMap: removing container" podUID="a12daa38-7df2-4784-be09-c966271bf213" containerName="discovery" Oct 30 09:09:37.345781 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:37.345516 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a12daa38-7df2-4784-be09-c966271bf213" containerName="discovery" Oct 30 09:09:37.345781 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:37.345571 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="a12daa38-7df2-4784-be09-c966271bf213" containerName="discovery" Oct 30 09:09:37.348622 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:37.348602 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-wdbpk" Oct 30 09:09:37.364339 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:37.364311 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-wdbpk"] Oct 30 09:09:37.375843 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:37.375819 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 09:09:37.375984 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:37.375887 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 09:09:37.375984 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:37.375948 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 09:09:37.376901 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:37.376875 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-2zmf2\"" Oct 30 09:09:37.420008 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:37.419980 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h257f\" (UniqueName: \"kubernetes.io/projected/378fef5c-408c-496b-ad0b-12361165a62c-kube-api-access-h257f\") pod \"limitador-limitador-74dfcfb894-wdbpk\" (UID: \"378fef5c-408c-496b-ad0b-12361165a62c\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-wdbpk" Oct 30 09:09:37.420177 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:37.420064 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/378fef5c-408c-496b-ad0b-12361165a62c-config-file\") pod \"limitador-limitador-74dfcfb894-wdbpk\" (UID: \"378fef5c-408c-496b-ad0b-12361165a62c\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-wdbpk" Oct 30 09:09:37.476177 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:37.476147 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-wdbpk"] Oct 30 09:09:37.521230 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:37.521199 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h257f\" (UniqueName: \"kubernetes.io/projected/378fef5c-408c-496b-ad0b-12361165a62c-kube-api-access-h257f\") pod \"limitador-limitador-74dfcfb894-wdbpk\" (UID: \"378fef5c-408c-496b-ad0b-12361165a62c\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-wdbpk" Oct 30 09:09:37.521411 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:37.521253 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/378fef5c-408c-496b-ad0b-12361165a62c-config-file\") pod \"limitador-limitador-74dfcfb894-wdbpk\" (UID: \"378fef5c-408c-496b-ad0b-12361165a62c\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-wdbpk" Oct 30 09:09:37.521886 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:37.521862 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/378fef5c-408c-496b-ad0b-12361165a62c-config-file\") pod \"limitador-limitador-74dfcfb894-wdbpk\" (UID: \"378fef5c-408c-496b-ad0b-12361165a62c\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-wdbpk" Oct 30 09:09:37.541333 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:37.541306 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h257f\" (UniqueName: \"kubernetes.io/projected/378fef5c-408c-496b-ad0b-12361165a62c-kube-api-access-h257f\") pod \"limitador-limitador-74dfcfb894-wdbpk\" (UID: \"378fef5c-408c-496b-ad0b-12361165a62c\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-wdbpk" Oct 30 09:09:37.663040 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:37.662981 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-wdbpk" Oct 30 09:09:37.956139 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:09:37.956111 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod378fef5c_408c_496b_ad0b_12361165a62c.slice/crio-a73069365d3bd7bede37e09b4a3f81fd8c6bbb296d1a4a9b5e1aebbb309ad997 WatchSource:0}: Error finding container a73069365d3bd7bede37e09b4a3f81fd8c6bbb296d1a4a9b5e1aebbb309ad997: Status 404 returned error can't find the container with id a73069365d3bd7bede37e09b4a3f81fd8c6bbb296d1a4a9b5e1aebbb309ad997 Oct 30 09:09:37.956595 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:37.956571 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-wdbpk"] Oct 30 09:09:37.958187 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:37.958169 4562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 09:09:38.207758 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:38.207681 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-9prmh"] Oct 30 09:09:38.211927 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:38.211907 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-9prmh" Oct 30 09:09:38.225005 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:38.224981 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-9prmh"] Oct 30 09:09:38.227993 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:38.227971 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-cqn5p\"" Oct 30 09:09:38.325844 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:38.325810 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-wdbpk" event={"ID":"378fef5c-408c-496b-ad0b-12361165a62c","Type":"ContainerStarted","Data":"a73069365d3bd7bede37e09b4a3f81fd8c6bbb296d1a4a9b5e1aebbb309ad997"} Oct 30 09:09:38.328123 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:38.328101 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzk6d\" (UniqueName: \"kubernetes.io/projected/60d51da4-3cd8-4e64-b7ed-f1760c1f82b7-kube-api-access-hzk6d\") pod \"authorino-57f7f8659f-9prmh\" (UID: \"60d51da4-3cd8-4e64-b7ed-f1760c1f82b7\") " pod="kuadrant-system/authorino-57f7f8659f-9prmh" Oct 30 09:09:38.429262 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:38.429233 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzk6d\" (UniqueName: \"kubernetes.io/projected/60d51da4-3cd8-4e64-b7ed-f1760c1f82b7-kube-api-access-hzk6d\") pod \"authorino-57f7f8659f-9prmh\" (UID: \"60d51da4-3cd8-4e64-b7ed-f1760c1f82b7\") " pod="kuadrant-system/authorino-57f7f8659f-9prmh" Oct 30 09:09:38.448829 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:38.448797 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzk6d\" (UniqueName: \"kubernetes.io/projected/60d51da4-3cd8-4e64-b7ed-f1760c1f82b7-kube-api-access-hzk6d\") pod \"authorino-57f7f8659f-9prmh\" (UID: \"60d51da4-3cd8-4e64-b7ed-f1760c1f82b7\") " pod="kuadrant-system/authorino-57f7f8659f-9prmh" Oct 30 09:09:38.535487 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:38.535422 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-9prmh" Oct 30 09:09:38.827134 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:38.826460 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-9prmh"] Oct 30 09:09:38.827945 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:09:38.827908 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod60d51da4_3cd8_4e64_b7ed_f1760c1f82b7.slice/crio-9d8fce583e9507c1400a59591b7b99dc04a9af80d0c7e1a113f37fc3e4af050e WatchSource:0}: Error finding container 9d8fce583e9507c1400a59591b7b99dc04a9af80d0c7e1a113f37fc3e4af050e: Status 404 returned error can't find the container with id 9d8fce583e9507c1400a59591b7b99dc04a9af80d0c7e1a113f37fc3e4af050e Oct 30 09:09:39.331948 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:39.331909 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-9prmh" event={"ID":"60d51da4-3cd8-4e64-b7ed-f1760c1f82b7","Type":"ContainerStarted","Data":"9d8fce583e9507c1400a59591b7b99dc04a9af80d0c7e1a113f37fc3e4af050e"} Oct 30 09:09:41.342838 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:41.342755 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-wdbpk" event={"ID":"378fef5c-408c-496b-ad0b-12361165a62c","Type":"ContainerStarted","Data":"09018293ee760dcb24f125b7735bc40f74c2ca41e59fc6ca1567618ec36480b6"} Oct 30 09:09:41.343287 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:41.342851 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-wdbpk" Oct 30 09:09:41.344236 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:41.344209 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-9prmh" event={"ID":"60d51da4-3cd8-4e64-b7ed-f1760c1f82b7","Type":"ContainerStarted","Data":"5fba260a0e6465f94ed145814ab0abd252bfd31f7877052b4d6a148da92167c5"} Oct 30 09:09:41.381910 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:41.381869 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-wdbpk" podStartSLOduration=1.951811183 podStartE2EDuration="4.381857306s" totalImagesPullingTime="2.430046123s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:09:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:09:41.381213728 +0000 UTC m=+941.642774716" watchObservedRunningTime="2026-10-30 09:09:41.381857306 +0000 UTC m=+941.643418294" Oct 30 09:09:41.418831 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:41.418792 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-9prmh" podStartSLOduration=1.248487223 podStartE2EDuration="3.418780475s" totalImagesPullingTime="2.170293252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:09:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:09:41.417329033 +0000 UTC m=+941.678890020" watchObservedRunningTime="2026-10-30 09:09:41.418780475 +0000 UTC m=+941.680341462" Oct 30 09:09:42.234272 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:42.234240 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-9prmh"] Oct 30 09:09:43.351508 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:43.351428 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-9prmh" podUID="60d51da4-3cd8-4e64-b7ed-f1760c1f82b7" containerName="authorino" containerID="cri-o://5fba260a0e6465f94ed145814ab0abd252bfd31f7877052b4d6a148da92167c5" gracePeriod=30 Oct 30 09:09:43.592926 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:43.592902 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-9prmh" Oct 30 09:09:43.670113 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:43.670043 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60d51da4-3cd8-4e64-b7ed-f1760c1f82b7-kube-api-access-hzk6d\" (UniqueName: \"kubernetes.io/projected/60d51da4-3cd8-4e64-b7ed-f1760c1f82b7-kube-api-access-hzk6d\") pod \"60d51da4-3cd8-4e64-b7ed-f1760c1f82b7\" (UID: \"60d51da4-3cd8-4e64-b7ed-f1760c1f82b7\") " Oct 30 09:09:43.672134 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:43.672110 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60d51da4-3cd8-4e64-b7ed-f1760c1f82b7-kube-api-access-hzk6d" pod "60d51da4-3cd8-4e64-b7ed-f1760c1f82b7" (UID: "60d51da4-3cd8-4e64-b7ed-f1760c1f82b7"). InnerVolumeSpecName "kube-api-access-hzk6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:09:43.770626 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:43.770599 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzk6d\" (UniqueName: \"kubernetes.io/projected/60d51da4-3cd8-4e64-b7ed-f1760c1f82b7-kube-api-access-hzk6d\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:09:44.355758 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:44.355725 4562 generic.go:356] "Generic (PLEG): container finished" podID="60d51da4-3cd8-4e64-b7ed-f1760c1f82b7" containerID="5fba260a0e6465f94ed145814ab0abd252bfd31f7877052b4d6a148da92167c5" exitCode=0 Oct 30 09:09:44.355758 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:44.355762 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-9prmh" event={"ID":"60d51da4-3cd8-4e64-b7ed-f1760c1f82b7","Type":"ContainerDied","Data":"5fba260a0e6465f94ed145814ab0abd252bfd31f7877052b4d6a148da92167c5"} Oct 30 09:09:44.356365 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:44.355769 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-9prmh" Oct 30 09:09:44.356365 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:44.355794 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-9prmh" event={"ID":"60d51da4-3cd8-4e64-b7ed-f1760c1f82b7","Type":"ContainerDied","Data":"9d8fce583e9507c1400a59591b7b99dc04a9af80d0c7e1a113f37fc3e4af050e"} Oct 30 09:09:44.356365 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:44.355837 4562 scope.go:122] "RemoveContainer" containerID="5fba260a0e6465f94ed145814ab0abd252bfd31f7877052b4d6a148da92167c5" Oct 30 09:09:44.363442 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:44.363419 4562 scope.go:122] "RemoveContainer" containerID="5fba260a0e6465f94ed145814ab0abd252bfd31f7877052b4d6a148da92167c5" Oct 30 09:09:44.363721 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:09:44.363697 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fba260a0e6465f94ed145814ab0abd252bfd31f7877052b4d6a148da92167c5\": container with ID starting with 5fba260a0e6465f94ed145814ab0abd252bfd31f7877052b4d6a148da92167c5 not found: ID does not exist" containerID="5fba260a0e6465f94ed145814ab0abd252bfd31f7877052b4d6a148da92167c5" Oct 30 09:09:44.363772 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:44.363732 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fba260a0e6465f94ed145814ab0abd252bfd31f7877052b4d6a148da92167c5"} err="failed to get container status \"5fba260a0e6465f94ed145814ab0abd252bfd31f7877052b4d6a148da92167c5\": rpc error: code = NotFound desc = could not find container \"5fba260a0e6465f94ed145814ab0abd252bfd31f7877052b4d6a148da92167c5\": container with ID starting with 5fba260a0e6465f94ed145814ab0abd252bfd31f7877052b4d6a148da92167c5 not found: ID does not exist" Oct 30 09:09:44.406022 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:44.405998 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-9prmh"] Oct 30 09:09:44.418964 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:44.418942 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-9prmh"] Oct 30 09:09:45.871442 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:45.871406 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60d51da4-3cd8-4e64-b7ed-f1760c1f82b7" path="/var/lib/kubelet/pods/60d51da4-3cd8-4e64-b7ed-f1760c1f82b7/volumes" Oct 30 09:09:51.421331 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:51.421298 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-wdbpk"] Oct 30 09:09:51.421719 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:51.421496 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-wdbpk" podUID="378fef5c-408c-496b-ad0b-12361165a62c" containerName="limitador" containerID="cri-o://09018293ee760dcb24f125b7735bc40f74c2ca41e59fc6ca1567618ec36480b6" gracePeriod=30 Oct 30 09:09:51.422126 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:51.422107 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-wdbpk" Oct 30 09:09:51.961983 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:51.961959 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-wdbpk" Oct 30 09:09:52.035838 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:52.035770 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/378fef5c-408c-496b-ad0b-12361165a62c-kube-api-access-h257f\" (UniqueName: \"kubernetes.io/projected/378fef5c-408c-496b-ad0b-12361165a62c-kube-api-access-h257f\") pod \"378fef5c-408c-496b-ad0b-12361165a62c\" (UID: \"378fef5c-408c-496b-ad0b-12361165a62c\") " Oct 30 09:09:52.035838 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:52.035826 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/378fef5c-408c-496b-ad0b-12361165a62c-config-file\" (UniqueName: \"kubernetes.io/configmap/378fef5c-408c-496b-ad0b-12361165a62c-config-file\") pod \"378fef5c-408c-496b-ad0b-12361165a62c\" (UID: \"378fef5c-408c-496b-ad0b-12361165a62c\") " Oct 30 09:09:52.036159 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:52.036137 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/378fef5c-408c-496b-ad0b-12361165a62c-config-file" pod "378fef5c-408c-496b-ad0b-12361165a62c" (UID: "378fef5c-408c-496b-ad0b-12361165a62c"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:09:52.037779 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:52.037754 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/378fef5c-408c-496b-ad0b-12361165a62c-kube-api-access-h257f" pod "378fef5c-408c-496b-ad0b-12361165a62c" (UID: "378fef5c-408c-496b-ad0b-12361165a62c"). InnerVolumeSpecName "kube-api-access-h257f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:09:52.137394 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:52.137348 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h257f\" (UniqueName: \"kubernetes.io/projected/378fef5c-408c-496b-ad0b-12361165a62c-kube-api-access-h257f\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:09:52.137394 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:52.137374 4562 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/378fef5c-408c-496b-ad0b-12361165a62c-config-file\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:09:52.387799 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:52.387714 4562 generic.go:356] "Generic (PLEG): container finished" podID="378fef5c-408c-496b-ad0b-12361165a62c" containerID="09018293ee760dcb24f125b7735bc40f74c2ca41e59fc6ca1567618ec36480b6" exitCode=0 Oct 30 09:09:52.387799 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:52.387764 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-wdbpk" Oct 30 09:09:52.388070 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:52.387805 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-wdbpk" event={"ID":"378fef5c-408c-496b-ad0b-12361165a62c","Type":"ContainerDied","Data":"09018293ee760dcb24f125b7735bc40f74c2ca41e59fc6ca1567618ec36480b6"} Oct 30 09:09:52.388070 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:52.387856 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-wdbpk" event={"ID":"378fef5c-408c-496b-ad0b-12361165a62c","Type":"ContainerDied","Data":"a73069365d3bd7bede37e09b4a3f81fd8c6bbb296d1a4a9b5e1aebbb309ad997"} Oct 30 09:09:52.388070 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:52.387878 4562 scope.go:122] "RemoveContainer" containerID="09018293ee760dcb24f125b7735bc40f74c2ca41e59fc6ca1567618ec36480b6" Oct 30 09:09:52.396440 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:52.396421 4562 scope.go:122] "RemoveContainer" containerID="09018293ee760dcb24f125b7735bc40f74c2ca41e59fc6ca1567618ec36480b6" Oct 30 09:09:52.396696 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:09:52.396677 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09018293ee760dcb24f125b7735bc40f74c2ca41e59fc6ca1567618ec36480b6\": container with ID starting with 09018293ee760dcb24f125b7735bc40f74c2ca41e59fc6ca1567618ec36480b6 not found: ID does not exist" containerID="09018293ee760dcb24f125b7735bc40f74c2ca41e59fc6ca1567618ec36480b6" Oct 30 09:09:52.396748 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:52.396704 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09018293ee760dcb24f125b7735bc40f74c2ca41e59fc6ca1567618ec36480b6"} err="failed to get container status \"09018293ee760dcb24f125b7735bc40f74c2ca41e59fc6ca1567618ec36480b6\": rpc error: code = NotFound desc = could not find container \"09018293ee760dcb24f125b7735bc40f74c2ca41e59fc6ca1567618ec36480b6\": container with ID starting with 09018293ee760dcb24f125b7735bc40f74c2ca41e59fc6ca1567618ec36480b6 not found: ID does not exist" Oct 30 09:09:52.438457 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:52.438430 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-wdbpk"] Oct 30 09:09:52.451592 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:52.451572 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-wdbpk"] Oct 30 09:09:53.753028 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:53.752993 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-jc4mv"] Oct 30 09:09:53.753394 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:53.753277 4562 cpu_manager.go:414] "containerMap: removing container" podUID="378fef5c-408c-496b-ad0b-12361165a62c" containerName="limitador" Oct 30 09:09:53.753394 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:53.753287 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="378fef5c-408c-496b-ad0b-12361165a62c" containerName="limitador" Oct 30 09:09:53.753394 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:53.753304 4562 cpu_manager.go:414] "containerMap: removing container" podUID="60d51da4-3cd8-4e64-b7ed-f1760c1f82b7" containerName="authorino" Oct 30 09:09:53.753394 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:53.753309 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60d51da4-3cd8-4e64-b7ed-f1760c1f82b7" containerName="authorino" Oct 30 09:09:53.753394 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:53.753361 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="378fef5c-408c-496b-ad0b-12361165a62c" containerName="limitador" Oct 30 09:09:53.753394 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:53.753369 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="60d51da4-3cd8-4e64-b7ed-f1760c1f82b7" containerName="authorino" Oct 30 09:09:53.757345 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:53.757329 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jc4mv" Oct 30 09:09:53.771703 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:53.771679 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-jc4mv"] Oct 30 09:09:53.783408 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:53.783372 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 09:09:53.783522 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:53.783434 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 09:09:53.841877 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:53.841834 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Oct 30 09:09:53.842020 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:53.841922 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-ppgtt\"" Oct 30 09:09:53.853222 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:53.853199 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/6276e9c0-0ed3-4eb3-83b2-0d64a4e3763b-data\") pod \"postgres-7cdbf6dc44-jc4mv\" (UID: \"6276e9c0-0ed3-4eb3-83b2-0d64a4e3763b\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jc4mv" Oct 30 09:09:53.853307 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:53.853235 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dq7r\" (UniqueName: \"kubernetes.io/projected/6276e9c0-0ed3-4eb3-83b2-0d64a4e3763b-kube-api-access-9dq7r\") pod \"postgres-7cdbf6dc44-jc4mv\" (UID: \"6276e9c0-0ed3-4eb3-83b2-0d64a4e3763b\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jc4mv" Oct 30 09:09:53.875798 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:53.875770 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="378fef5c-408c-496b-ad0b-12361165a62c" path="/var/lib/kubelet/pods/378fef5c-408c-496b-ad0b-12361165a62c/volumes" Oct 30 09:09:53.954185 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:53.954154 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/6276e9c0-0ed3-4eb3-83b2-0d64a4e3763b-data\") pod \"postgres-7cdbf6dc44-jc4mv\" (UID: \"6276e9c0-0ed3-4eb3-83b2-0d64a4e3763b\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jc4mv" Oct 30 09:09:53.954318 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:53.954207 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dq7r\" (UniqueName: \"kubernetes.io/projected/6276e9c0-0ed3-4eb3-83b2-0d64a4e3763b-kube-api-access-9dq7r\") pod \"postgres-7cdbf6dc44-jc4mv\" (UID: \"6276e9c0-0ed3-4eb3-83b2-0d64a4e3763b\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jc4mv" Oct 30 09:09:53.954543 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:53.954521 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/6276e9c0-0ed3-4eb3-83b2-0d64a4e3763b-data\") pod \"postgres-7cdbf6dc44-jc4mv\" (UID: \"6276e9c0-0ed3-4eb3-83b2-0d64a4e3763b\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jc4mv" Oct 30 09:09:53.974227 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:53.974199 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dq7r\" (UniqueName: \"kubernetes.io/projected/6276e9c0-0ed3-4eb3-83b2-0d64a4e3763b-kube-api-access-9dq7r\") pod \"postgres-7cdbf6dc44-jc4mv\" (UID: \"6276e9c0-0ed3-4eb3-83b2-0d64a4e3763b\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jc4mv" Oct 30 09:09:54.069964 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:54.069898 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jc4mv" Oct 30 09:09:54.361046 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:54.361016 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-jc4mv"] Oct 30 09:09:54.361906 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:09:54.361884 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6276e9c0_0ed3_4eb3_83b2_0d64a4e3763b.slice/crio-b01d36fe98738d5577c352ef5b6562c8f4f5fb40007690ad05d7d5f637210b4e WatchSource:0}: Error finding container b01d36fe98738d5577c352ef5b6562c8f4f5fb40007690ad05d7d5f637210b4e: Status 404 returned error can't find the container with id b01d36fe98738d5577c352ef5b6562c8f4f5fb40007690ad05d7d5f637210b4e Oct 30 09:09:54.398179 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:54.398155 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jc4mv" event={"ID":"6276e9c0-0ed3-4eb3-83b2-0d64a4e3763b","Type":"ContainerStarted","Data":"b01d36fe98738d5577c352ef5b6562c8f4f5fb40007690ad05d7d5f637210b4e"} Oct 30 09:09:58.597743 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:09:58.597704 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:09:59.420454 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:59.420368 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jc4mv" event={"ID":"6276e9c0-0ed3-4eb3-83b2-0d64a4e3763b","Type":"ContainerStarted","Data":"4580e51f48ab06ca15f30976917587c94aecf254acc6c699c11bccb130ff1189"} Oct 30 09:09:59.420624 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:59.420524 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jc4mv" Oct 30 09:09:59.459061 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:09:59.459014 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jc4mv" podStartSLOduration=1.659001928 podStartE2EDuration="6.459002001s" totalImagesPullingTime="4.800000073s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:09:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:09:59.458928818 +0000 UTC m=+959.720489806" watchObservedRunningTime="2026-10-30 09:09:59.459002001 +0000 UTC m=+959.720562989" Oct 30 09:10:05.453905 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:05.453868 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jc4mv" Oct 30 09:10:12.506713 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:12.506679 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-786557dc8d-4rrhf"] Oct 30 09:10:12.527185 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:12.527161 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-786557dc8d-4rrhf" Oct 30 09:10:12.532079 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:12.532050 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-786557dc8d-4rrhf"] Oct 30 09:10:12.543267 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:12.543243 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 09:10:12.543267 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:12.543265 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 09:10:12.543512 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:12.543495 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 09:10:12.543598 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:12.543547 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 09:10:12.543670 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:12.543649 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 09:10:12.556297 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:12.556273 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-2gt44\"" Oct 30 09:10:12.698615 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:12.698576 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/012138f5-db63-4f9e-8f22-5f3588997114-webhook-cert\") pod \"maas-controller-786557dc8d-4rrhf\" (UID: \"012138f5-db63-4f9e-8f22-5f3588997114\") " pod="opendatahub/maas-controller-786557dc8d-4rrhf" Oct 30 09:10:12.698788 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:12.698646 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcp2n\" (UniqueName: \"kubernetes.io/projected/012138f5-db63-4f9e-8f22-5f3588997114-kube-api-access-rcp2n\") pod \"maas-controller-786557dc8d-4rrhf\" (UID: \"012138f5-db63-4f9e-8f22-5f3588997114\") " pod="opendatahub/maas-controller-786557dc8d-4rrhf" Oct 30 09:10:12.698788 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:12.698738 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/012138f5-db63-4f9e-8f22-5f3588997114-metrics-certs\") pod \"maas-controller-786557dc8d-4rrhf\" (UID: \"012138f5-db63-4f9e-8f22-5f3588997114\") " pod="opendatahub/maas-controller-786557dc8d-4rrhf" Oct 30 09:10:12.799364 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:12.799298 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/012138f5-db63-4f9e-8f22-5f3588997114-webhook-cert\") pod \"maas-controller-786557dc8d-4rrhf\" (UID: \"012138f5-db63-4f9e-8f22-5f3588997114\") " pod="opendatahub/maas-controller-786557dc8d-4rrhf" Oct 30 09:10:12.799364 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:12.799351 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcp2n\" (UniqueName: \"kubernetes.io/projected/012138f5-db63-4f9e-8f22-5f3588997114-kube-api-access-rcp2n\") pod \"maas-controller-786557dc8d-4rrhf\" (UID: \"012138f5-db63-4f9e-8f22-5f3588997114\") " pod="opendatahub/maas-controller-786557dc8d-4rrhf" Oct 30 09:10:12.799606 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:12.799374 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/012138f5-db63-4f9e-8f22-5f3588997114-metrics-certs\") pod \"maas-controller-786557dc8d-4rrhf\" (UID: \"012138f5-db63-4f9e-8f22-5f3588997114\") " pod="opendatahub/maas-controller-786557dc8d-4rrhf" Oct 30 09:10:12.801766 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:12.801736 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/012138f5-db63-4f9e-8f22-5f3588997114-metrics-certs\") pod \"maas-controller-786557dc8d-4rrhf\" (UID: \"012138f5-db63-4f9e-8f22-5f3588997114\") " pod="opendatahub/maas-controller-786557dc8d-4rrhf" Oct 30 09:10:12.801873 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:12.801802 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/012138f5-db63-4f9e-8f22-5f3588997114-webhook-cert\") pod \"maas-controller-786557dc8d-4rrhf\" (UID: \"012138f5-db63-4f9e-8f22-5f3588997114\") " pod="opendatahub/maas-controller-786557dc8d-4rrhf" Oct 30 09:10:12.823430 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:12.823404 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcp2n\" (UniqueName: \"kubernetes.io/projected/012138f5-db63-4f9e-8f22-5f3588997114-kube-api-access-rcp2n\") pod \"maas-controller-786557dc8d-4rrhf\" (UID: \"012138f5-db63-4f9e-8f22-5f3588997114\") " pod="opendatahub/maas-controller-786557dc8d-4rrhf" Oct 30 09:10:12.841245 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:12.841225 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-786557dc8d-4rrhf" Oct 30 09:10:13.004271 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:13.004235 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-687fd9c8d8-gt7tl"] Oct 30 09:10:13.007256 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:13.007239 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-687fd9c8d8-gt7tl" Oct 30 09:10:13.024179 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:13.024155 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-687fd9c8d8-gt7tl"] Oct 30 09:10:13.103246 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:13.103187 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/625507b9-d06d-4071-b33a-b959a6e3129a-metrics-certs\") pod \"maas-controller-687fd9c8d8-gt7tl\" (UID: \"625507b9-d06d-4071-b33a-b959a6e3129a\") " pod="opendatahub/maas-controller-687fd9c8d8-gt7tl" Oct 30 09:10:13.103246 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:13.103230 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdl7k\" (UniqueName: \"kubernetes.io/projected/625507b9-d06d-4071-b33a-b959a6e3129a-kube-api-access-bdl7k\") pod \"maas-controller-687fd9c8d8-gt7tl\" (UID: \"625507b9-d06d-4071-b33a-b959a6e3129a\") " pod="opendatahub/maas-controller-687fd9c8d8-gt7tl" Oct 30 09:10:13.103246 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:13.103248 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/625507b9-d06d-4071-b33a-b959a6e3129a-webhook-cert\") pod \"maas-controller-687fd9c8d8-gt7tl\" (UID: \"625507b9-d06d-4071-b33a-b959a6e3129a\") " pod="opendatahub/maas-controller-687fd9c8d8-gt7tl" Oct 30 09:10:13.128839 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:13.128817 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-786557dc8d-4rrhf"] Oct 30 09:10:13.128923 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:10:13.128834 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod012138f5_db63_4f9e_8f22_5f3588997114.slice/crio-4edd84b874af8010c0862b84029a302deca89c58b94d68ebf5ec41fb22a489eb WatchSource:0}: Error finding container 4edd84b874af8010c0862b84029a302deca89c58b94d68ebf5ec41fb22a489eb: Status 404 returned error can't find the container with id 4edd84b874af8010c0862b84029a302deca89c58b94d68ebf5ec41fb22a489eb Oct 30 09:10:13.204681 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:13.204656 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/625507b9-d06d-4071-b33a-b959a6e3129a-metrics-certs\") pod \"maas-controller-687fd9c8d8-gt7tl\" (UID: \"625507b9-d06d-4071-b33a-b959a6e3129a\") " pod="opendatahub/maas-controller-687fd9c8d8-gt7tl" Oct 30 09:10:13.204792 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:13.204697 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdl7k\" (UniqueName: \"kubernetes.io/projected/625507b9-d06d-4071-b33a-b959a6e3129a-kube-api-access-bdl7k\") pod \"maas-controller-687fd9c8d8-gt7tl\" (UID: \"625507b9-d06d-4071-b33a-b959a6e3129a\") " pod="opendatahub/maas-controller-687fd9c8d8-gt7tl" Oct 30 09:10:13.204792 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:13.204724 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/625507b9-d06d-4071-b33a-b959a6e3129a-webhook-cert\") pod \"maas-controller-687fd9c8d8-gt7tl\" (UID: \"625507b9-d06d-4071-b33a-b959a6e3129a\") " pod="opendatahub/maas-controller-687fd9c8d8-gt7tl" Oct 30 09:10:13.207104 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:13.207081 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/625507b9-d06d-4071-b33a-b959a6e3129a-metrics-certs\") pod \"maas-controller-687fd9c8d8-gt7tl\" (UID: \"625507b9-d06d-4071-b33a-b959a6e3129a\") " pod="opendatahub/maas-controller-687fd9c8d8-gt7tl" Oct 30 09:10:13.207188 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:13.207119 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/625507b9-d06d-4071-b33a-b959a6e3129a-webhook-cert\") pod \"maas-controller-687fd9c8d8-gt7tl\" (UID: \"625507b9-d06d-4071-b33a-b959a6e3129a\") " pod="opendatahub/maas-controller-687fd9c8d8-gt7tl" Oct 30 09:10:13.225141 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:13.225120 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdl7k\" (UniqueName: \"kubernetes.io/projected/625507b9-d06d-4071-b33a-b959a6e3129a-kube-api-access-bdl7k\") pod \"maas-controller-687fd9c8d8-gt7tl\" (UID: \"625507b9-d06d-4071-b33a-b959a6e3129a\") " pod="opendatahub/maas-controller-687fd9c8d8-gt7tl" Oct 30 09:10:13.319128 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:13.319102 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-687fd9c8d8-gt7tl" Oct 30 09:10:13.472339 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:13.472301 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-786557dc8d-4rrhf" event={"ID":"012138f5-db63-4f9e-8f22-5f3588997114","Type":"ContainerStarted","Data":"4edd84b874af8010c0862b84029a302deca89c58b94d68ebf5ec41fb22a489eb"} Oct 30 09:10:13.608668 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:13.608640 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-687fd9c8d8-gt7tl"] Oct 30 09:10:13.609034 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:10:13.609007 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod625507b9_d06d_4071_b33a_b959a6e3129a.slice/crio-d7207ece1bdd1f12f5c37a930fcb565d44d6211b23f68039f344dc12e23d18ec WatchSource:0}: Error finding container d7207ece1bdd1f12f5c37a930fcb565d44d6211b23f68039f344dc12e23d18ec: Status 404 returned error can't find the container with id d7207ece1bdd1f12f5c37a930fcb565d44d6211b23f68039f344dc12e23d18ec Oct 30 09:10:14.481582 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:14.481544 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-687fd9c8d8-gt7tl" event={"ID":"625507b9-d06d-4071-b33a-b959a6e3129a","Type":"ContainerStarted","Data":"d7207ece1bdd1f12f5c37a930fcb565d44d6211b23f68039f344dc12e23d18ec"} Oct 30 09:10:16.491266 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:16.491233 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-786557dc8d-4rrhf" event={"ID":"012138f5-db63-4f9e-8f22-5f3588997114","Type":"ContainerStarted","Data":"c595af04a2e94cafc8a92fe1d99b7b9095bde7ffc5b70992211ee42811f0ef10"} Oct 30 09:10:16.491745 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:16.491319 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-786557dc8d-4rrhf" Oct 30 09:10:16.492887 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:16.492861 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-687fd9c8d8-gt7tl" event={"ID":"625507b9-d06d-4071-b33a-b959a6e3129a","Type":"ContainerStarted","Data":"08d59a0a4c0f5f4d69a497ed1b972498b09d9f58fbfef3e2bc0ad8033a2d4bdc"} Oct 30 09:10:16.493012 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:16.492983 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-687fd9c8d8-gt7tl" Oct 30 09:10:16.532117 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:16.532073 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-786557dc8d-4rrhf" podStartSLOduration=1.8959431169999998 podStartE2EDuration="4.532061149s" totalImagesPullingTime="2.636118032s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:10:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:10:16.531726711 +0000 UTC m=+976.793287700" watchObservedRunningTime="2026-10-30 09:10:16.532061149 +0000 UTC m=+976.793622137" Oct 30 09:10:16.572543 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:16.572498 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-687fd9c8d8-gt7tl" podStartSLOduration=2.041252873 podStartE2EDuration="4.572486132s" totalImagesPullingTime="2.531233259s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:10:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:10:16.571691501 +0000 UTC m=+976.833252491" watchObservedRunningTime="2026-10-30 09:10:16.572486132 +0000 UTC m=+976.834047120" Oct 30 09:10:17.767931 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:17.767900 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gv64t"] Oct 30 09:10:17.770555 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:17.770532 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gv64t" Oct 30 09:10:17.801452 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:17.801367 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Oct 30 09:10:17.801628 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:17.801487 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Oct 30 09:10:17.802817 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:17.802798 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gv64t"] Oct 30 09:10:17.877253 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:17.877222 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Oct 30 09:10:17.877253 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:17.877238 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-wdspr\"" Oct 30 09:10:17.951042 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:17.951012 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a9a30ca0-995e-4e77-bf92-b3923ecc14c0-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-gv64t\" (UID: \"a9a30ca0-995e-4e77-bf92-b3923ecc14c0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gv64t" Oct 30 09:10:17.951042 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:17.951049 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a9a30ca0-995e-4e77-bf92-b3923ecc14c0-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-gv64t\" (UID: \"a9a30ca0-995e-4e77-bf92-b3923ecc14c0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gv64t" Oct 30 09:10:17.951253 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:17.951081 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp69f\" (UniqueName: \"kubernetes.io/projected/a9a30ca0-995e-4e77-bf92-b3923ecc14c0-kube-api-access-sp69f\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-gv64t\" (UID: \"a9a30ca0-995e-4e77-bf92-b3923ecc14c0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gv64t" Oct 30 09:10:17.951253 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:17.951236 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/a9a30ca0-995e-4e77-bf92-b3923ecc14c0-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-gv64t\" (UID: \"a9a30ca0-995e-4e77-bf92-b3923ecc14c0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gv64t" Oct 30 09:10:18.052472 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:18.052373 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/a9a30ca0-995e-4e77-bf92-b3923ecc14c0-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-gv64t\" (UID: \"a9a30ca0-995e-4e77-bf92-b3923ecc14c0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gv64t" Oct 30 09:10:18.052472 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:18.052442 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a9a30ca0-995e-4e77-bf92-b3923ecc14c0-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-gv64t\" (UID: \"a9a30ca0-995e-4e77-bf92-b3923ecc14c0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gv64t" Oct 30 09:10:18.052472 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:18.052463 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a9a30ca0-995e-4e77-bf92-b3923ecc14c0-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-gv64t\" (UID: \"a9a30ca0-995e-4e77-bf92-b3923ecc14c0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gv64t" Oct 30 09:10:18.052747 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:18.052482 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp69f\" (UniqueName: \"kubernetes.io/projected/a9a30ca0-995e-4e77-bf92-b3923ecc14c0-kube-api-access-sp69f\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-gv64t\" (UID: \"a9a30ca0-995e-4e77-bf92-b3923ecc14c0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gv64t" Oct 30 09:10:18.053135 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:18.053108 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a9a30ca0-995e-4e77-bf92-b3923ecc14c0-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-gv64t\" (UID: \"a9a30ca0-995e-4e77-bf92-b3923ecc14c0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gv64t" Oct 30 09:10:18.053267 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:18.053247 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/a9a30ca0-995e-4e77-bf92-b3923ecc14c0-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-gv64t\" (UID: \"a9a30ca0-995e-4e77-bf92-b3923ecc14c0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gv64t" Oct 30 09:10:18.054855 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:18.054839 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a9a30ca0-995e-4e77-bf92-b3923ecc14c0-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-gv64t\" (UID: \"a9a30ca0-995e-4e77-bf92-b3923ecc14c0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gv64t" Oct 30 09:10:18.071576 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:18.071557 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp69f\" (UniqueName: \"kubernetes.io/projected/a9a30ca0-995e-4e77-bf92-b3923ecc14c0-kube-api-access-sp69f\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-gv64t\" (UID: \"a9a30ca0-995e-4e77-bf92-b3923ecc14c0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gv64t" Oct 30 09:10:18.082410 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:18.082391 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gv64t" Oct 30 09:10:18.372252 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:10:18.372221 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9a30ca0_995e_4e77_bf92_b3923ecc14c0.slice/crio-a4e5b2d387538900ec94f252bd006597aabda96e6572d94670f8f64e680d3e5a WatchSource:0}: Error finding container a4e5b2d387538900ec94f252bd006597aabda96e6572d94670f8f64e680d3e5a: Status 404 returned error can't find the container with id a4e5b2d387538900ec94f252bd006597aabda96e6572d94670f8f64e680d3e5a Oct 30 09:10:18.372585 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:18.372560 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gv64t"] Oct 30 09:10:18.506899 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:18.506851 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gv64t" event={"ID":"a9a30ca0-995e-4e77-bf92-b3923ecc14c0","Type":"ContainerStarted","Data":"a4e5b2d387538900ec94f252bd006597aabda96e6572d94670f8f64e680d3e5a"} Oct 30 09:10:27.499052 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:27.499015 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-687fd9c8d8-gt7tl" Oct 30 09:10:27.499529 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:27.499119 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-786557dc8d-4rrhf" Oct 30 09:10:27.566297 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:27.566244 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-786557dc8d-4rrhf"] Oct 30 09:10:27.566532 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:27.566496 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-786557dc8d-4rrhf" podUID="012138f5-db63-4f9e-8f22-5f3588997114" containerName="manager" containerID="cri-o://c595af04a2e94cafc8a92fe1d99b7b9095bde7ffc5b70992211ee42811f0ef10" gracePeriod=30 Oct 30 09:10:27.824325 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:27.824295 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-786557dc8d-4rrhf" Oct 30 09:10:27.834278 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:27.834252 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/012138f5-db63-4f9e-8f22-5f3588997114-metrics-certs\" (UniqueName: \"kubernetes.io/secret/012138f5-db63-4f9e-8f22-5f3588997114-metrics-certs\") pod \"012138f5-db63-4f9e-8f22-5f3588997114\" (UID: \"012138f5-db63-4f9e-8f22-5f3588997114\") " Oct 30 09:10:27.834446 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:27.834312 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/012138f5-db63-4f9e-8f22-5f3588997114-webhook-cert\" (UniqueName: \"kubernetes.io/secret/012138f5-db63-4f9e-8f22-5f3588997114-webhook-cert\") pod \"012138f5-db63-4f9e-8f22-5f3588997114\" (UID: \"012138f5-db63-4f9e-8f22-5f3588997114\") " Oct 30 09:10:27.834446 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:27.834435 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/012138f5-db63-4f9e-8f22-5f3588997114-kube-api-access-rcp2n\" (UniqueName: \"kubernetes.io/projected/012138f5-db63-4f9e-8f22-5f3588997114-kube-api-access-rcp2n\") pod \"012138f5-db63-4f9e-8f22-5f3588997114\" (UID: \"012138f5-db63-4f9e-8f22-5f3588997114\") " Oct 30 09:10:27.836792 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:27.836757 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/012138f5-db63-4f9e-8f22-5f3588997114-metrics-certs" pod "012138f5-db63-4f9e-8f22-5f3588997114" (UID: "012138f5-db63-4f9e-8f22-5f3588997114"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:10:27.836916 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:27.836793 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/012138f5-db63-4f9e-8f22-5f3588997114-kube-api-access-rcp2n" pod "012138f5-db63-4f9e-8f22-5f3588997114" (UID: "012138f5-db63-4f9e-8f22-5f3588997114"). InnerVolumeSpecName "kube-api-access-rcp2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:10:27.836916 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:27.836861 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/012138f5-db63-4f9e-8f22-5f3588997114-webhook-cert" pod "012138f5-db63-4f9e-8f22-5f3588997114" (UID: "012138f5-db63-4f9e-8f22-5f3588997114"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:10:27.935514 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:27.935479 4562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/012138f5-db63-4f9e-8f22-5f3588997114-metrics-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:10:27.935514 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:27.935516 4562 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/012138f5-db63-4f9e-8f22-5f3588997114-webhook-cert\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:10:27.935717 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:27.935533 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcp2n\" (UniqueName: \"kubernetes.io/projected/012138f5-db63-4f9e-8f22-5f3588997114-kube-api-access-rcp2n\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:10:28.557152 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:28.557116 4562 generic.go:356] "Generic (PLEG): container finished" podID="012138f5-db63-4f9e-8f22-5f3588997114" containerID="c595af04a2e94cafc8a92fe1d99b7b9095bde7ffc5b70992211ee42811f0ef10" exitCode=0 Oct 30 09:10:28.557633 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:28.557163 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-786557dc8d-4rrhf" Oct 30 09:10:28.557633 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:28.557200 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-786557dc8d-4rrhf" event={"ID":"012138f5-db63-4f9e-8f22-5f3588997114","Type":"ContainerDied","Data":"c595af04a2e94cafc8a92fe1d99b7b9095bde7ffc5b70992211ee42811f0ef10"} Oct 30 09:10:28.557633 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:28.557246 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-786557dc8d-4rrhf" event={"ID":"012138f5-db63-4f9e-8f22-5f3588997114","Type":"ContainerDied","Data":"4edd84b874af8010c0862b84029a302deca89c58b94d68ebf5ec41fb22a489eb"} Oct 30 09:10:28.557633 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:28.557269 4562 scope.go:122] "RemoveContainer" containerID="c595af04a2e94cafc8a92fe1d99b7b9095bde7ffc5b70992211ee42811f0ef10" Oct 30 09:10:28.565048 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:28.565030 4562 scope.go:122] "RemoveContainer" containerID="c595af04a2e94cafc8a92fe1d99b7b9095bde7ffc5b70992211ee42811f0ef10" Oct 30 09:10:28.565289 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:10:28.565270 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c595af04a2e94cafc8a92fe1d99b7b9095bde7ffc5b70992211ee42811f0ef10\": container with ID starting with c595af04a2e94cafc8a92fe1d99b7b9095bde7ffc5b70992211ee42811f0ef10 not found: ID does not exist" containerID="c595af04a2e94cafc8a92fe1d99b7b9095bde7ffc5b70992211ee42811f0ef10" Oct 30 09:10:28.565358 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:28.565303 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c595af04a2e94cafc8a92fe1d99b7b9095bde7ffc5b70992211ee42811f0ef10"} err="failed to get container status \"c595af04a2e94cafc8a92fe1d99b7b9095bde7ffc5b70992211ee42811f0ef10\": rpc error: code = NotFound desc = could not find container \"c595af04a2e94cafc8a92fe1d99b7b9095bde7ffc5b70992211ee42811f0ef10\": container with ID starting with c595af04a2e94cafc8a92fe1d99b7b9095bde7ffc5b70992211ee42811f0ef10 not found: ID does not exist" Oct 30 09:10:28.593290 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:10:28.593258 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:10:28.608969 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:28.608939 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-786557dc8d-4rrhf"] Oct 30 09:10:28.621881 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:28.621857 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-786557dc8d-4rrhf"] Oct 30 09:10:29.562911 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:29.562876 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gv64t" event={"ID":"a9a30ca0-995e-4e77-bf92-b3923ecc14c0","Type":"ContainerStarted","Data":"ccd49f3ef020e9e42c78b1c4734f1df3b27f76aa5d81eb86df9ff9528ff2db9f"} Oct 30 09:10:29.563302 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:29.562993 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gv64t" Oct 30 09:10:29.603729 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:29.603682 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gv64t" podStartSLOduration=2.324869284 podStartE2EDuration="12.603669725s" totalImagesPullingTime="10.278800441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:10:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:10:29.602590806 +0000 UTC m=+989.864151804" watchObservedRunningTime="2026-10-30 09:10:29.603669725 +0000 UTC m=+989.865230712" Oct 30 09:10:29.872168 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:29.872098 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="012138f5-db63-4f9e-8f22-5f3588997114" path="/var/lib/kubelet/pods/012138f5-db63-4f9e-8f22-5f3588997114/volumes" Oct 30 09:10:40.572190 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:40.572158 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-gv64t" Oct 30 09:10:46.778417 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:46.778360 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-868774c7cf-kjms7"] Oct 30 09:10:46.778901 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:46.778673 4562 cpu_manager.go:414] "containerMap: removing container" podUID="012138f5-db63-4f9e-8f22-5f3588997114" containerName="manager" Oct 30 09:10:46.778901 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:46.778683 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="012138f5-db63-4f9e-8f22-5f3588997114" containerName="manager" Oct 30 09:10:46.778901 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:46.778756 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="012138f5-db63-4f9e-8f22-5f3588997114" containerName="manager" Oct 30 09:10:46.782665 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:46.782639 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-868774c7cf-kjms7" Oct 30 09:10:46.799053 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:46.799029 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-868774c7cf-kjms7"] Oct 30 09:10:46.810484 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:46.810451 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 09:10:46.810484 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:46.810468 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-hx9dk\"" Oct 30 09:10:46.810691 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:46.810538 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 09:10:46.883772 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:46.883743 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b03fcec5-026e-4c18-ae6b-6ffc896a0759-openshift-service-ca\") pod \"maas-api-868774c7cf-kjms7\" (UID: \"b03fcec5-026e-4c18-ae6b-6ffc896a0759\") " pod="odh-ai-gateway-infra/maas-api-868774c7cf-kjms7" Oct 30 09:10:46.883904 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:46.883818 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bcsv\" (UniqueName: \"kubernetes.io/projected/b03fcec5-026e-4c18-ae6b-6ffc896a0759-kube-api-access-5bcsv\") pod \"maas-api-868774c7cf-kjms7\" (UID: \"b03fcec5-026e-4c18-ae6b-6ffc896a0759\") " pod="odh-ai-gateway-infra/maas-api-868774c7cf-kjms7" Oct 30 09:10:46.883904 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:46.883846 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b03fcec5-026e-4c18-ae6b-6ffc896a0759-metrics-certs\") pod \"maas-api-868774c7cf-kjms7\" (UID: \"b03fcec5-026e-4c18-ae6b-6ffc896a0759\") " pod="odh-ai-gateway-infra/maas-api-868774c7cf-kjms7" Oct 30 09:10:46.883904 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:46.883876 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b03fcec5-026e-4c18-ae6b-6ffc896a0759-maas-api-tls\") pod \"maas-api-868774c7cf-kjms7\" (UID: \"b03fcec5-026e-4c18-ae6b-6ffc896a0759\") " pod="odh-ai-gateway-infra/maas-api-868774c7cf-kjms7" Oct 30 09:10:46.984815 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:46.984783 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b03fcec5-026e-4c18-ae6b-6ffc896a0759-maas-api-tls\") pod \"maas-api-868774c7cf-kjms7\" (UID: \"b03fcec5-026e-4c18-ae6b-6ffc896a0759\") " pod="odh-ai-gateway-infra/maas-api-868774c7cf-kjms7" Oct 30 09:10:46.984979 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:46.984821 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b03fcec5-026e-4c18-ae6b-6ffc896a0759-openshift-service-ca\") pod \"maas-api-868774c7cf-kjms7\" (UID: \"b03fcec5-026e-4c18-ae6b-6ffc896a0759\") " pod="odh-ai-gateway-infra/maas-api-868774c7cf-kjms7" Oct 30 09:10:46.984979 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:46.984866 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bcsv\" (UniqueName: \"kubernetes.io/projected/b03fcec5-026e-4c18-ae6b-6ffc896a0759-kube-api-access-5bcsv\") pod \"maas-api-868774c7cf-kjms7\" (UID: \"b03fcec5-026e-4c18-ae6b-6ffc896a0759\") " pod="odh-ai-gateway-infra/maas-api-868774c7cf-kjms7" Oct 30 09:10:46.984979 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:46.984885 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b03fcec5-026e-4c18-ae6b-6ffc896a0759-metrics-certs\") pod \"maas-api-868774c7cf-kjms7\" (UID: \"b03fcec5-026e-4c18-ae6b-6ffc896a0759\") " pod="odh-ai-gateway-infra/maas-api-868774c7cf-kjms7" Oct 30 09:10:46.985598 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:46.985572 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b03fcec5-026e-4c18-ae6b-6ffc896a0759-openshift-service-ca\") pod \"maas-api-868774c7cf-kjms7\" (UID: \"b03fcec5-026e-4c18-ae6b-6ffc896a0759\") " pod="odh-ai-gateway-infra/maas-api-868774c7cf-kjms7" Oct 30 09:10:46.987324 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:46.987299 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b03fcec5-026e-4c18-ae6b-6ffc896a0759-metrics-certs\") pod \"maas-api-868774c7cf-kjms7\" (UID: \"b03fcec5-026e-4c18-ae6b-6ffc896a0759\") " pod="odh-ai-gateway-infra/maas-api-868774c7cf-kjms7" Oct 30 09:10:46.987447 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:46.987358 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b03fcec5-026e-4c18-ae6b-6ffc896a0759-maas-api-tls\") pod \"maas-api-868774c7cf-kjms7\" (UID: \"b03fcec5-026e-4c18-ae6b-6ffc896a0759\") " pod="odh-ai-gateway-infra/maas-api-868774c7cf-kjms7" Oct 30 09:10:47.003901 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:47.003882 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bcsv\" (UniqueName: \"kubernetes.io/projected/b03fcec5-026e-4c18-ae6b-6ffc896a0759-kube-api-access-5bcsv\") pod \"maas-api-868774c7cf-kjms7\" (UID: \"b03fcec5-026e-4c18-ae6b-6ffc896a0759\") " pod="odh-ai-gateway-infra/maas-api-868774c7cf-kjms7" Oct 30 09:10:47.094576 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:47.094510 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-868774c7cf-kjms7" Oct 30 09:10:47.386022 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:47.385988 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-868774c7cf-kjms7"] Oct 30 09:10:47.386510 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:10:47.386487 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb03fcec5_026e_4c18_ae6b_6ffc896a0759.slice/crio-20999a0a9fd4b3e61d3b3911303fd3e2e11472e47797173791ba503f0550ca81 WatchSource:0}: Error finding container 20999a0a9fd4b3e61d3b3911303fd3e2e11472e47797173791ba503f0550ca81: Status 404 returned error can't find the container with id 20999a0a9fd4b3e61d3b3911303fd3e2e11472e47797173791ba503f0550ca81 Oct 30 09:10:47.636030 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:47.635949 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-868774c7cf-kjms7" event={"ID":"b03fcec5-026e-4c18-ae6b-6ffc896a0759","Type":"ContainerStarted","Data":"20999a0a9fd4b3e61d3b3911303fd3e2e11472e47797173791ba503f0550ca81"} Oct 30 09:10:49.646737 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:49.646695 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-868774c7cf-kjms7" event={"ID":"b03fcec5-026e-4c18-ae6b-6ffc896a0759","Type":"ContainerStarted","Data":"aa3ae72247d0391fdfa0cf6d93a8af09f0bea225996c876131c6e9e155cd00ae"} Oct 30 09:10:49.647112 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:49.646771 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-868774c7cf-kjms7" Oct 30 09:10:49.686900 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:49.686853 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-868774c7cf-kjms7" podStartSLOduration=2.211192532 podStartE2EDuration="3.686839463s" totalImagesPullingTime="1.475646931s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:10:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:10:49.686044692 +0000 UTC m=+1009.947605683" watchObservedRunningTime="2026-10-30 09:10:49.686839463 +0000 UTC m=+1009.948400451" Oct 30 09:10:55.656043 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:55.656015 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-868774c7cf-kjms7" Oct 30 09:10:56.426810 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:56.426778 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-pkb2p"] Oct 30 09:10:56.770447 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:56.770412 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-pkb2p"] Oct 30 09:10:56.770845 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:56.770524 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-pkb2p" Oct 30 09:10:56.770845 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:56.770539 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-pkb2p"] Oct 30 09:10:56.770845 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:56.770569 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-p6vws"] Oct 30 09:10:56.770948 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:10:56.770857 4562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-z978p], unattached volumes=[], failed to process volumes=[kube-api-access-z978p]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-pkb2p" podUID="cb46c6ce-84c4-43df-ba03-53a7374e30eb" Oct 30 09:10:56.788669 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:56.788646 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-p6vws"] Oct 30 09:10:56.788765 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:56.788713 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-p6vws" Oct 30 09:10:56.804767 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:56.804740 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 30 09:10:56.817799 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:56.817768 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 09:10:56.817799 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:56.817788 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-cqn5p\"" Oct 30 09:10:56.817946 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:56.817856 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 09:10:56.820409 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:56.820373 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-p6vws"] Oct 30 09:10:56.825080 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:10:56.825059 4562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-fdbph tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6889cbd8c4-p6vws" podUID="d4f6caa7-a6cd-4d42-98fd-febb12b034eb" Oct 30 09:10:56.865270 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:56.865243 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z978p\" (UniqueName: \"kubernetes.io/projected/cb46c6ce-84c4-43df-ba03-53a7374e30eb-kube-api-access-z978p\") pod \"authorino-6f6944bcdd-pkb2p\" (UID: \"cb46c6ce-84c4-43df-ba03-53a7374e30eb\") " pod="kuadrant-system/authorino-6f6944bcdd-pkb2p" Oct 30 09:10:56.865409 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:56.865284 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d4f6caa7-a6cd-4d42-98fd-febb12b034eb-tls-cert\") pod \"authorino-6889cbd8c4-p6vws\" (UID: \"d4f6caa7-a6cd-4d42-98fd-febb12b034eb\") " pod="kuadrant-system/authorino-6889cbd8c4-p6vws" Oct 30 09:10:56.865500 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:56.865479 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdbph\" (UniqueName: \"kubernetes.io/projected/d4f6caa7-a6cd-4d42-98fd-febb12b034eb-kube-api-access-fdbph\") pod \"authorino-6889cbd8c4-p6vws\" (UID: \"d4f6caa7-a6cd-4d42-98fd-febb12b034eb\") " pod="kuadrant-system/authorino-6889cbd8c4-p6vws" Oct 30 09:10:56.966596 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:56.966566 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdbph\" (UniqueName: \"kubernetes.io/projected/d4f6caa7-a6cd-4d42-98fd-febb12b034eb-kube-api-access-fdbph\") pod \"authorino-6889cbd8c4-p6vws\" (UID: \"d4f6caa7-a6cd-4d42-98fd-febb12b034eb\") " pod="kuadrant-system/authorino-6889cbd8c4-p6vws" Oct 30 09:10:56.966756 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:56.966609 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z978p\" (UniqueName: \"kubernetes.io/projected/cb46c6ce-84c4-43df-ba03-53a7374e30eb-kube-api-access-z978p\") pod \"authorino-6f6944bcdd-pkb2p\" (UID: \"cb46c6ce-84c4-43df-ba03-53a7374e30eb\") " pod="kuadrant-system/authorino-6f6944bcdd-pkb2p" Oct 30 09:10:56.966756 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:56.966625 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d4f6caa7-a6cd-4d42-98fd-febb12b034eb-tls-cert\") pod \"authorino-6889cbd8c4-p6vws\" (UID: \"d4f6caa7-a6cd-4d42-98fd-febb12b034eb\") " pod="kuadrant-system/authorino-6889cbd8c4-p6vws" Oct 30 09:10:56.968949 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:56.968931 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d4f6caa7-a6cd-4d42-98fd-febb12b034eb-tls-cert\") pod \"authorino-6889cbd8c4-p6vws\" (UID: \"d4f6caa7-a6cd-4d42-98fd-febb12b034eb\") " pod="kuadrant-system/authorino-6889cbd8c4-p6vws" Oct 30 09:10:56.986050 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:56.986025 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdbph\" (UniqueName: \"kubernetes.io/projected/d4f6caa7-a6cd-4d42-98fd-febb12b034eb-kube-api-access-fdbph\") pod \"authorino-6889cbd8c4-p6vws\" (UID: \"d4f6caa7-a6cd-4d42-98fd-febb12b034eb\") " pod="kuadrant-system/authorino-6889cbd8c4-p6vws" Oct 30 09:10:56.986282 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:56.986261 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z978p\" (UniqueName: \"kubernetes.io/projected/cb46c6ce-84c4-43df-ba03-53a7374e30eb-kube-api-access-z978p\") pod \"authorino-6f6944bcdd-pkb2p\" (UID: \"cb46c6ce-84c4-43df-ba03-53a7374e30eb\") " pod="kuadrant-system/authorino-6f6944bcdd-pkb2p" Oct 30 09:10:57.676406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:57.676362 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-pkb2p" Oct 30 09:10:57.676733 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:57.676373 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-p6vws" Oct 30 09:10:57.700577 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:57.700500 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-pkb2p" Oct 30 09:10:57.716563 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:57.716541 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-p6vws" Oct 30 09:10:57.772538 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:57.772510 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4f6caa7-a6cd-4d42-98fd-febb12b034eb-tls-cert\" (UniqueName: \"kubernetes.io/secret/d4f6caa7-a6cd-4d42-98fd-febb12b034eb-tls-cert\") pod \"d4f6caa7-a6cd-4d42-98fd-febb12b034eb\" (UID: \"d4f6caa7-a6cd-4d42-98fd-febb12b034eb\") " Oct 30 09:10:57.772859 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:57.772554 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb46c6ce-84c4-43df-ba03-53a7374e30eb-kube-api-access-z978p\" (UniqueName: \"kubernetes.io/projected/cb46c6ce-84c4-43df-ba03-53a7374e30eb-kube-api-access-z978p\") pod \"cb46c6ce-84c4-43df-ba03-53a7374e30eb\" (UID: \"cb46c6ce-84c4-43df-ba03-53a7374e30eb\") " Oct 30 09:10:57.772859 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:57.772622 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4f6caa7-a6cd-4d42-98fd-febb12b034eb-kube-api-access-fdbph\" (UniqueName: \"kubernetes.io/projected/d4f6caa7-a6cd-4d42-98fd-febb12b034eb-kube-api-access-fdbph\") pod \"d4f6caa7-a6cd-4d42-98fd-febb12b034eb\" (UID: \"d4f6caa7-a6cd-4d42-98fd-febb12b034eb\") " Oct 30 09:10:57.774670 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:57.774643 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4f6caa7-a6cd-4d42-98fd-febb12b034eb-tls-cert" pod "d4f6caa7-a6cd-4d42-98fd-febb12b034eb" (UID: "d4f6caa7-a6cd-4d42-98fd-febb12b034eb"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:10:57.774769 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:57.774674 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb46c6ce-84c4-43df-ba03-53a7374e30eb-kube-api-access-z978p" pod "cb46c6ce-84c4-43df-ba03-53a7374e30eb" (UID: "cb46c6ce-84c4-43df-ba03-53a7374e30eb"). InnerVolumeSpecName "kube-api-access-z978p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:10:57.774769 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:57.774711 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4f6caa7-a6cd-4d42-98fd-febb12b034eb-kube-api-access-fdbph" pod "d4f6caa7-a6cd-4d42-98fd-febb12b034eb" (UID: "d4f6caa7-a6cd-4d42-98fd-febb12b034eb"). InnerVolumeSpecName "kube-api-access-fdbph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:10:57.873241 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:57.873214 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdbph\" (UniqueName: \"kubernetes.io/projected/d4f6caa7-a6cd-4d42-98fd-febb12b034eb-kube-api-access-fdbph\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:10:57.873241 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:57.873238 4562 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d4f6caa7-a6cd-4d42-98fd-febb12b034eb-tls-cert\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:10:57.873432 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:57.873250 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z978p\" (UniqueName: \"kubernetes.io/projected/cb46c6ce-84c4-43df-ba03-53a7374e30eb-kube-api-access-z978p\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:10:58.597451 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:10:58.597407 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:10:58.680239 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:58.680202 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-p6vws" Oct 30 09:10:58.680417 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:58.680202 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-pkb2p" Oct 30 09:10:58.767968 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:58.767937 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-pkb2p"] Oct 30 09:10:58.771069 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:58.771045 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-pkb2p"] Oct 30 09:10:58.829018 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:58.828990 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-p6vws"] Oct 30 09:10:58.831251 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:58.831231 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-p6vws"] Oct 30 09:10:59.872310 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:59.872276 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb46c6ce-84c4-43df-ba03-53a7374e30eb" path="/var/lib/kubelet/pods/cb46c6ce-84c4-43df-ba03-53a7374e30eb/volumes" Oct 30 09:10:59.872692 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:10:59.872545 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4f6caa7-a6cd-4d42-98fd-febb12b034eb" path="/var/lib/kubelet/pods/d4f6caa7-a6cd-4d42-98fd-febb12b034eb/volumes" Oct 30 09:11:06.629985 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:06.629951 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-868774c7cf-kjms7"] Oct 30 09:11:06.630353 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:06.630201 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-868774c7cf-kjms7" podUID="b03fcec5-026e-4c18-ae6b-6ffc896a0759" containerName="maas-api" containerID="cri-o://aa3ae72247d0391fdfa0cf6d93a8af09f0bea225996c876131c6e9e155cd00ae" gracePeriod=30 Oct 30 09:11:06.862179 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:06.862155 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-868774c7cf-kjms7" Oct 30 09:11:06.946290 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:06.946266 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b03fcec5-026e-4c18-ae6b-6ffc896a0759-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b03fcec5-026e-4c18-ae6b-6ffc896a0759-openshift-service-ca\") pod \"b03fcec5-026e-4c18-ae6b-6ffc896a0759\" (UID: \"b03fcec5-026e-4c18-ae6b-6ffc896a0759\") " Oct 30 09:11:06.946471 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:06.946319 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b03fcec5-026e-4c18-ae6b-6ffc896a0759-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b03fcec5-026e-4c18-ae6b-6ffc896a0759-metrics-certs\") pod \"b03fcec5-026e-4c18-ae6b-6ffc896a0759\" (UID: \"b03fcec5-026e-4c18-ae6b-6ffc896a0759\") " Oct 30 09:11:06.946471 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:06.946398 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b03fcec5-026e-4c18-ae6b-6ffc896a0759-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b03fcec5-026e-4c18-ae6b-6ffc896a0759-maas-api-tls\") pod \"b03fcec5-026e-4c18-ae6b-6ffc896a0759\" (UID: \"b03fcec5-026e-4c18-ae6b-6ffc896a0759\") " Oct 30 09:11:06.946590 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:06.946484 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b03fcec5-026e-4c18-ae6b-6ffc896a0759-kube-api-access-5bcsv\" (UniqueName: \"kubernetes.io/projected/b03fcec5-026e-4c18-ae6b-6ffc896a0759-kube-api-access-5bcsv\") pod \"b03fcec5-026e-4c18-ae6b-6ffc896a0759\" (UID: \"b03fcec5-026e-4c18-ae6b-6ffc896a0759\") " Oct 30 09:11:06.946832 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:06.946704 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b03fcec5-026e-4c18-ae6b-6ffc896a0759-openshift-service-ca" pod "b03fcec5-026e-4c18-ae6b-6ffc896a0759" (UID: "b03fcec5-026e-4c18-ae6b-6ffc896a0759"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:11:06.946832 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:06.946792 4562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b03fcec5-026e-4c18-ae6b-6ffc896a0759-openshift-service-ca\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:11:06.948717 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:06.948689 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b03fcec5-026e-4c18-ae6b-6ffc896a0759-kube-api-access-5bcsv" pod "b03fcec5-026e-4c18-ae6b-6ffc896a0759" (UID: "b03fcec5-026e-4c18-ae6b-6ffc896a0759"). InnerVolumeSpecName "kube-api-access-5bcsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:11:06.948800 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:06.948745 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b03fcec5-026e-4c18-ae6b-6ffc896a0759-maas-api-tls" pod "b03fcec5-026e-4c18-ae6b-6ffc896a0759" (UID: "b03fcec5-026e-4c18-ae6b-6ffc896a0759"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:11:06.948800 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:06.948771 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b03fcec5-026e-4c18-ae6b-6ffc896a0759-metrics-certs" pod "b03fcec5-026e-4c18-ae6b-6ffc896a0759" (UID: "b03fcec5-026e-4c18-ae6b-6ffc896a0759"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:11:07.047737 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:07.047699 4562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b03fcec5-026e-4c18-ae6b-6ffc896a0759-maas-api-tls\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:11:07.047737 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:07.047732 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bcsv\" (UniqueName: \"kubernetes.io/projected/b03fcec5-026e-4c18-ae6b-6ffc896a0759-kube-api-access-5bcsv\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:11:07.047737 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:07.047742 4562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b03fcec5-026e-4c18-ae6b-6ffc896a0759-metrics-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:11:07.721494 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:07.721457 4562 generic.go:356] "Generic (PLEG): container finished" podID="b03fcec5-026e-4c18-ae6b-6ffc896a0759" containerID="aa3ae72247d0391fdfa0cf6d93a8af09f0bea225996c876131c6e9e155cd00ae" exitCode=0 Oct 30 09:11:07.721986 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:07.721531 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-868774c7cf-kjms7" Oct 30 09:11:07.721986 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:07.721539 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-868774c7cf-kjms7" event={"ID":"b03fcec5-026e-4c18-ae6b-6ffc896a0759","Type":"ContainerDied","Data":"aa3ae72247d0391fdfa0cf6d93a8af09f0bea225996c876131c6e9e155cd00ae"} Oct 30 09:11:07.721986 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:07.721614 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-868774c7cf-kjms7" event={"ID":"b03fcec5-026e-4c18-ae6b-6ffc896a0759","Type":"ContainerDied","Data":"20999a0a9fd4b3e61d3b3911303fd3e2e11472e47797173791ba503f0550ca81"} Oct 30 09:11:07.721986 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:07.721637 4562 scope.go:122] "RemoveContainer" containerID="aa3ae72247d0391fdfa0cf6d93a8af09f0bea225996c876131c6e9e155cd00ae" Oct 30 09:11:07.730492 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:07.730474 4562 scope.go:122] "RemoveContainer" containerID="aa3ae72247d0391fdfa0cf6d93a8af09f0bea225996c876131c6e9e155cd00ae" Oct 30 09:11:07.730762 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:11:07.730742 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa3ae72247d0391fdfa0cf6d93a8af09f0bea225996c876131c6e9e155cd00ae\": container with ID starting with aa3ae72247d0391fdfa0cf6d93a8af09f0bea225996c876131c6e9e155cd00ae not found: ID does not exist" containerID="aa3ae72247d0391fdfa0cf6d93a8af09f0bea225996c876131c6e9e155cd00ae" Oct 30 09:11:07.730817 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:07.730769 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa3ae72247d0391fdfa0cf6d93a8af09f0bea225996c876131c6e9e155cd00ae"} err="failed to get container status \"aa3ae72247d0391fdfa0cf6d93a8af09f0bea225996c876131c6e9e155cd00ae\": rpc error: code = NotFound desc = could not find container \"aa3ae72247d0391fdfa0cf6d93a8af09f0bea225996c876131c6e9e155cd00ae\": container with ID starting with aa3ae72247d0391fdfa0cf6d93a8af09f0bea225996c876131c6e9e155cd00ae not found: ID does not exist" Oct 30 09:11:07.773758 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:07.773733 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-868774c7cf-kjms7"] Oct 30 09:11:07.786830 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:07.786804 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-868774c7cf-kjms7"] Oct 30 09:11:07.872331 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:07.872297 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b03fcec5-026e-4c18-ae6b-6ffc896a0759" path="/var/lib/kubelet/pods/b03fcec5-026e-4c18-ae6b-6ffc896a0759/volumes" Oct 30 09:11:28.594139 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:11:28.594106 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:11:33.685797 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:33.685761 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-88c6v"] Oct 30 09:11:33.686183 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:33.686063 4562 cpu_manager.go:414] "containerMap: removing container" podUID="b03fcec5-026e-4c18-ae6b-6ffc896a0759" containerName="maas-api" Oct 30 09:11:33.686183 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:33.686074 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b03fcec5-026e-4c18-ae6b-6ffc896a0759" containerName="maas-api" Oct 30 09:11:33.686183 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:33.686145 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="b03fcec5-026e-4c18-ae6b-6ffc896a0759" containerName="maas-api" Oct 30 09:11:33.689059 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:33.689043 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-88c6v" Oct 30 09:11:33.705578 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:33.705551 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-88c6v"] Oct 30 09:11:33.717463 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:33.717437 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 30 09:11:33.717614 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:33.717501 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 30 09:11:33.717614 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:33.717570 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-267dd\"" Oct 30 09:11:33.749194 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:33.749162 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb697\" (UniqueName: \"kubernetes.io/projected/7815ade6-dd03-4f8e-b349-42788cd3a3c6-kube-api-access-gb697\") pod \"ai-gateway-controller-7d85675f98-88c6v\" (UID: \"7815ade6-dd03-4f8e-b349-42788cd3a3c6\") " pod="opendatahub/ai-gateway-controller-7d85675f98-88c6v" Oct 30 09:11:33.749359 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:33.749203 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7815ade6-dd03-4f8e-b349-42788cd3a3c6-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-88c6v\" (UID: \"7815ade6-dd03-4f8e-b349-42788cd3a3c6\") " pod="opendatahub/ai-gateway-controller-7d85675f98-88c6v" Oct 30 09:11:33.850304 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:33.850273 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb697\" (UniqueName: \"kubernetes.io/projected/7815ade6-dd03-4f8e-b349-42788cd3a3c6-kube-api-access-gb697\") pod \"ai-gateway-controller-7d85675f98-88c6v\" (UID: \"7815ade6-dd03-4f8e-b349-42788cd3a3c6\") " pod="opendatahub/ai-gateway-controller-7d85675f98-88c6v" Oct 30 09:11:33.850304 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:33.850308 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7815ade6-dd03-4f8e-b349-42788cd3a3c6-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-88c6v\" (UID: \"7815ade6-dd03-4f8e-b349-42788cd3a3c6\") " pod="opendatahub/ai-gateway-controller-7d85675f98-88c6v" Oct 30 09:11:33.852786 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:33.852762 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7815ade6-dd03-4f8e-b349-42788cd3a3c6-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-88c6v\" (UID: \"7815ade6-dd03-4f8e-b349-42788cd3a3c6\") " pod="opendatahub/ai-gateway-controller-7d85675f98-88c6v" Oct 30 09:11:33.871073 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:33.871042 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb697\" (UniqueName: \"kubernetes.io/projected/7815ade6-dd03-4f8e-b349-42788cd3a3c6-kube-api-access-gb697\") pod \"ai-gateway-controller-7d85675f98-88c6v\" (UID: \"7815ade6-dd03-4f8e-b349-42788cd3a3c6\") " pod="opendatahub/ai-gateway-controller-7d85675f98-88c6v" Oct 30 09:11:34.001611 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:34.001576 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-88c6v" Oct 30 09:11:34.288938 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:34.288902 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-88c6v"] Oct 30 09:11:34.289895 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:11:34.289866 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7815ade6_dd03_4f8e_b349_42788cd3a3c6.slice/crio-7d64f72878fa259cbc1634fe97cfe0eba106ce5a4cb70cba305542efdd68e104 WatchSource:0}: Error finding container 7d64f72878fa259cbc1634fe97cfe0eba106ce5a4cb70cba305542efdd68e104: Status 404 returned error can't find the container with id 7d64f72878fa259cbc1634fe97cfe0eba106ce5a4cb70cba305542efdd68e104 Oct 30 09:11:34.825075 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:34.825038 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-88c6v" event={"ID":"7815ade6-dd03-4f8e-b349-42788cd3a3c6","Type":"ContainerStarted","Data":"7d64f72878fa259cbc1634fe97cfe0eba106ce5a4cb70cba305542efdd68e104"} Oct 30 09:11:36.834597 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:36.834555 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-88c6v" event={"ID":"7815ade6-dd03-4f8e-b349-42788cd3a3c6","Type":"ContainerStarted","Data":"0b57e8debcd43ae3a0fb3dc69d0eda7f288a91368e76a379fc1be718bd77c4b0"} Oct 30 09:11:36.835089 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:36.834688 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d85675f98-88c6v" Oct 30 09:11:36.874321 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:36.874276 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d85675f98-88c6v" podStartSLOduration=2.322253597 podStartE2EDuration="3.874264831s" totalImagesPullingTime="1.552011234s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:11:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:11:36.873527012 +0000 UTC m=+1057.135088002" watchObservedRunningTime="2026-10-30 09:11:36.874264831 +0000 UTC m=+1057.135825820" Oct 30 09:11:38.173586 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:38.173554 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-85rhg"] Oct 30 09:11:38.176941 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:38.176921 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-85rhg" Oct 30 09:11:38.182371 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:38.182347 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f5e6dbb-9b81-4734-ba2a-5c6a677e49e8-plugins-config-volume\") pod \"payload-processing-5f78687d5b-85rhg\" (UID: \"2f5e6dbb-9b81-4734-ba2a-5c6a677e49e8\") " pod="openshift-ingress/payload-processing-5f78687d5b-85rhg" Oct 30 09:11:38.182477 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:38.182412 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmxfs\" (UniqueName: \"kubernetes.io/projected/2f5e6dbb-9b81-4734-ba2a-5c6a677e49e8-kube-api-access-mmxfs\") pod \"payload-processing-5f78687d5b-85rhg\" (UID: \"2f5e6dbb-9b81-4734-ba2a-5c6a677e49e8\") " pod="openshift-ingress/payload-processing-5f78687d5b-85rhg" Oct 30 09:11:38.194253 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:38.194229 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-85rhg"] Oct 30 09:11:38.205950 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:38.205926 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 30 09:11:38.206042 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:38.205961 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-gzbjz\"" Oct 30 09:11:38.233644 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:38.233617 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-l6sj2"] Oct 30 09:11:38.236944 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:38.236925 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-l6sj2" Oct 30 09:11:38.252244 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:38.252220 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-l6sj2"] Oct 30 09:11:38.283262 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:38.283234 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqgml\" (UniqueName: \"kubernetes.io/projected/076fc963-efa1-46c0-ae7f-3773c788ed00-kube-api-access-sqgml\") pod \"payload-pre-processing-7b98bb678c-l6sj2\" (UID: \"076fc963-efa1-46c0-ae7f-3773c788ed00\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-l6sj2" Oct 30 09:11:38.283429 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:38.283284 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/076fc963-efa1-46c0-ae7f-3773c788ed00-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-l6sj2\" (UID: \"076fc963-efa1-46c0-ae7f-3773c788ed00\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-l6sj2" Oct 30 09:11:38.283429 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:38.283394 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f5e6dbb-9b81-4734-ba2a-5c6a677e49e8-plugins-config-volume\") pod \"payload-processing-5f78687d5b-85rhg\" (UID: \"2f5e6dbb-9b81-4734-ba2a-5c6a677e49e8\") " pod="openshift-ingress/payload-processing-5f78687d5b-85rhg" Oct 30 09:11:38.283429 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:38.283425 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmxfs\" (UniqueName: \"kubernetes.io/projected/2f5e6dbb-9b81-4734-ba2a-5c6a677e49e8-kube-api-access-mmxfs\") pod \"payload-processing-5f78687d5b-85rhg\" (UID: \"2f5e6dbb-9b81-4734-ba2a-5c6a677e49e8\") " pod="openshift-ingress/payload-processing-5f78687d5b-85rhg" Oct 30 09:11:38.284072 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:38.284054 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f5e6dbb-9b81-4734-ba2a-5c6a677e49e8-plugins-config-volume\") pod \"payload-processing-5f78687d5b-85rhg\" (UID: \"2f5e6dbb-9b81-4734-ba2a-5c6a677e49e8\") " pod="openshift-ingress/payload-processing-5f78687d5b-85rhg" Oct 30 09:11:38.303685 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:38.303606 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmxfs\" (UniqueName: \"kubernetes.io/projected/2f5e6dbb-9b81-4734-ba2a-5c6a677e49e8-kube-api-access-mmxfs\") pod \"payload-processing-5f78687d5b-85rhg\" (UID: \"2f5e6dbb-9b81-4734-ba2a-5c6a677e49e8\") " pod="openshift-ingress/payload-processing-5f78687d5b-85rhg" Oct 30 09:11:38.384673 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:38.384638 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqgml\" (UniqueName: \"kubernetes.io/projected/076fc963-efa1-46c0-ae7f-3773c788ed00-kube-api-access-sqgml\") pod \"payload-pre-processing-7b98bb678c-l6sj2\" (UID: \"076fc963-efa1-46c0-ae7f-3773c788ed00\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-l6sj2" Oct 30 09:11:38.384876 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:38.384711 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/076fc963-efa1-46c0-ae7f-3773c788ed00-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-l6sj2\" (UID: \"076fc963-efa1-46c0-ae7f-3773c788ed00\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-l6sj2" Oct 30 09:11:38.385498 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:38.385474 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/076fc963-efa1-46c0-ae7f-3773c788ed00-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-l6sj2\" (UID: \"076fc963-efa1-46c0-ae7f-3773c788ed00\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-l6sj2" Oct 30 09:11:38.404179 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:38.404146 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqgml\" (UniqueName: \"kubernetes.io/projected/076fc963-efa1-46c0-ae7f-3773c788ed00-kube-api-access-sqgml\") pod \"payload-pre-processing-7b98bb678c-l6sj2\" (UID: \"076fc963-efa1-46c0-ae7f-3773c788ed00\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-l6sj2" Oct 30 09:11:38.489465 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:38.489425 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-85rhg" Oct 30 09:11:38.549849 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:38.549718 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-l6sj2" Oct 30 09:11:38.778325 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:38.778297 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-85rhg"] Oct 30 09:11:38.778562 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:11:38.778535 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f5e6dbb_9b81_4734_ba2a_5c6a677e49e8.slice/crio-2d25c5f7584e37839037753af829d0d839616db5f30c49ecf6380f3c6b3474d3 WatchSource:0}: Error finding container 2d25c5f7584e37839037753af829d0d839616db5f30c49ecf6380f3c6b3474d3: Status 404 returned error can't find the container with id 2d25c5f7584e37839037753af829d0d839616db5f30c49ecf6380f3c6b3474d3 Oct 30 09:11:38.842423 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:11:38.842391 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod076fc963_efa1_46c0_ae7f_3773c788ed00.slice/crio-2d49792157b0db93ee3f01eeb620e8b1f855f666a6c247e67dbf1986b5b32487 WatchSource:0}: Error finding container 2d49792157b0db93ee3f01eeb620e8b1f855f666a6c247e67dbf1986b5b32487: Status 404 returned error can't find the container with id 2d49792157b0db93ee3f01eeb620e8b1f855f666a6c247e67dbf1986b5b32487 Oct 30 09:11:38.842573 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:38.842555 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-l6sj2"] Oct 30 09:11:38.845041 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:38.844964 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-85rhg" event={"ID":"2f5e6dbb-9b81-4734-ba2a-5c6a677e49e8","Type":"ContainerStarted","Data":"2d25c5f7584e37839037753af829d0d839616db5f30c49ecf6380f3c6b3474d3"} Oct 30 09:11:39.851945 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:39.851907 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-l6sj2" event={"ID":"076fc963-efa1-46c0-ae7f-3773c788ed00","Type":"ContainerStarted","Data":"2d49792157b0db93ee3f01eeb620e8b1f855f666a6c247e67dbf1986b5b32487"} Oct 30 09:11:41.862776 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:41.862729 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-85rhg" event={"ID":"2f5e6dbb-9b81-4734-ba2a-5c6a677e49e8","Type":"ContainerStarted","Data":"4d1fc5cb4056f5ba2774e3182b18d5f28d5abc81c6e816d9e92afb510798b0b5"} Oct 30 09:11:41.863198 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:41.862839 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5f78687d5b-85rhg" Oct 30 09:11:41.864435 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:41.864404 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-l6sj2" event={"ID":"076fc963-efa1-46c0-ae7f-3773c788ed00","Type":"ContainerStarted","Data":"f6580976fb900a87308ddd33c613b4ce14be6e43392ed0f49a2a4e9157a9fba1"} Oct 30 09:11:41.864583 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:41.864464 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-l6sj2" Oct 30 09:11:41.902008 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:41.901960 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5f78687d5b-85rhg" podStartSLOduration=1.6238785569999998 podStartE2EDuration="3.901950633s" totalImagesPullingTime="2.278072076s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:11:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:11:41.900604576 +0000 UTC m=+1062.162165576" watchObservedRunningTime="2026-10-30 09:11:41.901950633 +0000 UTC m=+1062.163511612" Oct 30 09:11:41.940709 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:41.940662 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-l6sj2" podStartSLOduration=1.6940192710000002 podStartE2EDuration="3.94065096s" totalImagesPullingTime="2.246631689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:11:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:11:41.939820498 +0000 UTC m=+1062.201381486" watchObservedRunningTime="2026-10-30 09:11:41.94065096 +0000 UTC m=+1062.202211968" Oct 30 09:11:47.841004 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:47.840972 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d85675f98-88c6v" Oct 30 09:11:48.127353 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:48.127267 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5bbbc667d4-sdbtx"] Oct 30 09:11:48.130889 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:48.130867 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5bbbc667d4-sdbtx" Oct 30 09:11:48.147883 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:48.147862 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5bbbc667d4-sdbtx"] Oct 30 09:11:48.166024 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:48.165998 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd98de9c-4d7f-4f22-b042-2867781cf88c-metrics-certs\") pod \"ai-gateway-controller-5bbbc667d4-sdbtx\" (UID: \"fd98de9c-4d7f-4f22-b042-2867781cf88c\") " pod="opendatahub/ai-gateway-controller-5bbbc667d4-sdbtx" Oct 30 09:11:48.166117 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:48.166030 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d66df\" (UniqueName: \"kubernetes.io/projected/fd98de9c-4d7f-4f22-b042-2867781cf88c-kube-api-access-d66df\") pod \"ai-gateway-controller-5bbbc667d4-sdbtx\" (UID: \"fd98de9c-4d7f-4f22-b042-2867781cf88c\") " pod="opendatahub/ai-gateway-controller-5bbbc667d4-sdbtx" Oct 30 09:11:48.266862 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:48.266830 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd98de9c-4d7f-4f22-b042-2867781cf88c-metrics-certs\") pod \"ai-gateway-controller-5bbbc667d4-sdbtx\" (UID: \"fd98de9c-4d7f-4f22-b042-2867781cf88c\") " pod="opendatahub/ai-gateway-controller-5bbbc667d4-sdbtx" Oct 30 09:11:48.267024 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:48.266872 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d66df\" (UniqueName: \"kubernetes.io/projected/fd98de9c-4d7f-4f22-b042-2867781cf88c-kube-api-access-d66df\") pod \"ai-gateway-controller-5bbbc667d4-sdbtx\" (UID: \"fd98de9c-4d7f-4f22-b042-2867781cf88c\") " pod="opendatahub/ai-gateway-controller-5bbbc667d4-sdbtx" Oct 30 09:11:48.269307 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:48.269288 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd98de9c-4d7f-4f22-b042-2867781cf88c-metrics-certs\") pod \"ai-gateway-controller-5bbbc667d4-sdbtx\" (UID: \"fd98de9c-4d7f-4f22-b042-2867781cf88c\") " pod="opendatahub/ai-gateway-controller-5bbbc667d4-sdbtx" Oct 30 09:11:48.286615 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:48.286589 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d66df\" (UniqueName: \"kubernetes.io/projected/fd98de9c-4d7f-4f22-b042-2867781cf88c-kube-api-access-d66df\") pod \"ai-gateway-controller-5bbbc667d4-sdbtx\" (UID: \"fd98de9c-4d7f-4f22-b042-2867781cf88c\") " pod="opendatahub/ai-gateway-controller-5bbbc667d4-sdbtx" Oct 30 09:11:48.442809 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:48.442777 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5bbbc667d4-sdbtx" Oct 30 09:11:48.733027 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:11:48.732982 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd98de9c_4d7f_4f22_b042_2867781cf88c.slice/crio-b0cebb70e96699de8ceff7eeb37ae53f865a88e71a9aba0bf458bb2edc8cd053 WatchSource:0}: Error finding container b0cebb70e96699de8ceff7eeb37ae53f865a88e71a9aba0bf458bb2edc8cd053: Status 404 returned error can't find the container with id b0cebb70e96699de8ceff7eeb37ae53f865a88e71a9aba0bf458bb2edc8cd053 Oct 30 09:11:48.733569 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:48.733542 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5bbbc667d4-sdbtx"] Oct 30 09:11:48.894620 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:48.894585 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5bbbc667d4-sdbtx" event={"ID":"fd98de9c-4d7f-4f22-b042-2867781cf88c","Type":"ContainerStarted","Data":"b0cebb70e96699de8ceff7eeb37ae53f865a88e71a9aba0bf458bb2edc8cd053"} Oct 30 09:11:49.899766 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:49.899729 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5bbbc667d4-sdbtx" event={"ID":"fd98de9c-4d7f-4f22-b042-2867781cf88c","Type":"ContainerStarted","Data":"6065aec7432c1aee9edacef2118b8de89061763ef1c2266ed2d098df0fa01695"} Oct 30 09:11:49.900114 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:49.899790 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5bbbc667d4-sdbtx" Oct 30 09:11:52.872018 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:52.871985 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5f78687d5b-85rhg" Oct 30 09:11:52.872427 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:52.872058 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-l6sj2" Oct 30 09:11:52.913186 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:11:52.913139 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5bbbc667d4-sdbtx" podStartSLOduration=4.5857574020000005 podStartE2EDuration="4.913125587s" totalImagesPullingTime="327.368185ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:11:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:11:49.951259813 +0000 UTC m=+1070.212820813" watchObservedRunningTime="2026-10-30 09:11:52.913125587 +0000 UTC m=+1073.174686624" Oct 30 09:11:58.594652 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:11:58.594613 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:12:00.908690 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:00.908657 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5bbbc667d4-sdbtx" Oct 30 09:12:00.973022 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:00.972988 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-88c6v"] Oct 30 09:12:00.973237 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:00.973209 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7d85675f98-88c6v" podUID="7815ade6-dd03-4f8e-b349-42788cd3a3c6" containerName="manager" containerID="cri-o://0b57e8debcd43ae3a0fb3dc69d0eda7f288a91368e76a379fc1be718bd77c4b0" gracePeriod=30 Oct 30 09:12:01.210673 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:01.210648 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-88c6v" Oct 30 09:12:01.275617 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:01.275589 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7815ade6-dd03-4f8e-b349-42788cd3a3c6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7815ade6-dd03-4f8e-b349-42788cd3a3c6-metrics-certs\") pod \"7815ade6-dd03-4f8e-b349-42788cd3a3c6\" (UID: \"7815ade6-dd03-4f8e-b349-42788cd3a3c6\") " Oct 30 09:12:01.275774 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:01.275717 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7815ade6-dd03-4f8e-b349-42788cd3a3c6-kube-api-access-gb697\" (UniqueName: \"kubernetes.io/projected/7815ade6-dd03-4f8e-b349-42788cd3a3c6-kube-api-access-gb697\") pod \"7815ade6-dd03-4f8e-b349-42788cd3a3c6\" (UID: \"7815ade6-dd03-4f8e-b349-42788cd3a3c6\") " Oct 30 09:12:01.277621 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:01.277591 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7815ade6-dd03-4f8e-b349-42788cd3a3c6-kube-api-access-gb697" pod "7815ade6-dd03-4f8e-b349-42788cd3a3c6" (UID: "7815ade6-dd03-4f8e-b349-42788cd3a3c6"). InnerVolumeSpecName "kube-api-access-gb697". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:12:01.277748 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:01.277600 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7815ade6-dd03-4f8e-b349-42788cd3a3c6-metrics-certs" pod "7815ade6-dd03-4f8e-b349-42788cd3a3c6" (UID: "7815ade6-dd03-4f8e-b349-42788cd3a3c6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:12:01.376428 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:01.376399 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb697\" (UniqueName: \"kubernetes.io/projected/7815ade6-dd03-4f8e-b349-42788cd3a3c6-kube-api-access-gb697\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:12:01.376428 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:01.376423 4562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7815ade6-dd03-4f8e-b349-42788cd3a3c6-metrics-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:12:01.948947 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:01.948912 4562 generic.go:356] "Generic (PLEG): container finished" podID="7815ade6-dd03-4f8e-b349-42788cd3a3c6" containerID="0b57e8debcd43ae3a0fb3dc69d0eda7f288a91368e76a379fc1be718bd77c4b0" exitCode=0 Oct 30 09:12:01.949328 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:01.948952 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-88c6v" Oct 30 09:12:01.949328 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:01.948973 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-88c6v" event={"ID":"7815ade6-dd03-4f8e-b349-42788cd3a3c6","Type":"ContainerDied","Data":"0b57e8debcd43ae3a0fb3dc69d0eda7f288a91368e76a379fc1be718bd77c4b0"} Oct 30 09:12:01.949328 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:01.949016 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-88c6v" event={"ID":"7815ade6-dd03-4f8e-b349-42788cd3a3c6","Type":"ContainerDied","Data":"7d64f72878fa259cbc1634fe97cfe0eba106ce5a4cb70cba305542efdd68e104"} Oct 30 09:12:01.949328 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:01.949037 4562 scope.go:122] "RemoveContainer" containerID="0b57e8debcd43ae3a0fb3dc69d0eda7f288a91368e76a379fc1be718bd77c4b0" Oct 30 09:12:01.956914 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:01.956890 4562 scope.go:122] "RemoveContainer" containerID="0b57e8debcd43ae3a0fb3dc69d0eda7f288a91368e76a379fc1be718bd77c4b0" Oct 30 09:12:01.957173 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:12:01.957156 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b57e8debcd43ae3a0fb3dc69d0eda7f288a91368e76a379fc1be718bd77c4b0\": container with ID starting with 0b57e8debcd43ae3a0fb3dc69d0eda7f288a91368e76a379fc1be718bd77c4b0 not found: ID does not exist" containerID="0b57e8debcd43ae3a0fb3dc69d0eda7f288a91368e76a379fc1be718bd77c4b0" Oct 30 09:12:01.957214 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:01.957183 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b57e8debcd43ae3a0fb3dc69d0eda7f288a91368e76a379fc1be718bd77c4b0"} err="failed to get container status \"0b57e8debcd43ae3a0fb3dc69d0eda7f288a91368e76a379fc1be718bd77c4b0\": rpc error: code = NotFound desc = could not find container \"0b57e8debcd43ae3a0fb3dc69d0eda7f288a91368e76a379fc1be718bd77c4b0\": container with ID starting with 0b57e8debcd43ae3a0fb3dc69d0eda7f288a91368e76a379fc1be718bd77c4b0 not found: ID does not exist" Oct 30 09:12:02.001064 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:02.001029 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-88c6v"] Oct 30 09:12:02.006942 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:02.006925 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-88c6v"] Oct 30 09:12:03.872375 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:03.872342 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7815ade6-dd03-4f8e-b349-42788cd3a3c6" path="/var/lib/kubelet/pods/7815ade6-dd03-4f8e-b349-42788cd3a3c6/volumes" Oct 30 09:12:08.894545 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:08.894514 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-g5jjc"] Oct 30 09:12:08.894928 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:08.894837 4562 cpu_manager.go:414] "containerMap: removing container" podUID="7815ade6-dd03-4f8e-b349-42788cd3a3c6" containerName="manager" Oct 30 09:12:08.894928 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:08.894848 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7815ade6-dd03-4f8e-b349-42788cd3a3c6" containerName="manager" Oct 30 09:12:08.894928 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:08.894902 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="7815ade6-dd03-4f8e-b349-42788cd3a3c6" containerName="manager" Oct 30 09:12:08.899267 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:08.899247 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-g5jjc" Oct 30 09:12:08.915396 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:08.915360 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-g5jjc"] Oct 30 09:12:08.926447 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:08.926418 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 09:12:08.926604 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:08.926421 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 09:12:08.926604 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:08.926476 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Oct 30 09:12:08.928134 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:08.928117 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-kglgv\"" Oct 30 09:12:09.037426 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:09.037399 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1449660d-c4f4-49da-b916-52dd67a92dcf-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-g5jjc\" (UID: \"1449660d-c4f4-49da-b916-52dd67a92dcf\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-g5jjc" Oct 30 09:12:09.037426 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:09.037429 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1449660d-c4f4-49da-b916-52dd67a92dcf-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-g5jjc\" (UID: \"1449660d-c4f4-49da-b916-52dd67a92dcf\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-g5jjc" Oct 30 09:12:09.037601 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:09.037455 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1449660d-c4f4-49da-b916-52dd67a92dcf-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-g5jjc\" (UID: \"1449660d-c4f4-49da-b916-52dd67a92dcf\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-g5jjc" Oct 30 09:12:09.037601 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:09.037537 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1449660d-c4f4-49da-b916-52dd67a92dcf-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-g5jjc\" (UID: \"1449660d-c4f4-49da-b916-52dd67a92dcf\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-g5jjc" Oct 30 09:12:09.037680 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:09.037665 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1449660d-c4f4-49da-b916-52dd67a92dcf-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-g5jjc\" (UID: \"1449660d-c4f4-49da-b916-52dd67a92dcf\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-g5jjc" Oct 30 09:12:09.037715 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:09.037693 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb9tj\" (UniqueName: \"kubernetes.io/projected/1449660d-c4f4-49da-b916-52dd67a92dcf-kube-api-access-kb9tj\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-g5jjc\" (UID: \"1449660d-c4f4-49da-b916-52dd67a92dcf\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-g5jjc" Oct 30 09:12:09.138397 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:09.138356 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1449660d-c4f4-49da-b916-52dd67a92dcf-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-g5jjc\" (UID: \"1449660d-c4f4-49da-b916-52dd67a92dcf\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-g5jjc" Oct 30 09:12:09.138549 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:09.138406 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb9tj\" (UniqueName: \"kubernetes.io/projected/1449660d-c4f4-49da-b916-52dd67a92dcf-kube-api-access-kb9tj\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-g5jjc\" (UID: \"1449660d-c4f4-49da-b916-52dd67a92dcf\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-g5jjc" Oct 30 09:12:09.138549 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:09.138432 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1449660d-c4f4-49da-b916-52dd67a92dcf-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-g5jjc\" (UID: \"1449660d-c4f4-49da-b916-52dd67a92dcf\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-g5jjc" Oct 30 09:12:09.138549 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:09.138455 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1449660d-c4f4-49da-b916-52dd67a92dcf-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-g5jjc\" (UID: \"1449660d-c4f4-49da-b916-52dd67a92dcf\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-g5jjc" Oct 30 09:12:09.138549 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:09.138494 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1449660d-c4f4-49da-b916-52dd67a92dcf-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-g5jjc\" (UID: \"1449660d-c4f4-49da-b916-52dd67a92dcf\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-g5jjc" Oct 30 09:12:09.138549 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:09.138518 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1449660d-c4f4-49da-b916-52dd67a92dcf-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-g5jjc\" (UID: \"1449660d-c4f4-49da-b916-52dd67a92dcf\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-g5jjc" Oct 30 09:12:09.138880 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:09.138856 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1449660d-c4f4-49da-b916-52dd67a92dcf-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-g5jjc\" (UID: \"1449660d-c4f4-49da-b916-52dd67a92dcf\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-g5jjc" Oct 30 09:12:09.138923 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:09.138877 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1449660d-c4f4-49da-b916-52dd67a92dcf-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-g5jjc\" (UID: \"1449660d-c4f4-49da-b916-52dd67a92dcf\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-g5jjc" Oct 30 09:12:09.138923 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:09.138907 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1449660d-c4f4-49da-b916-52dd67a92dcf-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-g5jjc\" (UID: \"1449660d-c4f4-49da-b916-52dd67a92dcf\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-g5jjc" Oct 30 09:12:09.140703 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:09.140677 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1449660d-c4f4-49da-b916-52dd67a92dcf-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-g5jjc\" (UID: \"1449660d-c4f4-49da-b916-52dd67a92dcf\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-g5jjc" Oct 30 09:12:09.140953 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:09.140936 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1449660d-c4f4-49da-b916-52dd67a92dcf-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-g5jjc\" (UID: \"1449660d-c4f4-49da-b916-52dd67a92dcf\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-g5jjc" Oct 30 09:12:09.158005 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:09.157956 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb9tj\" (UniqueName: \"kubernetes.io/projected/1449660d-c4f4-49da-b916-52dd67a92dcf-kube-api-access-kb9tj\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-g5jjc\" (UID: \"1449660d-c4f4-49da-b916-52dd67a92dcf\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-g5jjc" Oct 30 09:12:09.217646 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:09.217609 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-g5jjc" Oct 30 09:12:09.507856 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:09.507827 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-g5jjc"] Oct 30 09:12:09.508132 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:12:09.508104 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1449660d_c4f4_49da_b916_52dd67a92dcf.slice/crio-c45f618f2d6ca55861590908534729afa959474b36c0d5a78baff551c9e4928d WatchSource:0}: Error finding container c45f618f2d6ca55861590908534729afa959474b36c0d5a78baff551c9e4928d: Status 404 returned error can't find the container with id c45f618f2d6ca55861590908534729afa959474b36c0d5a78baff551c9e4928d Oct 30 09:12:09.984097 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:09.984058 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-g5jjc" event={"ID":"1449660d-c4f4-49da-b916-52dd67a92dcf","Type":"ContainerStarted","Data":"c45f618f2d6ca55861590908534729afa959474b36c0d5a78baff551c9e4928d"} Oct 30 09:12:10.989663 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:10.989627 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-g5jjc" event={"ID":"1449660d-c4f4-49da-b916-52dd67a92dcf","Type":"ContainerStarted","Data":"f8e9e515332f20330a6f7fbeb4f0fa536c69ff58b47b58e9b0c621348e28f652"} Oct 30 09:12:11.029852 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:11.029807 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-g5jjc" podStartSLOduration=1.863022388 podStartE2EDuration="3.029793514s" totalImagesPullingTime="1.166771126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:12:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:12:11.029306275 +0000 UTC m=+1091.290867276" watchObservedRunningTime="2026-10-30 09:12:11.029793514 +0000 UTC m=+1091.291354501" Oct 30 09:12:11.218208 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:11.218174 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-g5jjc" Oct 30 09:12:12.403034 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:12.403003 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-zt6jf"] Oct 30 09:12:12.409055 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:12.409035 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zt6jf" Oct 30 09:12:12.426833 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:12.426801 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-zt6jf"] Oct 30 09:12:12.437889 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:12.437863 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Oct 30 09:12:12.576248 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:12.576211 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8h5b\" (UniqueName: \"kubernetes.io/projected/697547af-60fc-4555-ac54-37c087f649e8-kube-api-access-s8h5b\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zt6jf\" (UID: \"697547af-60fc-4555-ac54-37c087f649e8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zt6jf" Oct 30 09:12:12.576448 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:12.576432 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/697547af-60fc-4555-ac54-37c087f649e8-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zt6jf\" (UID: \"697547af-60fc-4555-ac54-37c087f649e8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zt6jf" Oct 30 09:12:12.576664 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:12.576643 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/697547af-60fc-4555-ac54-37c087f649e8-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zt6jf\" (UID: \"697547af-60fc-4555-ac54-37c087f649e8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zt6jf" Oct 30 09:12:12.576706 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:12.576674 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/697547af-60fc-4555-ac54-37c087f649e8-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zt6jf\" (UID: \"697547af-60fc-4555-ac54-37c087f649e8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zt6jf" Oct 30 09:12:12.576825 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:12.576807 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/697547af-60fc-4555-ac54-37c087f649e8-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zt6jf\" (UID: \"697547af-60fc-4555-ac54-37c087f649e8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zt6jf" Oct 30 09:12:12.576862 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:12.576832 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/697547af-60fc-4555-ac54-37c087f649e8-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zt6jf\" (UID: \"697547af-60fc-4555-ac54-37c087f649e8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zt6jf" Oct 30 09:12:12.677436 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:12.677324 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/697547af-60fc-4555-ac54-37c087f649e8-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zt6jf\" (UID: \"697547af-60fc-4555-ac54-37c087f649e8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zt6jf" Oct 30 09:12:12.677436 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:12.677366 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/697547af-60fc-4555-ac54-37c087f649e8-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zt6jf\" (UID: \"697547af-60fc-4555-ac54-37c087f649e8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zt6jf" Oct 30 09:12:12.677436 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:12.677433 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/697547af-60fc-4555-ac54-37c087f649e8-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zt6jf\" (UID: \"697547af-60fc-4555-ac54-37c087f649e8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zt6jf" Oct 30 09:12:12.677723 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:12.677451 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/697547af-60fc-4555-ac54-37c087f649e8-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zt6jf\" (UID: \"697547af-60fc-4555-ac54-37c087f649e8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zt6jf" Oct 30 09:12:12.677723 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:12.677519 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8h5b\" (UniqueName: \"kubernetes.io/projected/697547af-60fc-4555-ac54-37c087f649e8-kube-api-access-s8h5b\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zt6jf\" (UID: \"697547af-60fc-4555-ac54-37c087f649e8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zt6jf" Oct 30 09:12:12.677723 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:12.677561 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/697547af-60fc-4555-ac54-37c087f649e8-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zt6jf\" (UID: \"697547af-60fc-4555-ac54-37c087f649e8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zt6jf" Oct 30 09:12:12.677879 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:12.677840 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/697547af-60fc-4555-ac54-37c087f649e8-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zt6jf\" (UID: \"697547af-60fc-4555-ac54-37c087f649e8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zt6jf" Oct 30 09:12:12.677937 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:12.677914 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/697547af-60fc-4555-ac54-37c087f649e8-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zt6jf\" (UID: \"697547af-60fc-4555-ac54-37c087f649e8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zt6jf" Oct 30 09:12:12.677973 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:12.677948 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/697547af-60fc-4555-ac54-37c087f649e8-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zt6jf\" (UID: \"697547af-60fc-4555-ac54-37c087f649e8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zt6jf" Oct 30 09:12:12.680201 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:12.680181 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/697547af-60fc-4555-ac54-37c087f649e8-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zt6jf\" (UID: \"697547af-60fc-4555-ac54-37c087f649e8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zt6jf" Oct 30 09:12:12.680348 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:12.680330 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/697547af-60fc-4555-ac54-37c087f649e8-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zt6jf\" (UID: \"697547af-60fc-4555-ac54-37c087f649e8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zt6jf" Oct 30 09:12:12.696963 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:12.696932 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8h5b\" (UniqueName: \"kubernetes.io/projected/697547af-60fc-4555-ac54-37c087f649e8-kube-api-access-s8h5b\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zt6jf\" (UID: \"697547af-60fc-4555-ac54-37c087f649e8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zt6jf" Oct 30 09:12:12.723961 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:12.723930 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zt6jf" Oct 30 09:12:13.015192 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:13.015165 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-zt6jf"] Oct 30 09:12:13.015820 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:12:13.015794 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod697547af_60fc_4555_ac54_37c087f649e8.slice/crio-c5e13947ef0146e2c681d677c19f7193abd54c673eaee469e457165d4612cfd7 WatchSource:0}: Error finding container c5e13947ef0146e2c681d677c19f7193abd54c673eaee469e457165d4612cfd7: Status 404 returned error can't find the container with id c5e13947ef0146e2c681d677c19f7193abd54c673eaee469e457165d4612cfd7 Oct 30 09:12:14.002188 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:14.002149 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zt6jf" event={"ID":"697547af-60fc-4555-ac54-37c087f649e8","Type":"ContainerStarted","Data":"fc9a3b93270bb21093aef2d8c5d593ea0fcb5ea79ba6e3acf6afc36927ebfa1a"} Oct 30 09:12:14.002592 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:14.002212 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zt6jf" event={"ID":"697547af-60fc-4555-ac54-37c087f649e8","Type":"ContainerStarted","Data":"c5e13947ef0146e2c681d677c19f7193abd54c673eaee469e457165d4612cfd7"} Oct 30 09:12:14.043581 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:14.043515 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zt6jf" podStartSLOduration=1.826814658 podStartE2EDuration="2.043498135s" totalImagesPullingTime="216.683477ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:12:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:12:14.042179255 +0000 UTC m=+1094.303740242" watchObservedRunningTime="2026-10-30 09:12:14.043498135 +0000 UTC m=+1094.305059124" Oct 30 09:12:14.147444 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:14.147411 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-2h599"] Oct 30 09:12:14.150918 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:14.150897 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2h599" Oct 30 09:12:14.166466 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:14.166444 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Oct 30 09:12:14.166951 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:14.166918 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-2h599"] Oct 30 09:12:14.293327 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:14.293237 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bce5ba79-e37c-4ff8-9b95-b2d27bdd3a6d-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2h599\" (UID: \"bce5ba79-e37c-4ff8-9b95-b2d27bdd3a6d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2h599" Oct 30 09:12:14.293327 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:14.293284 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bce5ba79-e37c-4ff8-9b95-b2d27bdd3a6d-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2h599\" (UID: \"bce5ba79-e37c-4ff8-9b95-b2d27bdd3a6d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2h599" Oct 30 09:12:14.293327 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:14.293300 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2476p\" (UniqueName: \"kubernetes.io/projected/bce5ba79-e37c-4ff8-9b95-b2d27bdd3a6d-kube-api-access-2476p\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2h599\" (UID: \"bce5ba79-e37c-4ff8-9b95-b2d27bdd3a6d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2h599" Oct 30 09:12:14.293582 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:14.293437 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bce5ba79-e37c-4ff8-9b95-b2d27bdd3a6d-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2h599\" (UID: \"bce5ba79-e37c-4ff8-9b95-b2d27bdd3a6d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2h599" Oct 30 09:12:14.293582 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:14.293532 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bce5ba79-e37c-4ff8-9b95-b2d27bdd3a6d-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2h599\" (UID: \"bce5ba79-e37c-4ff8-9b95-b2d27bdd3a6d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2h599" Oct 30 09:12:14.293582 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:14.293572 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bce5ba79-e37c-4ff8-9b95-b2d27bdd3a6d-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2h599\" (UID: \"bce5ba79-e37c-4ff8-9b95-b2d27bdd3a6d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2h599" Oct 30 09:12:14.393988 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:14.393951 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bce5ba79-e37c-4ff8-9b95-b2d27bdd3a6d-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2h599\" (UID: \"bce5ba79-e37c-4ff8-9b95-b2d27bdd3a6d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2h599" Oct 30 09:12:14.394170 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:14.394022 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bce5ba79-e37c-4ff8-9b95-b2d27bdd3a6d-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2h599\" (UID: \"bce5ba79-e37c-4ff8-9b95-b2d27bdd3a6d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2h599" Oct 30 09:12:14.394170 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:14.394061 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bce5ba79-e37c-4ff8-9b95-b2d27bdd3a6d-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2h599\" (UID: \"bce5ba79-e37c-4ff8-9b95-b2d27bdd3a6d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2h599" Oct 30 09:12:14.394170 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:14.394098 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bce5ba79-e37c-4ff8-9b95-b2d27bdd3a6d-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2h599\" (UID: \"bce5ba79-e37c-4ff8-9b95-b2d27bdd3a6d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2h599" Oct 30 09:12:14.394170 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:14.394139 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bce5ba79-e37c-4ff8-9b95-b2d27bdd3a6d-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2h599\" (UID: \"bce5ba79-e37c-4ff8-9b95-b2d27bdd3a6d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2h599" Oct 30 09:12:14.394170 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:14.394160 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2476p\" (UniqueName: \"kubernetes.io/projected/bce5ba79-e37c-4ff8-9b95-b2d27bdd3a6d-kube-api-access-2476p\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2h599\" (UID: \"bce5ba79-e37c-4ff8-9b95-b2d27bdd3a6d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2h599" Oct 30 09:12:14.394448 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:14.394414 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bce5ba79-e37c-4ff8-9b95-b2d27bdd3a6d-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2h599\" (UID: \"bce5ba79-e37c-4ff8-9b95-b2d27bdd3a6d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2h599" Oct 30 09:12:14.394510 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:14.394464 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bce5ba79-e37c-4ff8-9b95-b2d27bdd3a6d-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2h599\" (UID: \"bce5ba79-e37c-4ff8-9b95-b2d27bdd3a6d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2h599" Oct 30 09:12:14.394551 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:14.394508 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bce5ba79-e37c-4ff8-9b95-b2d27bdd3a6d-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2h599\" (UID: \"bce5ba79-e37c-4ff8-9b95-b2d27bdd3a6d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2h599" Oct 30 09:12:14.396426 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:14.396406 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bce5ba79-e37c-4ff8-9b95-b2d27bdd3a6d-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2h599\" (UID: \"bce5ba79-e37c-4ff8-9b95-b2d27bdd3a6d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2h599" Oct 30 09:12:14.396625 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:14.396607 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bce5ba79-e37c-4ff8-9b95-b2d27bdd3a6d-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2h599\" (UID: \"bce5ba79-e37c-4ff8-9b95-b2d27bdd3a6d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2h599" Oct 30 09:12:14.413727 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:14.413699 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2476p\" (UniqueName: \"kubernetes.io/projected/bce5ba79-e37c-4ff8-9b95-b2d27bdd3a6d-kube-api-access-2476p\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2h599\" (UID: \"bce5ba79-e37c-4ff8-9b95-b2d27bdd3a6d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2h599" Oct 30 09:12:14.465768 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:14.465737 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2h599" Oct 30 09:12:14.724482 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:14.724451 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zt6jf" Oct 30 09:12:14.754370 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:12:14.754329 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbce5ba79_e37c_4ff8_9b95_b2d27bdd3a6d.slice/crio-9756b2ba085afaaab10b9c242dc8d5c3581ef4d8a12c3072a9b6c89313476122 WatchSource:0}: Error finding container 9756b2ba085afaaab10b9c242dc8d5c3581ef4d8a12c3072a9b6c89313476122: Status 404 returned error can't find the container with id 9756b2ba085afaaab10b9c242dc8d5c3581ef4d8a12c3072a9b6c89313476122 Oct 30 09:12:14.754655 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:14.754635 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-2h599"] Oct 30 09:12:15.007869 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:15.007839 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2h599" event={"ID":"bce5ba79-e37c-4ff8-9b95-b2d27bdd3a6d","Type":"ContainerStarted","Data":"9756b2ba085afaaab10b9c242dc8d5c3581ef4d8a12c3072a9b6c89313476122"} Oct 30 09:12:16.014038 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:16.014000 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2h599" event={"ID":"bce5ba79-e37c-4ff8-9b95-b2d27bdd3a6d","Type":"ContainerStarted","Data":"cd4c619df38883bbbd641e67f379b6a9d80c411e17b6384c4ccb1efd97895144"} Oct 30 09:12:16.054532 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:16.054482 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2h599" podStartSLOduration=1.83775882 podStartE2EDuration="2.054469218s" totalImagesPullingTime="216.710398ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:12:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:12:16.054024211 +0000 UTC m=+1096.315585199" watchObservedRunningTime="2026-10-30 09:12:16.054469218 +0000 UTC m=+1096.316030206" Oct 30 09:12:16.466619 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:16.466585 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2h599" Oct 30 09:12:16.949717 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:16.949685 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88xq7t"] Oct 30 09:12:16.953938 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:16.953916 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88xq7t" Oct 30 09:12:16.970316 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:16.970292 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Oct 30 09:12:16.984524 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:16.984495 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88xq7t"] Oct 30 09:12:17.118403 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:17.118351 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0893a22f-fb50-42c5-af68-af12521387f1-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88xq7t\" (UID: \"0893a22f-fb50-42c5-af68-af12521387f1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88xq7t" Oct 30 09:12:17.118872 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:17.118511 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5qnt\" (UniqueName: \"kubernetes.io/projected/0893a22f-fb50-42c5-af68-af12521387f1-kube-api-access-p5qnt\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88xq7t\" (UID: \"0893a22f-fb50-42c5-af68-af12521387f1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88xq7t" Oct 30 09:12:17.118872 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:17.118557 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0893a22f-fb50-42c5-af68-af12521387f1-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88xq7t\" (UID: \"0893a22f-fb50-42c5-af68-af12521387f1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88xq7t" Oct 30 09:12:17.118872 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:17.118603 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0893a22f-fb50-42c5-af68-af12521387f1-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88xq7t\" (UID: \"0893a22f-fb50-42c5-af68-af12521387f1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88xq7t" Oct 30 09:12:17.118872 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:17.118697 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0893a22f-fb50-42c5-af68-af12521387f1-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88xq7t\" (UID: \"0893a22f-fb50-42c5-af68-af12521387f1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88xq7t" Oct 30 09:12:17.118872 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:17.118777 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0893a22f-fb50-42c5-af68-af12521387f1-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88xq7t\" (UID: \"0893a22f-fb50-42c5-af68-af12521387f1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88xq7t" Oct 30 09:12:17.219788 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:17.219698 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0893a22f-fb50-42c5-af68-af12521387f1-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88xq7t\" (UID: \"0893a22f-fb50-42c5-af68-af12521387f1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88xq7t" Oct 30 09:12:17.219788 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:17.219741 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5qnt\" (UniqueName: \"kubernetes.io/projected/0893a22f-fb50-42c5-af68-af12521387f1-kube-api-access-p5qnt\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88xq7t\" (UID: \"0893a22f-fb50-42c5-af68-af12521387f1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88xq7t" Oct 30 09:12:17.219788 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:17.219760 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0893a22f-fb50-42c5-af68-af12521387f1-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88xq7t\" (UID: \"0893a22f-fb50-42c5-af68-af12521387f1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88xq7t" Oct 30 09:12:17.220071 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:17.219796 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0893a22f-fb50-42c5-af68-af12521387f1-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88xq7t\" (UID: \"0893a22f-fb50-42c5-af68-af12521387f1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88xq7t" Oct 30 09:12:17.220071 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:17.219844 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0893a22f-fb50-42c5-af68-af12521387f1-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88xq7t\" (UID: \"0893a22f-fb50-42c5-af68-af12521387f1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88xq7t" Oct 30 09:12:17.220071 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:17.219885 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0893a22f-fb50-42c5-af68-af12521387f1-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88xq7t\" (UID: \"0893a22f-fb50-42c5-af68-af12521387f1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88xq7t" Oct 30 09:12:17.220230 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:17.220192 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0893a22f-fb50-42c5-af68-af12521387f1-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88xq7t\" (UID: \"0893a22f-fb50-42c5-af68-af12521387f1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88xq7t" Oct 30 09:12:17.220276 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:17.220196 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0893a22f-fb50-42c5-af68-af12521387f1-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88xq7t\" (UID: \"0893a22f-fb50-42c5-af68-af12521387f1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88xq7t" Oct 30 09:12:17.220312 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:17.220280 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0893a22f-fb50-42c5-af68-af12521387f1-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88xq7t\" (UID: \"0893a22f-fb50-42c5-af68-af12521387f1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88xq7t" Oct 30 09:12:17.222069 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:17.222049 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0893a22f-fb50-42c5-af68-af12521387f1-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88xq7t\" (UID: \"0893a22f-fb50-42c5-af68-af12521387f1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88xq7t" Oct 30 09:12:17.222302 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:17.222285 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0893a22f-fb50-42c5-af68-af12521387f1-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88xq7t\" (UID: \"0893a22f-fb50-42c5-af68-af12521387f1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88xq7t" Oct 30 09:12:17.239671 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:17.239642 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5qnt\" (UniqueName: \"kubernetes.io/projected/0893a22f-fb50-42c5-af68-af12521387f1-kube-api-access-p5qnt\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88xq7t\" (UID: \"0893a22f-fb50-42c5-af68-af12521387f1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88xq7t" Oct 30 09:12:17.269315 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:17.269288 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88xq7t" Oct 30 09:12:17.561198 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:17.561146 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88xq7t"] Oct 30 09:12:17.561504 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:12:17.561462 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0893a22f_fb50_42c5_af68_af12521387f1.slice/crio-17bd40c2587942e6c63f4fc219e994a0b5b282b0ca674d78e238554cdcfc6b89 WatchSource:0}: Error finding container 17bd40c2587942e6c63f4fc219e994a0b5b282b0ca674d78e238554cdcfc6b89: Status 404 returned error can't find the container with id 17bd40c2587942e6c63f4fc219e994a0b5b282b0ca674d78e238554cdcfc6b89 Oct 30 09:12:18.027816 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:18.027776 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88xq7t" event={"ID":"0893a22f-fb50-42c5-af68-af12521387f1","Type":"ContainerStarted","Data":"46c761e2c4a52a46eec59065e3e7b6ec6775cbfc47ebfb237214d081b657b260"} Oct 30 09:12:18.028030 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:18.027843 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88xq7t" event={"ID":"0893a22f-fb50-42c5-af68-af12521387f1","Type":"ContainerStarted","Data":"17bd40c2587942e6c63f4fc219e994a0b5b282b0ca674d78e238554cdcfc6b89"} Oct 30 09:12:18.071402 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:18.069212 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88xq7t" podStartSLOduration=1.860894734 podStartE2EDuration="2.069194747s" totalImagesPullingTime="208.300013ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:12:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:12:18.06793907 +0000 UTC m=+1098.329500060" watchObservedRunningTime="2026-10-30 09:12:18.069194747 +0000 UTC m=+1098.330755736" Oct 30 09:12:18.269897 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:18.269860 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88xq7t" Oct 30 09:12:18.334322 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:18.334247 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-jbjpj"] Oct 30 09:12:18.338186 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:18.338153 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-jbjpj" Oct 30 09:12:18.354295 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:18.354261 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Oct 30 09:12:18.356125 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:18.356100 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-jbjpj"] Oct 30 09:12:18.429839 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:18.429802 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfc464d1-6f5c-4ab5-8b5a-41e439035213-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-jbjpj\" (UID: \"cfc464d1-6f5c-4ab5-8b5a-41e439035213\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jbjpj" Oct 30 09:12:18.430009 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:18.429847 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfc464d1-6f5c-4ab5-8b5a-41e439035213-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-jbjpj\" (UID: \"cfc464d1-6f5c-4ab5-8b5a-41e439035213\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jbjpj" Oct 30 09:12:18.430009 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:18.429961 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfc464d1-6f5c-4ab5-8b5a-41e439035213-home\") pod \"echo-simulator-kserve-64b6f8f96c-jbjpj\" (UID: \"cfc464d1-6f5c-4ab5-8b5a-41e439035213\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jbjpj" Oct 30 09:12:18.430009 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:18.429987 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfc464d1-6f5c-4ab5-8b5a-41e439035213-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-jbjpj\" (UID: \"cfc464d1-6f5c-4ab5-8b5a-41e439035213\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jbjpj" Oct 30 09:12:18.430009 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:18.430005 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h294c\" (UniqueName: \"kubernetes.io/projected/cfc464d1-6f5c-4ab5-8b5a-41e439035213-kube-api-access-h294c\") pod \"echo-simulator-kserve-64b6f8f96c-jbjpj\" (UID: \"cfc464d1-6f5c-4ab5-8b5a-41e439035213\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jbjpj" Oct 30 09:12:18.430142 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:18.430104 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfc464d1-6f5c-4ab5-8b5a-41e439035213-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-jbjpj\" (UID: \"cfc464d1-6f5c-4ab5-8b5a-41e439035213\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jbjpj" Oct 30 09:12:18.530696 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:18.530661 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfc464d1-6f5c-4ab5-8b5a-41e439035213-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-jbjpj\" (UID: \"cfc464d1-6f5c-4ab5-8b5a-41e439035213\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jbjpj" Oct 30 09:12:18.530696 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:18.530697 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfc464d1-6f5c-4ab5-8b5a-41e439035213-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-jbjpj\" (UID: \"cfc464d1-6f5c-4ab5-8b5a-41e439035213\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jbjpj" Oct 30 09:12:18.530938 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:18.530720 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfc464d1-6f5c-4ab5-8b5a-41e439035213-home\") pod \"echo-simulator-kserve-64b6f8f96c-jbjpj\" (UID: \"cfc464d1-6f5c-4ab5-8b5a-41e439035213\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jbjpj" Oct 30 09:12:18.530938 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:18.530747 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfc464d1-6f5c-4ab5-8b5a-41e439035213-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-jbjpj\" (UID: \"cfc464d1-6f5c-4ab5-8b5a-41e439035213\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jbjpj" Oct 30 09:12:18.530938 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:18.530765 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h294c\" (UniqueName: \"kubernetes.io/projected/cfc464d1-6f5c-4ab5-8b5a-41e439035213-kube-api-access-h294c\") pod \"echo-simulator-kserve-64b6f8f96c-jbjpj\" (UID: \"cfc464d1-6f5c-4ab5-8b5a-41e439035213\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jbjpj" Oct 30 09:12:18.530938 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:18.530807 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfc464d1-6f5c-4ab5-8b5a-41e439035213-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-jbjpj\" (UID: \"cfc464d1-6f5c-4ab5-8b5a-41e439035213\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jbjpj" Oct 30 09:12:18.531146 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:18.531120 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfc464d1-6f5c-4ab5-8b5a-41e439035213-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-jbjpj\" (UID: \"cfc464d1-6f5c-4ab5-8b5a-41e439035213\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jbjpj" Oct 30 09:12:18.531245 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:18.531225 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfc464d1-6f5c-4ab5-8b5a-41e439035213-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-jbjpj\" (UID: \"cfc464d1-6f5c-4ab5-8b5a-41e439035213\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jbjpj" Oct 30 09:12:18.531314 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:18.531289 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfc464d1-6f5c-4ab5-8b5a-41e439035213-home\") pod \"echo-simulator-kserve-64b6f8f96c-jbjpj\" (UID: \"cfc464d1-6f5c-4ab5-8b5a-41e439035213\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jbjpj" Oct 30 09:12:18.533103 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:18.533079 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfc464d1-6f5c-4ab5-8b5a-41e439035213-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-jbjpj\" (UID: \"cfc464d1-6f5c-4ab5-8b5a-41e439035213\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jbjpj" Oct 30 09:12:18.533207 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:18.533190 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfc464d1-6f5c-4ab5-8b5a-41e439035213-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-jbjpj\" (UID: \"cfc464d1-6f5c-4ab5-8b5a-41e439035213\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jbjpj" Oct 30 09:12:18.550374 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:18.550345 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h294c\" (UniqueName: \"kubernetes.io/projected/cfc464d1-6f5c-4ab5-8b5a-41e439035213-kube-api-access-h294c\") pod \"echo-simulator-kserve-64b6f8f96c-jbjpj\" (UID: \"cfc464d1-6f5c-4ab5-8b5a-41e439035213\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jbjpj" Oct 30 09:12:18.653857 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:18.653773 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-jbjpj" Oct 30 09:12:18.945200 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:18.945171 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-jbjpj"] Oct 30 09:12:18.945699 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:12:18.945675 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfc464d1_6f5c_4ab5_8b5a_41e439035213.slice/crio-37f10b8040aa177a3ef04d7ed588216a61a23412065cb98d0699d9774396ced2 WatchSource:0}: Error finding container 37f10b8040aa177a3ef04d7ed588216a61a23412065cb98d0699d9774396ced2: Status 404 returned error can't find the container with id 37f10b8040aa177a3ef04d7ed588216a61a23412065cb98d0699d9774396ced2 Oct 30 09:12:19.032855 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:19.032821 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-jbjpj" event={"ID":"cfc464d1-6f5c-4ab5-8b5a-41e439035213","Type":"ContainerStarted","Data":"37f10b8040aa177a3ef04d7ed588216a61a23412065cb98d0699d9774396ced2"} Oct 30 09:12:19.218129 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:19.218059 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-g5jjc" Oct 30 09:12:19.230893 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:19.230860 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-g5jjc" Oct 30 09:12:20.048516 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:20.048484 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-g5jjc" Oct 30 09:12:21.041952 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:21.041915 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-jbjpj" event={"ID":"cfc464d1-6f5c-4ab5-8b5a-41e439035213","Type":"ContainerStarted","Data":"a7395b3a35c11d7ca7eccdb4e4a4ea32908123104f1a41c0e2b10b9d7ffd77e3"} Oct 30 09:12:21.083421 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:21.083340 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-jbjpj" podStartSLOduration=1.594662874 podStartE2EDuration="3.083325769s" totalImagesPullingTime="1.488662895s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:12:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:12:21.082721612 +0000 UTC m=+1101.344282600" watchObservedRunningTime="2026-10-30 09:12:21.083325769 +0000 UTC m=+1101.344886756" Oct 30 09:12:21.654295 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:21.654257 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-jbjpj" Oct 30 09:12:22.724263 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:22.724231 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zt6jf" Oct 30 09:12:22.736672 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:22.736648 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zt6jf" Oct 30 09:12:23.062038 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:23.061956 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zt6jf" Oct 30 09:12:24.466195 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:24.466163 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2h599" Oct 30 09:12:24.478670 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:24.478642 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2h599" Oct 30 09:12:25.069527 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:25.069496 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2h599" Oct 30 09:12:27.270373 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:27.270344 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88xq7t" Oct 30 09:12:27.282907 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:27.282877 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88xq7t" Oct 30 09:12:28.082365 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:28.082339 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88xq7t" Oct 30 09:12:28.595138 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:12:28.595100 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:12:28.654574 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:28.654540 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-jbjpj" Oct 30 09:12:28.666815 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:28.666792 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-jbjpj" Oct 30 09:12:29.086246 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:29.086219 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-jbjpj" Oct 30 09:12:34.872919 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:34.872883 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vbklx"] Oct 30 09:12:34.876884 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:34.876863 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vbklx" Oct 30 09:12:34.908045 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:34.908014 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-j7g5p\"" Oct 30 09:12:34.909466 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:34.909441 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vbklx"] Oct 30 09:12:34.968750 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:34.968716 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/051c87da-acbb-4941-8fea-57107abb4720-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-vbklx\" (UID: \"051c87da-acbb-4941-8fea-57107abb4720\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vbklx" Oct 30 09:12:34.968750 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:34.968752 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/051c87da-acbb-4941-8fea-57107abb4720-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-vbklx\" (UID: \"051c87da-acbb-4941-8fea-57107abb4720\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vbklx" Oct 30 09:12:34.968982 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:34.968782 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9sgg\" (UniqueName: \"kubernetes.io/projected/051c87da-acbb-4941-8fea-57107abb4720-kube-api-access-k9sgg\") pod \"maas-default-gateway-openshift-default-77f556d647-vbklx\" (UID: \"051c87da-acbb-4941-8fea-57107abb4720\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vbklx" Oct 30 09:12:34.968982 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:34.968943 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/051c87da-acbb-4941-8fea-57107abb4720-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-vbklx\" (UID: \"051c87da-acbb-4941-8fea-57107abb4720\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vbklx" Oct 30 09:12:34.969081 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:34.968981 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/051c87da-acbb-4941-8fea-57107abb4720-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-vbklx\" (UID: \"051c87da-acbb-4941-8fea-57107abb4720\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vbklx" Oct 30 09:12:34.969081 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:34.969051 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/051c87da-acbb-4941-8fea-57107abb4720-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-vbklx\" (UID: \"051c87da-acbb-4941-8fea-57107abb4720\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vbklx" Oct 30 09:12:34.969160 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:34.969092 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/051c87da-acbb-4941-8fea-57107abb4720-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-vbklx\" (UID: \"051c87da-acbb-4941-8fea-57107abb4720\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vbklx" Oct 30 09:12:34.969160 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:34.969110 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/051c87da-acbb-4941-8fea-57107abb4720-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-vbklx\" (UID: \"051c87da-acbb-4941-8fea-57107abb4720\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vbklx" Oct 30 09:12:34.969265 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:34.969225 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/051c87da-acbb-4941-8fea-57107abb4720-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-vbklx\" (UID: \"051c87da-acbb-4941-8fea-57107abb4720\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vbklx" Oct 30 09:12:35.069736 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:35.069702 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/051c87da-acbb-4941-8fea-57107abb4720-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-vbklx\" (UID: \"051c87da-acbb-4941-8fea-57107abb4720\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vbklx" Oct 30 09:12:35.069736 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:35.069737 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/051c87da-acbb-4941-8fea-57107abb4720-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-vbklx\" (UID: \"051c87da-acbb-4941-8fea-57107abb4720\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vbklx" Oct 30 09:12:35.069956 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:35.069766 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/051c87da-acbb-4941-8fea-57107abb4720-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-vbklx\" (UID: \"051c87da-acbb-4941-8fea-57107abb4720\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vbklx" Oct 30 09:12:35.069956 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:35.069805 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/051c87da-acbb-4941-8fea-57107abb4720-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-vbklx\" (UID: \"051c87da-acbb-4941-8fea-57107abb4720\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vbklx" Oct 30 09:12:35.069956 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:35.069830 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/051c87da-acbb-4941-8fea-57107abb4720-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-vbklx\" (UID: \"051c87da-acbb-4941-8fea-57107abb4720\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vbklx" Oct 30 09:12:35.069956 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:35.069881 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/051c87da-acbb-4941-8fea-57107abb4720-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-vbklx\" (UID: \"051c87da-acbb-4941-8fea-57107abb4720\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vbklx" Oct 30 09:12:35.069956 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:35.069914 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/051c87da-acbb-4941-8fea-57107abb4720-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-vbklx\" (UID: \"051c87da-acbb-4941-8fea-57107abb4720\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vbklx" Oct 30 09:12:35.069956 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:35.069940 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/051c87da-acbb-4941-8fea-57107abb4720-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-vbklx\" (UID: \"051c87da-acbb-4941-8fea-57107abb4720\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vbklx" Oct 30 09:12:35.070267 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:35.069981 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9sgg\" (UniqueName: \"kubernetes.io/projected/051c87da-acbb-4941-8fea-57107abb4720-kube-api-access-k9sgg\") pod \"maas-default-gateway-openshift-default-77f556d647-vbklx\" (UID: \"051c87da-acbb-4941-8fea-57107abb4720\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vbklx" Oct 30 09:12:35.070267 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:35.070219 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/051c87da-acbb-4941-8fea-57107abb4720-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-vbklx\" (UID: \"051c87da-acbb-4941-8fea-57107abb4720\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vbklx" Oct 30 09:12:35.070366 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:35.070342 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/051c87da-acbb-4941-8fea-57107abb4720-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-vbklx\" (UID: \"051c87da-acbb-4941-8fea-57107abb4720\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vbklx" Oct 30 09:12:35.070437 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:35.070370 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/051c87da-acbb-4941-8fea-57107abb4720-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-vbklx\" (UID: \"051c87da-acbb-4941-8fea-57107abb4720\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vbklx" Oct 30 09:12:35.070522 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:35.070503 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/051c87da-acbb-4941-8fea-57107abb4720-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-vbklx\" (UID: \"051c87da-acbb-4941-8fea-57107abb4720\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vbklx" Oct 30 09:12:35.070613 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:35.070594 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/051c87da-acbb-4941-8fea-57107abb4720-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-vbklx\" (UID: \"051c87da-acbb-4941-8fea-57107abb4720\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vbklx" Oct 30 09:12:35.072083 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:35.072058 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/051c87da-acbb-4941-8fea-57107abb4720-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-vbklx\" (UID: \"051c87da-acbb-4941-8fea-57107abb4720\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vbklx" Oct 30 09:12:35.072369 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:35.072352 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/051c87da-acbb-4941-8fea-57107abb4720-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-vbklx\" (UID: \"051c87da-acbb-4941-8fea-57107abb4720\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vbklx" Oct 30 09:12:35.104300 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:35.104268 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/051c87da-acbb-4941-8fea-57107abb4720-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-vbklx\" (UID: \"051c87da-acbb-4941-8fea-57107abb4720\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vbklx" Oct 30 09:12:35.104431 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:35.104279 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9sgg\" (UniqueName: \"kubernetes.io/projected/051c87da-acbb-4941-8fea-57107abb4720-kube-api-access-k9sgg\") pod \"maas-default-gateway-openshift-default-77f556d647-vbklx\" (UID: \"051c87da-acbb-4941-8fea-57107abb4720\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vbklx" Oct 30 09:12:35.189655 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:35.189628 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vbklx" Oct 30 09:12:35.493601 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:35.493565 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vbklx"] Oct 30 09:12:35.493769 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:12:35.493743 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod051c87da_acbb_4941_8fea_57107abb4720.slice/crio-383947c5c234a369635d32dea3d8450c5733a845728a4dded149e8ddc3ff9c19 WatchSource:0}: Error finding container 383947c5c234a369635d32dea3d8450c5733a845728a4dded149e8ddc3ff9c19: Status 404 returned error can't find the container with id 383947c5c234a369635d32dea3d8450c5733a845728a4dded149e8ddc3ff9c19 Oct 30 09:12:36.105821 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:36.105788 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vbklx" event={"ID":"051c87da-acbb-4941-8fea-57107abb4720","Type":"ContainerStarted","Data":"383947c5c234a369635d32dea3d8450c5733a845728a4dded149e8ddc3ff9c19"} Oct 30 09:12:38.445603 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:38.445574 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:12:38.445858 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:38.445643 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:12:38.445858 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:38.445672 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:12:38.797556 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:38.797483 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-74d66bdc9c-wxffc"] Oct 30 09:12:38.801119 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:38.801097 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74d66bdc9c-wxffc" Oct 30 09:12:38.815199 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:38.815174 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-74d66bdc9c-wxffc"] Oct 30 09:12:38.829298 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:38.829264 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 09:12:38.829298 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:38.829271 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 09:12:38.829298 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:38.829299 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 30 09:12:38.829496 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:38.829280 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-cqn5p\"" Oct 30 09:12:38.906049 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:38.906011 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/664fe1ef-e0ec-4bc3-afe1-9eac486c4ac4-tls-cert\") pod \"authorino-74d66bdc9c-wxffc\" (UID: \"664fe1ef-e0ec-4bc3-afe1-9eac486c4ac4\") " pod="kuadrant-system/authorino-74d66bdc9c-wxffc" Oct 30 09:12:38.906049 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:38.906050 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnbxb\" (UniqueName: \"kubernetes.io/projected/664fe1ef-e0ec-4bc3-afe1-9eac486c4ac4-kube-api-access-dnbxb\") pod \"authorino-74d66bdc9c-wxffc\" (UID: \"664fe1ef-e0ec-4bc3-afe1-9eac486c4ac4\") " pod="kuadrant-system/authorino-74d66bdc9c-wxffc" Oct 30 09:12:39.007065 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:39.007022 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/664fe1ef-e0ec-4bc3-afe1-9eac486c4ac4-tls-cert\") pod \"authorino-74d66bdc9c-wxffc\" (UID: \"664fe1ef-e0ec-4bc3-afe1-9eac486c4ac4\") " pod="kuadrant-system/authorino-74d66bdc9c-wxffc" Oct 30 09:12:39.007065 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:39.007064 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnbxb\" (UniqueName: \"kubernetes.io/projected/664fe1ef-e0ec-4bc3-afe1-9eac486c4ac4-kube-api-access-dnbxb\") pod \"authorino-74d66bdc9c-wxffc\" (UID: \"664fe1ef-e0ec-4bc3-afe1-9eac486c4ac4\") " pod="kuadrant-system/authorino-74d66bdc9c-wxffc" Oct 30 09:12:39.009478 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:39.009454 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/664fe1ef-e0ec-4bc3-afe1-9eac486c4ac4-tls-cert\") pod \"authorino-74d66bdc9c-wxffc\" (UID: \"664fe1ef-e0ec-4bc3-afe1-9eac486c4ac4\") " pod="kuadrant-system/authorino-74d66bdc9c-wxffc" Oct 30 09:12:39.026109 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:39.026079 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnbxb\" (UniqueName: \"kubernetes.io/projected/664fe1ef-e0ec-4bc3-afe1-9eac486c4ac4-kube-api-access-dnbxb\") pod \"authorino-74d66bdc9c-wxffc\" (UID: \"664fe1ef-e0ec-4bc3-afe1-9eac486c4ac4\") " pod="kuadrant-system/authorino-74d66bdc9c-wxffc" Oct 30 09:12:39.120438 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:39.120334 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vbklx" event={"ID":"051c87da-acbb-4941-8fea-57107abb4720","Type":"ContainerStarted","Data":"310c6d2b893a6aad248318e2cc5beac3acff5e06d7cb6df6b723a2acea71d756"} Oct 30 09:12:39.129717 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:39.129698 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74d66bdc9c-wxffc" Oct 30 09:12:39.168740 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:39.168679 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vbklx" podStartSLOduration=2.219620918 podStartE2EDuration="5.168659605s" totalImagesPullingTime="2.949038687s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:12:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:12:39.161889512 +0000 UTC m=+1119.423450501" watchObservedRunningTime="2026-10-30 09:12:39.168659605 +0000 UTC m=+1119.430220594" Oct 30 09:12:39.194908 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:39.194877 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vbklx" Oct 30 09:12:39.196448 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:39.196424 4562 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-vbklx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 09:12:39.196526 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:39.196468 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vbklx" podUID="051c87da-acbb-4941-8fea-57107abb4720" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 09:12:39.461405 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:39.455424 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-74d66bdc9c-wxffc"] Oct 30 09:12:40.127138 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:40.127101 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-74d66bdc9c-wxffc" event={"ID":"664fe1ef-e0ec-4bc3-afe1-9eac486c4ac4","Type":"ContainerStarted","Data":"6c385495cf7aa4efc64a8acf58c79f94297a9ae72d5971ed3ed7c4f4ddd95db3"} Oct 30 09:12:40.190586 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:40.190552 4562 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-vbklx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 09:12:40.190749 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:40.190601 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vbklx" podUID="051c87da-acbb-4941-8fea-57107abb4720" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 09:12:41.134013 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:41.133974 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-74d66bdc9c-wxffc" event={"ID":"664fe1ef-e0ec-4bc3-afe1-9eac486c4ac4","Type":"ContainerStarted","Data":"b804636e9d9601b34f32d8bd42242f1457e0e066778e24553b902702c132b161"} Oct 30 09:12:41.184582 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:41.184525 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-74d66bdc9c-wxffc" podStartSLOduration=2.459212413 podStartE2EDuration="3.184511987s" totalImagesPullingTime="725.299574ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:12:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:12:41.182727167 +0000 UTC m=+1121.444288170" watchObservedRunningTime="2026-10-30 09:12:41.184511987 +0000 UTC m=+1121.446072975" Oct 30 09:12:41.190522 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:41.190488 4562 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-vbklx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 09:12:41.190679 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:41.190553 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vbklx" podUID="051c87da-acbb-4941-8fea-57107abb4720" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 09:12:42.190873 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:42.190832 4562 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-vbklx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 09:12:42.191547 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:42.190904 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vbklx" podUID="051c87da-acbb-4941-8fea-57107abb4720" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 09:12:43.190556 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:43.190521 4562 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-vbklx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 09:12:43.190738 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:43.190582 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vbklx" podUID="051c87da-acbb-4941-8fea-57107abb4720" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 09:12:44.190858 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:44.190819 4562 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-vbklx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 09:12:44.191346 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:44.190884 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vbklx" podUID="051c87da-acbb-4941-8fea-57107abb4720" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 09:12:45.190394 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:45.190341 4562 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-vbklx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 09:12:45.190568 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:45.190447 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vbklx" podUID="051c87da-acbb-4941-8fea-57107abb4720" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 09:12:46.894766 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:46.894710 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vbklx" Oct 30 09:12:46.895230 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:46.895199 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vbklx" Oct 30 09:12:46.896044 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:46.896019 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vbklx" Oct 30 09:12:49.929498 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:49.929463 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hl5zf"] Oct 30 09:12:49.931749 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:49.931732 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hl5zf" Oct 30 09:12:49.954713 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:49.954686 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hl5zf"] Oct 30 09:12:50.040369 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:50.040339 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b74384a-1957-4071-b9aa-e4f2c1e9cef0-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-hl5zf\" (UID: \"7b74384a-1957-4071-b9aa-e4f2c1e9cef0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hl5zf" Oct 30 09:12:50.040546 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:50.040371 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b74384a-1957-4071-b9aa-e4f2c1e9cef0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-hl5zf\" (UID: \"7b74384a-1957-4071-b9aa-e4f2c1e9cef0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hl5zf" Oct 30 09:12:50.040546 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:50.040416 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b74384a-1957-4071-b9aa-e4f2c1e9cef0-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-hl5zf\" (UID: \"7b74384a-1957-4071-b9aa-e4f2c1e9cef0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hl5zf" Oct 30 09:12:50.040546 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:50.040504 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b74384a-1957-4071-b9aa-e4f2c1e9cef0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-hl5zf\" (UID: \"7b74384a-1957-4071-b9aa-e4f2c1e9cef0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hl5zf" Oct 30 09:12:50.040683 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:50.040617 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b74384a-1957-4071-b9aa-e4f2c1e9cef0-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-hl5zf\" (UID: \"7b74384a-1957-4071-b9aa-e4f2c1e9cef0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hl5zf" Oct 30 09:12:50.040683 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:50.040634 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gnqt\" (UniqueName: \"kubernetes.io/projected/7b74384a-1957-4071-b9aa-e4f2c1e9cef0-kube-api-access-5gnqt\") pod \"maas-default-gateway-openshift-default-77f556d647-hl5zf\" (UID: \"7b74384a-1957-4071-b9aa-e4f2c1e9cef0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hl5zf" Oct 30 09:12:50.040683 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:50.040658 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b74384a-1957-4071-b9aa-e4f2c1e9cef0-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-hl5zf\" (UID: \"7b74384a-1957-4071-b9aa-e4f2c1e9cef0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hl5zf" Oct 30 09:12:50.040683 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:50.040677 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b74384a-1957-4071-b9aa-e4f2c1e9cef0-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-hl5zf\" (UID: \"7b74384a-1957-4071-b9aa-e4f2c1e9cef0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hl5zf" Oct 30 09:12:50.040815 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:50.040771 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b74384a-1957-4071-b9aa-e4f2c1e9cef0-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-hl5zf\" (UID: \"7b74384a-1957-4071-b9aa-e4f2c1e9cef0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hl5zf" Oct 30 09:12:50.141685 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:50.141654 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b74384a-1957-4071-b9aa-e4f2c1e9cef0-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-hl5zf\" (UID: \"7b74384a-1957-4071-b9aa-e4f2c1e9cef0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hl5zf" Oct 30 09:12:50.141685 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:50.141687 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gnqt\" (UniqueName: \"kubernetes.io/projected/7b74384a-1957-4071-b9aa-e4f2c1e9cef0-kube-api-access-5gnqt\") pod \"maas-default-gateway-openshift-default-77f556d647-hl5zf\" (UID: \"7b74384a-1957-4071-b9aa-e4f2c1e9cef0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hl5zf" Oct 30 09:12:50.141925 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:50.141710 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b74384a-1957-4071-b9aa-e4f2c1e9cef0-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-hl5zf\" (UID: \"7b74384a-1957-4071-b9aa-e4f2c1e9cef0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hl5zf" Oct 30 09:12:50.141925 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:50.141729 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b74384a-1957-4071-b9aa-e4f2c1e9cef0-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-hl5zf\" (UID: \"7b74384a-1957-4071-b9aa-e4f2c1e9cef0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hl5zf" Oct 30 09:12:50.141925 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:50.141760 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b74384a-1957-4071-b9aa-e4f2c1e9cef0-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-hl5zf\" (UID: \"7b74384a-1957-4071-b9aa-e4f2c1e9cef0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hl5zf" Oct 30 09:12:50.141925 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:50.141798 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b74384a-1957-4071-b9aa-e4f2c1e9cef0-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-hl5zf\" (UID: \"7b74384a-1957-4071-b9aa-e4f2c1e9cef0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hl5zf" Oct 30 09:12:50.141925 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:50.141824 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b74384a-1957-4071-b9aa-e4f2c1e9cef0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-hl5zf\" (UID: \"7b74384a-1957-4071-b9aa-e4f2c1e9cef0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hl5zf" Oct 30 09:12:50.142182 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:50.142137 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b74384a-1957-4071-b9aa-e4f2c1e9cef0-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-hl5zf\" (UID: \"7b74384a-1957-4071-b9aa-e4f2c1e9cef0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hl5zf" Oct 30 09:12:50.142238 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:50.142184 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b74384a-1957-4071-b9aa-e4f2c1e9cef0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-hl5zf\" (UID: \"7b74384a-1957-4071-b9aa-e4f2c1e9cef0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hl5zf" Oct 30 09:12:50.142238 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:50.142204 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b74384a-1957-4071-b9aa-e4f2c1e9cef0-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-hl5zf\" (UID: \"7b74384a-1957-4071-b9aa-e4f2c1e9cef0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hl5zf" Oct 30 09:12:50.142340 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:50.142307 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b74384a-1957-4071-b9aa-e4f2c1e9cef0-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-hl5zf\" (UID: \"7b74384a-1957-4071-b9aa-e4f2c1e9cef0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hl5zf" Oct 30 09:12:50.142340 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:50.142325 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b74384a-1957-4071-b9aa-e4f2c1e9cef0-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-hl5zf\" (UID: \"7b74384a-1957-4071-b9aa-e4f2c1e9cef0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hl5zf" Oct 30 09:12:50.142492 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:50.142472 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b74384a-1957-4071-b9aa-e4f2c1e9cef0-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-hl5zf\" (UID: \"7b74384a-1957-4071-b9aa-e4f2c1e9cef0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hl5zf" Oct 30 09:12:50.142789 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:50.142770 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b74384a-1957-4071-b9aa-e4f2c1e9cef0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-hl5zf\" (UID: \"7b74384a-1957-4071-b9aa-e4f2c1e9cef0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hl5zf" Oct 30 09:12:50.144166 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:50.144135 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b74384a-1957-4071-b9aa-e4f2c1e9cef0-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-hl5zf\" (UID: \"7b74384a-1957-4071-b9aa-e4f2c1e9cef0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hl5zf" Oct 30 09:12:50.144333 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:50.144315 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b74384a-1957-4071-b9aa-e4f2c1e9cef0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-hl5zf\" (UID: \"7b74384a-1957-4071-b9aa-e4f2c1e9cef0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hl5zf" Oct 30 09:12:50.162654 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:50.162625 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gnqt\" (UniqueName: \"kubernetes.io/projected/7b74384a-1957-4071-b9aa-e4f2c1e9cef0-kube-api-access-5gnqt\") pod \"maas-default-gateway-openshift-default-77f556d647-hl5zf\" (UID: \"7b74384a-1957-4071-b9aa-e4f2c1e9cef0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hl5zf" Oct 30 09:12:50.162816 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:50.162797 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b74384a-1957-4071-b9aa-e4f2c1e9cef0-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-hl5zf\" (UID: \"7b74384a-1957-4071-b9aa-e4f2c1e9cef0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hl5zf" Oct 30 09:12:50.253696 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:50.253666 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hl5zf" Oct 30 09:12:50.556861 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:12:50.556831 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b74384a_1957_4071_b9aa_e4f2c1e9cef0.slice/crio-3f17e7acd465da52d5c7e8d83c06a3c785c678f1fb8cd2f95e69a63cf7418835 WatchSource:0}: Error finding container 3f17e7acd465da52d5c7e8d83c06a3c785c678f1fb8cd2f95e69a63cf7418835: Status 404 returned error can't find the container with id 3f17e7acd465da52d5c7e8d83c06a3c785c678f1fb8cd2f95e69a63cf7418835 Oct 30 09:12:50.557128 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:50.557107 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hl5zf"] Oct 30 09:12:50.560104 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:50.560073 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:12:50.560192 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:50.560139 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:12:50.560192 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:50.560166 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:12:51.204760 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:51.204723 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hl5zf" event={"ID":"7b74384a-1957-4071-b9aa-e4f2c1e9cef0","Type":"ContainerStarted","Data":"9e12e151c95acb8f5b101a8ea725b397fd7b6d002029fc51eaf32e7d8c935a6b"} Oct 30 09:12:51.205173 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:51.204778 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hl5zf" event={"ID":"7b74384a-1957-4071-b9aa-e4f2c1e9cef0","Type":"ContainerStarted","Data":"3f17e7acd465da52d5c7e8d83c06a3c785c678f1fb8cd2f95e69a63cf7418835"} Oct 30 09:12:51.258404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:51.257469 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hl5zf" Oct 30 09:12:51.263636 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:51.263582 4562 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-hl5zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Oct 30 09:12:51.263817 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:51.263659 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hl5zf" podUID="7b74384a-1957-4071-b9aa-e4f2c1e9cef0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Oct 30 09:12:52.259404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:52.258853 4562 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-hl5zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Oct 30 09:12:52.259404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:52.258932 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hl5zf" podUID="7b74384a-1957-4071-b9aa-e4f2c1e9cef0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Oct 30 09:12:53.262407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:53.258822 4562 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-hl5zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Oct 30 09:12:53.262407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:53.258929 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hl5zf" podUID="7b74384a-1957-4071-b9aa-e4f2c1e9cef0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Oct 30 09:12:54.258405 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:54.257773 4562 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-hl5zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Oct 30 09:12:54.258405 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:54.257833 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hl5zf" podUID="7b74384a-1957-4071-b9aa-e4f2c1e9cef0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Oct 30 09:12:55.258406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:55.255803 4562 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-hl5zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Oct 30 09:12:55.258406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:55.255866 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hl5zf" podUID="7b74384a-1957-4071-b9aa-e4f2c1e9cef0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Oct 30 09:12:56.259405 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:56.258798 4562 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-hl5zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Oct 30 09:12:56.259405 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:56.258864 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hl5zf" podUID="7b74384a-1957-4071-b9aa-e4f2c1e9cef0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Oct 30 09:12:57.254351 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:57.254315 4562 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-hl5zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Oct 30 09:12:57.254561 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:57.254366 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hl5zf" podUID="7b74384a-1957-4071-b9aa-e4f2c1e9cef0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Oct 30 09:12:58.595657 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:12:58.595616 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:12:58.756987 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:58.756953 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hl5zf" Oct 30 09:12:58.757512 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:58.757489 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hl5zf" Oct 30 09:12:58.758219 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:58.758202 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hl5zf" Oct 30 09:12:58.819488 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:12:58.815870 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hl5zf" podStartSLOduration=9.81584528 podStartE2EDuration="9.81584528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:12:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:12:51.263754251 +0000 UTC m=+1131.525315241" watchObservedRunningTime="2026-10-30 09:12:58.81584528 +0000 UTC m=+1139.077406269" Oct 30 09:13:08.063177 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.063140 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6"] Oct 30 09:13:08.073146 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.073116 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6" Oct 30 09:13:08.086850 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.086824 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6"] Oct 30 09:13:08.119575 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.119533 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26fc1564-372f-491d-93aa-8daa11a251ee-istio-podinfo\") pod \"e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6\" (UID: \"26fc1564-372f-491d-93aa-8daa11a251ee\") " pod="openshift-ingress/e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6" Oct 30 09:13:08.119764 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.119592 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26fc1564-372f-491d-93aa-8daa11a251ee-istiod-ca-cert\") pod \"e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6\" (UID: \"26fc1564-372f-491d-93aa-8daa11a251ee\") " pod="openshift-ingress/e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6" Oct 30 09:13:08.119764 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.119687 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26fc1564-372f-491d-93aa-8daa11a251ee-istio-data\") pod \"e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6\" (UID: \"26fc1564-372f-491d-93aa-8daa11a251ee\") " pod="openshift-ingress/e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6" Oct 30 09:13:08.119764 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.119735 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26fc1564-372f-491d-93aa-8daa11a251ee-istio-token\") pod \"e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6\" (UID: \"26fc1564-372f-491d-93aa-8daa11a251ee\") " pod="openshift-ingress/e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6" Oct 30 09:13:08.119910 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.119801 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26fc1564-372f-491d-93aa-8daa11a251ee-workload-certs\") pod \"e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6\" (UID: \"26fc1564-372f-491d-93aa-8daa11a251ee\") " pod="openshift-ingress/e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6" Oct 30 09:13:08.119910 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.119846 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26fc1564-372f-491d-93aa-8daa11a251ee-istio-envoy\") pod \"e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6\" (UID: \"26fc1564-372f-491d-93aa-8daa11a251ee\") " pod="openshift-ingress/e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6" Oct 30 09:13:08.119910 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.119877 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26fc1564-372f-491d-93aa-8daa11a251ee-credential-socket\") pod \"e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6\" (UID: \"26fc1564-372f-491d-93aa-8daa11a251ee\") " pod="openshift-ingress/e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6" Oct 30 09:13:08.120035 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.119908 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26fc1564-372f-491d-93aa-8daa11a251ee-workload-socket\") pod \"e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6\" (UID: \"26fc1564-372f-491d-93aa-8daa11a251ee\") " pod="openshift-ingress/e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6" Oct 30 09:13:08.120035 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.119968 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cshr\" (UniqueName: \"kubernetes.io/projected/26fc1564-372f-491d-93aa-8daa11a251ee-kube-api-access-8cshr\") pod \"e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6\" (UID: \"26fc1564-372f-491d-93aa-8daa11a251ee\") " pod="openshift-ingress/e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6" Oct 30 09:13:08.130479 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.130446 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-70374d4e-openshift-default-dockercfg-wpbxb\"" Oct 30 09:13:08.220908 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.220876 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26fc1564-372f-491d-93aa-8daa11a251ee-credential-socket\") pod \"e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6\" (UID: \"26fc1564-372f-491d-93aa-8daa11a251ee\") " pod="openshift-ingress/e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6" Oct 30 09:13:08.221086 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.220923 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26fc1564-372f-491d-93aa-8daa11a251ee-workload-socket\") pod \"e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6\" (UID: \"26fc1564-372f-491d-93aa-8daa11a251ee\") " pod="openshift-ingress/e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6" Oct 30 09:13:08.221086 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.220948 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cshr\" (UniqueName: \"kubernetes.io/projected/26fc1564-372f-491d-93aa-8daa11a251ee-kube-api-access-8cshr\") pod \"e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6\" (UID: \"26fc1564-372f-491d-93aa-8daa11a251ee\") " pod="openshift-ingress/e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6" Oct 30 09:13:08.221086 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.221063 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26fc1564-372f-491d-93aa-8daa11a251ee-istio-podinfo\") pod \"e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6\" (UID: \"26fc1564-372f-491d-93aa-8daa11a251ee\") " pod="openshift-ingress/e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6" Oct 30 09:13:08.221250 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.221094 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26fc1564-372f-491d-93aa-8daa11a251ee-istiod-ca-cert\") pod \"e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6\" (UID: \"26fc1564-372f-491d-93aa-8daa11a251ee\") " pod="openshift-ingress/e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6" Oct 30 09:13:08.221250 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.221130 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26fc1564-372f-491d-93aa-8daa11a251ee-istio-data\") pod \"e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6\" (UID: \"26fc1564-372f-491d-93aa-8daa11a251ee\") " pod="openshift-ingress/e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6" Oct 30 09:13:08.221250 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.221172 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26fc1564-372f-491d-93aa-8daa11a251ee-istio-token\") pod \"e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6\" (UID: \"26fc1564-372f-491d-93aa-8daa11a251ee\") " pod="openshift-ingress/e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6" Oct 30 09:13:08.221250 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.221198 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26fc1564-372f-491d-93aa-8daa11a251ee-workload-certs\") pod \"e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6\" (UID: \"26fc1564-372f-491d-93aa-8daa11a251ee\") " pod="openshift-ingress/e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6" Oct 30 09:13:08.221478 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.221257 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26fc1564-372f-491d-93aa-8daa11a251ee-istio-envoy\") pod \"e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6\" (UID: \"26fc1564-372f-491d-93aa-8daa11a251ee\") " pod="openshift-ingress/e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6" Oct 30 09:13:08.221478 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.221431 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26fc1564-372f-491d-93aa-8daa11a251ee-workload-socket\") pod \"e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6\" (UID: \"26fc1564-372f-491d-93aa-8daa11a251ee\") " pod="openshift-ingress/e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6" Oct 30 09:13:08.221791 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.221638 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26fc1564-372f-491d-93aa-8daa11a251ee-istio-data\") pod \"e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6\" (UID: \"26fc1564-372f-491d-93aa-8daa11a251ee\") " pod="openshift-ingress/e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6" Oct 30 09:13:08.222098 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.222072 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26fc1564-372f-491d-93aa-8daa11a251ee-istiod-ca-cert\") pod \"e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6\" (UID: \"26fc1564-372f-491d-93aa-8daa11a251ee\") " pod="openshift-ingress/e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6" Oct 30 09:13:08.222705 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.222673 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26fc1564-372f-491d-93aa-8daa11a251ee-workload-certs\") pod \"e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6\" (UID: \"26fc1564-372f-491d-93aa-8daa11a251ee\") " pod="openshift-ingress/e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6" Oct 30 09:13:08.222872 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.222753 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26fc1564-372f-491d-93aa-8daa11a251ee-credential-socket\") pod \"e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6\" (UID: \"26fc1564-372f-491d-93aa-8daa11a251ee\") " pod="openshift-ingress/e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6" Oct 30 09:13:08.225826 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.225806 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26fc1564-372f-491d-93aa-8daa11a251ee-istio-envoy\") pod \"e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6\" (UID: \"26fc1564-372f-491d-93aa-8daa11a251ee\") " pod="openshift-ingress/e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6" Oct 30 09:13:08.226079 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.226054 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26fc1564-372f-491d-93aa-8daa11a251ee-istio-podinfo\") pod \"e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6\" (UID: \"26fc1564-372f-491d-93aa-8daa11a251ee\") " pod="openshift-ingress/e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6" Oct 30 09:13:08.241687 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.241656 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26fc1564-372f-491d-93aa-8daa11a251ee-istio-token\") pod \"e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6\" (UID: \"26fc1564-372f-491d-93aa-8daa11a251ee\") " pod="openshift-ingress/e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6" Oct 30 09:13:08.242024 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.242001 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cshr\" (UniqueName: \"kubernetes.io/projected/26fc1564-372f-491d-93aa-8daa11a251ee-kube-api-access-8cshr\") pod \"e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6\" (UID: \"26fc1564-372f-491d-93aa-8daa11a251ee\") " pod="openshift-ingress/e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6" Oct 30 09:13:08.391230 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.391137 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6" Oct 30 09:13:08.646915 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.646836 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2"] Oct 30 09:13:08.656960 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.656930 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2" Oct 30 09:13:08.674650 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.674134 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-6233a0dd-gw-openshift-default-dockercfg-tz7nd\"" Oct 30 09:13:08.678548 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.678524 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2"] Oct 30 09:13:08.727701 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.727670 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/267dbedb-3d29-43ab-8619-2ef3ae10d6a7-istiod-ca-cert\") pod \"e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2\" (UID: \"267dbedb-3d29-43ab-8619-2ef3ae10d6a7\") " pod="openshift-ingress/e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2" Oct 30 09:13:08.727817 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.727702 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6"] Oct 30 09:13:08.727817 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:13:08.727696 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26fc1564_372f_491d_93aa_8daa11a251ee.slice/crio-128b6f2a89fc14c490b1de1316d21f65de30b93b40c6784e6a13c3893a745ab4 WatchSource:0}: Error finding container 128b6f2a89fc14c490b1de1316d21f65de30b93b40c6784e6a13c3893a745ab4: Status 404 returned error can't find the container with id 128b6f2a89fc14c490b1de1316d21f65de30b93b40c6784e6a13c3893a745ab4 Oct 30 09:13:08.727817 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.727761 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/267dbedb-3d29-43ab-8619-2ef3ae10d6a7-credential-socket\") pod \"e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2\" (UID: \"267dbedb-3d29-43ab-8619-2ef3ae10d6a7\") " pod="openshift-ingress/e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2" Oct 30 09:13:08.728010 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.727847 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/267dbedb-3d29-43ab-8619-2ef3ae10d6a7-istio-data\") pod \"e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2\" (UID: \"267dbedb-3d29-43ab-8619-2ef3ae10d6a7\") " pod="openshift-ingress/e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2" Oct 30 09:13:08.728010 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.727887 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/267dbedb-3d29-43ab-8619-2ef3ae10d6a7-istio-token\") pod \"e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2\" (UID: \"267dbedb-3d29-43ab-8619-2ef3ae10d6a7\") " pod="openshift-ingress/e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2" Oct 30 09:13:08.728010 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.727917 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/267dbedb-3d29-43ab-8619-2ef3ae10d6a7-workload-certs\") pod \"e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2\" (UID: \"267dbedb-3d29-43ab-8619-2ef3ae10d6a7\") " pod="openshift-ingress/e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2" Oct 30 09:13:08.728159 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.728023 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/267dbedb-3d29-43ab-8619-2ef3ae10d6a7-istio-podinfo\") pod \"e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2\" (UID: \"267dbedb-3d29-43ab-8619-2ef3ae10d6a7\") " pod="openshift-ingress/e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2" Oct 30 09:13:08.728159 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.728064 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/267dbedb-3d29-43ab-8619-2ef3ae10d6a7-workload-socket\") pod \"e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2\" (UID: \"267dbedb-3d29-43ab-8619-2ef3ae10d6a7\") " pod="openshift-ingress/e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2" Oct 30 09:13:08.728159 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.728087 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkcl9\" (UniqueName: \"kubernetes.io/projected/267dbedb-3d29-43ab-8619-2ef3ae10d6a7-kube-api-access-rkcl9\") pod \"e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2\" (UID: \"267dbedb-3d29-43ab-8619-2ef3ae10d6a7\") " pod="openshift-ingress/e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2" Oct 30 09:13:08.728301 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.728163 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/267dbedb-3d29-43ab-8619-2ef3ae10d6a7-istio-envoy\") pod \"e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2\" (UID: \"267dbedb-3d29-43ab-8619-2ef3ae10d6a7\") " pod="openshift-ingress/e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2" Oct 30 09:13:08.729963 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.729934 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:13:08.730044 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.729994 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:13:08.730044 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.730021 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:13:08.829323 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.829287 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/267dbedb-3d29-43ab-8619-2ef3ae10d6a7-workload-socket\") pod \"e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2\" (UID: \"267dbedb-3d29-43ab-8619-2ef3ae10d6a7\") " pod="openshift-ingress/e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2" Oct 30 09:13:08.829502 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.829343 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkcl9\" (UniqueName: \"kubernetes.io/projected/267dbedb-3d29-43ab-8619-2ef3ae10d6a7-kube-api-access-rkcl9\") pod \"e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2\" (UID: \"267dbedb-3d29-43ab-8619-2ef3ae10d6a7\") " pod="openshift-ingress/e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2" Oct 30 09:13:08.829502 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.829373 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/267dbedb-3d29-43ab-8619-2ef3ae10d6a7-istio-envoy\") pod \"e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2\" (UID: \"267dbedb-3d29-43ab-8619-2ef3ae10d6a7\") " pod="openshift-ingress/e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2" Oct 30 09:13:08.829502 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.829429 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/267dbedb-3d29-43ab-8619-2ef3ae10d6a7-istiod-ca-cert\") pod \"e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2\" (UID: \"267dbedb-3d29-43ab-8619-2ef3ae10d6a7\") " pod="openshift-ingress/e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2" Oct 30 09:13:08.829651 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.829511 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/267dbedb-3d29-43ab-8619-2ef3ae10d6a7-credential-socket\") pod \"e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2\" (UID: \"267dbedb-3d29-43ab-8619-2ef3ae10d6a7\") " pod="openshift-ingress/e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2" Oct 30 09:13:08.829651 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.829535 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/267dbedb-3d29-43ab-8619-2ef3ae10d6a7-istio-data\") pod \"e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2\" (UID: \"267dbedb-3d29-43ab-8619-2ef3ae10d6a7\") " pod="openshift-ingress/e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2" Oct 30 09:13:08.829651 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.829574 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/267dbedb-3d29-43ab-8619-2ef3ae10d6a7-istio-token\") pod \"e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2\" (UID: \"267dbedb-3d29-43ab-8619-2ef3ae10d6a7\") " pod="openshift-ingress/e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2" Oct 30 09:13:08.829651 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.829606 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/267dbedb-3d29-43ab-8619-2ef3ae10d6a7-workload-certs\") pod \"e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2\" (UID: \"267dbedb-3d29-43ab-8619-2ef3ae10d6a7\") " pod="openshift-ingress/e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2" Oct 30 09:13:08.829775 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.829685 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/267dbedb-3d29-43ab-8619-2ef3ae10d6a7-istio-podinfo\") pod \"e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2\" (UID: \"267dbedb-3d29-43ab-8619-2ef3ae10d6a7\") " pod="openshift-ingress/e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2" Oct 30 09:13:08.830768 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.830669 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/267dbedb-3d29-43ab-8619-2ef3ae10d6a7-istio-data\") pod \"e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2\" (UID: \"267dbedb-3d29-43ab-8619-2ef3ae10d6a7\") " pod="openshift-ingress/e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2" Oct 30 09:13:08.830768 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.830744 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/267dbedb-3d29-43ab-8619-2ef3ae10d6a7-credential-socket\") pod \"e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2\" (UID: \"267dbedb-3d29-43ab-8619-2ef3ae10d6a7\") " pod="openshift-ingress/e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2" Oct 30 09:13:08.830928 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.830832 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/267dbedb-3d29-43ab-8619-2ef3ae10d6a7-workload-socket\") pod \"e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2\" (UID: \"267dbedb-3d29-43ab-8619-2ef3ae10d6a7\") " pod="openshift-ingress/e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2" Oct 30 09:13:08.830985 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.830970 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/267dbedb-3d29-43ab-8619-2ef3ae10d6a7-istiod-ca-cert\") pod \"e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2\" (UID: \"267dbedb-3d29-43ab-8619-2ef3ae10d6a7\") " pod="openshift-ingress/e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2" Oct 30 09:13:08.831263 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.831233 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/267dbedb-3d29-43ab-8619-2ef3ae10d6a7-workload-certs\") pod \"e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2\" (UID: \"267dbedb-3d29-43ab-8619-2ef3ae10d6a7\") " pod="openshift-ingress/e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2" Oct 30 09:13:08.835419 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.832971 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/267dbedb-3d29-43ab-8619-2ef3ae10d6a7-istio-envoy\") pod \"e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2\" (UID: \"267dbedb-3d29-43ab-8619-2ef3ae10d6a7\") " pod="openshift-ingress/e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2" Oct 30 09:13:08.835419 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.833429 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/267dbedb-3d29-43ab-8619-2ef3ae10d6a7-istio-podinfo\") pod \"e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2\" (UID: \"267dbedb-3d29-43ab-8619-2ef3ae10d6a7\") " pod="openshift-ingress/e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2" Oct 30 09:13:08.857696 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.857663 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkcl9\" (UniqueName: \"kubernetes.io/projected/267dbedb-3d29-43ab-8619-2ef3ae10d6a7-kube-api-access-rkcl9\") pod \"e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2\" (UID: \"267dbedb-3d29-43ab-8619-2ef3ae10d6a7\") " pod="openshift-ingress/e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2" Oct 30 09:13:08.861150 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.861126 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/267dbedb-3d29-43ab-8619-2ef3ae10d6a7-istio-token\") pod \"e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2\" (UID: \"267dbedb-3d29-43ab-8619-2ef3ae10d6a7\") " pod="openshift-ingress/e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2" Oct 30 09:13:08.985507 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:08.985474 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2" Oct 30 09:13:09.284743 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:09.284669 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6" event={"ID":"26fc1564-372f-491d-93aa-8daa11a251ee","Type":"ContainerStarted","Data":"d1826dded4e2230f90b913cbdc18360b65fd3ce3ba7dda1f59e88d401a338369"} Oct 30 09:13:09.284743 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:09.284719 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6" event={"ID":"26fc1564-372f-491d-93aa-8daa11a251ee","Type":"ContainerStarted","Data":"128b6f2a89fc14c490b1de1316d21f65de30b93b40c6784e6a13c3893a745ab4"} Oct 30 09:13:09.296564 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:09.296541 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2"] Oct 30 09:13:09.297191 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:13:09.297167 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod267dbedb_3d29_43ab_8619_2ef3ae10d6a7.slice/crio-30a341c867b15d05ca1eda311fbd61c037525f5694e7d15e6c23d54b9027ba1c WatchSource:0}: Error finding container 30a341c867b15d05ca1eda311fbd61c037525f5694e7d15e6c23d54b9027ba1c: Status 404 returned error can't find the container with id 30a341c867b15d05ca1eda311fbd61c037525f5694e7d15e6c23d54b9027ba1c Oct 30 09:13:09.299362 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:09.299327 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:13:09.299461 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:09.299420 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:13:09.299506 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:09.299467 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:13:09.335063 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:09.335016 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6" podStartSLOduration=1.335002314 podStartE2EDuration="1.335002314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:13:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:13:09.333288356 +0000 UTC m=+1149.594849367" watchObservedRunningTime="2026-10-30 09:13:09.335002314 +0000 UTC m=+1149.596563302" Oct 30 09:13:09.391977 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:09.391939 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6" Oct 30 09:13:09.397750 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:09.397721 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6" Oct 30 09:13:09.877444 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:09.877409 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l"] Oct 30 09:13:09.881586 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:09.881560 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l" Oct 30 09:13:09.898244 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:09.898213 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-4c806f9c-openshift-default-dockercfg-88lqm\"" Oct 30 09:13:09.906488 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:09.906462 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l"] Oct 30 09:13:09.941168 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:09.941132 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05198b8c-2250-4810-873c-55d1ef91077a-istiod-ca-cert\") pod \"e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l\" (UID: \"05198b8c-2250-4810-873c-55d1ef91077a\") " pod="openshift-ingress/e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l" Oct 30 09:13:09.941349 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:09.941200 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05198b8c-2250-4810-873c-55d1ef91077a-istio-data\") pod \"e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l\" (UID: \"05198b8c-2250-4810-873c-55d1ef91077a\") " pod="openshift-ingress/e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l" Oct 30 09:13:09.941349 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:09.941220 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05198b8c-2250-4810-873c-55d1ef91077a-workload-socket\") pod \"e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l\" (UID: \"05198b8c-2250-4810-873c-55d1ef91077a\") " pod="openshift-ingress/e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l" Oct 30 09:13:09.941522 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:09.941371 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05198b8c-2250-4810-873c-55d1ef91077a-istio-envoy\") pod \"e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l\" (UID: \"05198b8c-2250-4810-873c-55d1ef91077a\") " pod="openshift-ingress/e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l" Oct 30 09:13:09.941522 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:09.941404 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05198b8c-2250-4810-873c-55d1ef91077a-istio-token\") pod \"e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l\" (UID: \"05198b8c-2250-4810-873c-55d1ef91077a\") " pod="openshift-ingress/e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l" Oct 30 09:13:09.941522 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:09.941439 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzzf7\" (UniqueName: \"kubernetes.io/projected/05198b8c-2250-4810-873c-55d1ef91077a-kube-api-access-xzzf7\") pod \"e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l\" (UID: \"05198b8c-2250-4810-873c-55d1ef91077a\") " pod="openshift-ingress/e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l" Oct 30 09:13:09.941669 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:09.941568 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05198b8c-2250-4810-873c-55d1ef91077a-istio-podinfo\") pod \"e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l\" (UID: \"05198b8c-2250-4810-873c-55d1ef91077a\") " pod="openshift-ingress/e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l" Oct 30 09:13:09.941735 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:09.941664 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05198b8c-2250-4810-873c-55d1ef91077a-credential-socket\") pod \"e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l\" (UID: \"05198b8c-2250-4810-873c-55d1ef91077a\") " pod="openshift-ingress/e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l" Oct 30 09:13:09.941735 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:09.941692 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05198b8c-2250-4810-873c-55d1ef91077a-workload-certs\") pod \"e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l\" (UID: \"05198b8c-2250-4810-873c-55d1ef91077a\") " pod="openshift-ingress/e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l" Oct 30 09:13:10.044095 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:10.044040 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05198b8c-2250-4810-873c-55d1ef91077a-istio-envoy\") pod \"e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l\" (UID: \"05198b8c-2250-4810-873c-55d1ef91077a\") " pod="openshift-ingress/e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l" Oct 30 09:13:10.044297 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:10.044106 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05198b8c-2250-4810-873c-55d1ef91077a-istio-token\") pod \"e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l\" (UID: \"05198b8c-2250-4810-873c-55d1ef91077a\") " pod="openshift-ingress/e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l" Oct 30 09:13:10.044297 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:10.044148 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzzf7\" (UniqueName: \"kubernetes.io/projected/05198b8c-2250-4810-873c-55d1ef91077a-kube-api-access-xzzf7\") pod \"e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l\" (UID: \"05198b8c-2250-4810-873c-55d1ef91077a\") " pod="openshift-ingress/e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l" Oct 30 09:13:10.044297 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:10.044220 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05198b8c-2250-4810-873c-55d1ef91077a-istio-podinfo\") pod \"e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l\" (UID: \"05198b8c-2250-4810-873c-55d1ef91077a\") " pod="openshift-ingress/e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l" Oct 30 09:13:10.044522 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:10.044304 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05198b8c-2250-4810-873c-55d1ef91077a-credential-socket\") pod \"e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l\" (UID: \"05198b8c-2250-4810-873c-55d1ef91077a\") " pod="openshift-ingress/e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l" Oct 30 09:13:10.044522 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:10.044336 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05198b8c-2250-4810-873c-55d1ef91077a-workload-certs\") pod \"e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l\" (UID: \"05198b8c-2250-4810-873c-55d1ef91077a\") " pod="openshift-ingress/e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l" Oct 30 09:13:10.044522 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:10.044396 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05198b8c-2250-4810-873c-55d1ef91077a-istiod-ca-cert\") pod \"e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l\" (UID: \"05198b8c-2250-4810-873c-55d1ef91077a\") " pod="openshift-ingress/e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l" Oct 30 09:13:10.044522 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:10.044453 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05198b8c-2250-4810-873c-55d1ef91077a-istio-data\") pod \"e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l\" (UID: \"05198b8c-2250-4810-873c-55d1ef91077a\") " pod="openshift-ingress/e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l" Oct 30 09:13:10.044522 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:10.044484 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05198b8c-2250-4810-873c-55d1ef91077a-workload-socket\") pod \"e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l\" (UID: \"05198b8c-2250-4810-873c-55d1ef91077a\") " pod="openshift-ingress/e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l" Oct 30 09:13:10.044894 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:10.044867 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05198b8c-2250-4810-873c-55d1ef91077a-workload-socket\") pod \"e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l\" (UID: \"05198b8c-2250-4810-873c-55d1ef91077a\") " pod="openshift-ingress/e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l" Oct 30 09:13:10.045160 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:10.045131 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05198b8c-2250-4810-873c-55d1ef91077a-credential-socket\") pod \"e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l\" (UID: \"05198b8c-2250-4810-873c-55d1ef91077a\") " pod="openshift-ingress/e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l" Oct 30 09:13:10.045427 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:10.045403 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05198b8c-2250-4810-873c-55d1ef91077a-workload-certs\") pod \"e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l\" (UID: \"05198b8c-2250-4810-873c-55d1ef91077a\") " pod="openshift-ingress/e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l" Oct 30 09:13:10.046123 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:10.046099 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05198b8c-2250-4810-873c-55d1ef91077a-istio-data\") pod \"e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l\" (UID: \"05198b8c-2250-4810-873c-55d1ef91077a\") " pod="openshift-ingress/e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l" Oct 30 09:13:10.046624 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:10.046245 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05198b8c-2250-4810-873c-55d1ef91077a-istiod-ca-cert\") pod \"e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l\" (UID: \"05198b8c-2250-4810-873c-55d1ef91077a\") " pod="openshift-ingress/e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l" Oct 30 09:13:10.066751 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:10.066667 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05198b8c-2250-4810-873c-55d1ef91077a-istio-podinfo\") pod \"e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l\" (UID: \"05198b8c-2250-4810-873c-55d1ef91077a\") " pod="openshift-ingress/e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l" Oct 30 09:13:10.075824 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:10.075208 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzzf7\" (UniqueName: \"kubernetes.io/projected/05198b8c-2250-4810-873c-55d1ef91077a-kube-api-access-xzzf7\") pod \"e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l\" (UID: \"05198b8c-2250-4810-873c-55d1ef91077a\") " pod="openshift-ingress/e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l" Oct 30 09:13:10.078039 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:10.077982 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05198b8c-2250-4810-873c-55d1ef91077a-istio-envoy\") pod \"e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l\" (UID: \"05198b8c-2250-4810-873c-55d1ef91077a\") " pod="openshift-ingress/e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l" Oct 30 09:13:10.087942 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:10.087906 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05198b8c-2250-4810-873c-55d1ef91077a-istio-token\") pod \"e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l\" (UID: \"05198b8c-2250-4810-873c-55d1ef91077a\") " pod="openshift-ingress/e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l" Oct 30 09:13:10.195501 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:10.195474 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l" Oct 30 09:13:10.291368 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:10.291333 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2" event={"ID":"267dbedb-3d29-43ab-8619-2ef3ae10d6a7","Type":"ContainerStarted","Data":"4659424aca33e89e42602a460866f016785afc0b80c74c331b244ef159f719af"} Oct 30 09:13:10.291804 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:10.291416 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2" event={"ID":"267dbedb-3d29-43ab-8619-2ef3ae10d6a7","Type":"ContainerStarted","Data":"30a341c867b15d05ca1eda311fbd61c037525f5694e7d15e6c23d54b9027ba1c"} Oct 30 09:13:10.291804 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:10.291533 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6" Oct 30 09:13:10.292531 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:10.292510 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6" Oct 30 09:13:10.335355 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:10.335309 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2" podStartSLOduration=2.335295099 podStartE2EDuration="2.335295099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:13:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:13:10.333741217 +0000 UTC m=+1150.595302202" watchObservedRunningTime="2026-10-30 09:13:10.335295099 +0000 UTC m=+1150.596856086" Oct 30 09:13:10.509496 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:10.509454 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l"] Oct 30 09:13:10.515183 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:13:10.515153 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05198b8c_2250_4810_873c_55d1ef91077a.slice/crio-a1a09c86c1c0e59947669a24de11f4478c8a2c46440a383299fe26e87ddfd448 WatchSource:0}: Error finding container a1a09c86c1c0e59947669a24de11f4478c8a2c46440a383299fe26e87ddfd448: Status 404 returned error can't find the container with id a1a09c86c1c0e59947669a24de11f4478c8a2c46440a383299fe26e87ddfd448 Oct 30 09:13:10.517266 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:10.517238 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:13:10.517330 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:10.517312 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:13:10.517365 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:10.517340 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:13:10.986647 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:10.986609 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2" Oct 30 09:13:10.991758 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:10.991728 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2" Oct 30 09:13:11.298039 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:11.297939 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l" event={"ID":"05198b8c-2250-4810-873c-55d1ef91077a","Type":"ContainerStarted","Data":"ab20d321378cfc63bd2f5389776c5b85efbc06a6503857c89f5e2034a18f8238"} Oct 30 09:13:11.298039 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:11.297993 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l" event={"ID":"05198b8c-2250-4810-873c-55d1ef91077a","Type":"ContainerStarted","Data":"a1a09c86c1c0e59947669a24de11f4478c8a2c46440a383299fe26e87ddfd448"} Oct 30 09:13:11.298548 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:11.298246 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2" Oct 30 09:13:11.299264 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:11.299246 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2" Oct 30 09:13:11.364270 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:11.364219 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l" podStartSLOduration=2.36420637 podStartE2EDuration="2.36420637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:13:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:13:11.362513551 +0000 UTC m=+1151.624074559" watchObservedRunningTime="2026-10-30 09:13:11.36420637 +0000 UTC m=+1151.625767358" Oct 30 09:13:12.195898 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:12.195861 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l" Oct 30 09:13:12.201109 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:12.201086 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l" Oct 30 09:13:12.302083 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:12.302053 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l" Oct 30 09:13:12.303092 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:12.303075 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l" Oct 30 09:13:13.921509 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:13.921463 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2ffc2a58-6cf8684647-d28fv"] Oct 30 09:13:13.925370 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:13.925350 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ffc2a58-6cf8684647-d28fv" Oct 30 09:13:13.942138 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:13.942112 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2ffc2a58-6cf8684647-d28fv"] Oct 30 09:13:13.996155 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:13.996120 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ae2b5d9-beef-4d8b-9351-77be0eff9d0d-openshift-service-ca\") pod \"maas-api-e2e-ait-2ffc2a58-6cf8684647-d28fv\" (UID: \"4ae2b5d9-beef-4d8b-9351-77be0eff9d0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ffc2a58-6cf8684647-d28fv" Oct 30 09:13:13.996351 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:13.996177 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5w54\" (UniqueName: \"kubernetes.io/projected/4ae2b5d9-beef-4d8b-9351-77be0eff9d0d-kube-api-access-j5w54\") pod \"maas-api-e2e-ait-2ffc2a58-6cf8684647-d28fv\" (UID: \"4ae2b5d9-beef-4d8b-9351-77be0eff9d0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ffc2a58-6cf8684647-d28fv" Oct 30 09:13:13.996351 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:13.996329 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ae2b5d9-beef-4d8b-9351-77be0eff9d0d-metrics-certs\") pod \"maas-api-e2e-ait-2ffc2a58-6cf8684647-d28fv\" (UID: \"4ae2b5d9-beef-4d8b-9351-77be0eff9d0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ffc2a58-6cf8684647-d28fv" Oct 30 09:13:13.996495 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:13.996390 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ae2b5d9-beef-4d8b-9351-77be0eff9d0d-maas-api-tls\") pod \"maas-api-e2e-ait-2ffc2a58-6cf8684647-d28fv\" (UID: \"4ae2b5d9-beef-4d8b-9351-77be0eff9d0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ffc2a58-6cf8684647-d28fv" Oct 30 09:13:14.029430 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:14.029391 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-2ffc2a58-serving-cert\"" Oct 30 09:13:14.029628 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:14.029482 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-hx9dk\"" Oct 30 09:13:14.029692 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:14.029669 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 09:13:14.097552 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:14.097521 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ae2b5d9-beef-4d8b-9351-77be0eff9d0d-metrics-certs\") pod \"maas-api-e2e-ait-2ffc2a58-6cf8684647-d28fv\" (UID: \"4ae2b5d9-beef-4d8b-9351-77be0eff9d0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ffc2a58-6cf8684647-d28fv" Oct 30 09:13:14.097709 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:14.097566 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ae2b5d9-beef-4d8b-9351-77be0eff9d0d-maas-api-tls\") pod \"maas-api-e2e-ait-2ffc2a58-6cf8684647-d28fv\" (UID: \"4ae2b5d9-beef-4d8b-9351-77be0eff9d0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ffc2a58-6cf8684647-d28fv" Oct 30 09:13:14.097709 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:14.097598 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ae2b5d9-beef-4d8b-9351-77be0eff9d0d-openshift-service-ca\") pod \"maas-api-e2e-ait-2ffc2a58-6cf8684647-d28fv\" (UID: \"4ae2b5d9-beef-4d8b-9351-77be0eff9d0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ffc2a58-6cf8684647-d28fv" Oct 30 09:13:14.097709 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:14.097631 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5w54\" (UniqueName: \"kubernetes.io/projected/4ae2b5d9-beef-4d8b-9351-77be0eff9d0d-kube-api-access-j5w54\") pod \"maas-api-e2e-ait-2ffc2a58-6cf8684647-d28fv\" (UID: \"4ae2b5d9-beef-4d8b-9351-77be0eff9d0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ffc2a58-6cf8684647-d28fv" Oct 30 09:13:14.097857 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:13:14.097741 4562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-2ffc2a58-serving-cert: secret "maas-api-e2e-ait-2ffc2a58-serving-cert" not found Oct 30 09:13:14.097857 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:13:14.097825 4562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ae2b5d9-beef-4d8b-9351-77be0eff9d0d-maas-api-tls podName:4ae2b5d9-beef-4d8b-9351-77be0eff9d0d nodeName:}" failed. No retries permitted until 2026-10-30 09:13:14.597806511 +0000 UTC m=+1154.859367495 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4ae2b5d9-beef-4d8b-9351-77be0eff9d0d-maas-api-tls") pod "maas-api-e2e-ait-2ffc2a58-6cf8684647-d28fv" (UID: "4ae2b5d9-beef-4d8b-9351-77be0eff9d0d") : secret "maas-api-e2e-ait-2ffc2a58-serving-cert" not found Oct 30 09:13:14.098330 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:14.098309 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ae2b5d9-beef-4d8b-9351-77be0eff9d0d-openshift-service-ca\") pod \"maas-api-e2e-ait-2ffc2a58-6cf8684647-d28fv\" (UID: \"4ae2b5d9-beef-4d8b-9351-77be0eff9d0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ffc2a58-6cf8684647-d28fv" Oct 30 09:13:14.099931 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:14.099913 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ae2b5d9-beef-4d8b-9351-77be0eff9d0d-metrics-certs\") pod \"maas-api-e2e-ait-2ffc2a58-6cf8684647-d28fv\" (UID: \"4ae2b5d9-beef-4d8b-9351-77be0eff9d0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ffc2a58-6cf8684647-d28fv" Oct 30 09:13:14.119171 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:14.119143 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5w54\" (UniqueName: \"kubernetes.io/projected/4ae2b5d9-beef-4d8b-9351-77be0eff9d0d-kube-api-access-j5w54\") pod \"maas-api-e2e-ait-2ffc2a58-6cf8684647-d28fv\" (UID: \"4ae2b5d9-beef-4d8b-9351-77be0eff9d0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ffc2a58-6cf8684647-d28fv" Oct 30 09:13:14.604399 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:14.604345 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ae2b5d9-beef-4d8b-9351-77be0eff9d0d-maas-api-tls\") pod \"maas-api-e2e-ait-2ffc2a58-6cf8684647-d28fv\" (UID: \"4ae2b5d9-beef-4d8b-9351-77be0eff9d0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ffc2a58-6cf8684647-d28fv" Oct 30 09:13:14.606762 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:14.606736 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ae2b5d9-beef-4d8b-9351-77be0eff9d0d-maas-api-tls\") pod \"maas-api-e2e-ait-2ffc2a58-6cf8684647-d28fv\" (UID: \"4ae2b5d9-beef-4d8b-9351-77be0eff9d0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ffc2a58-6cf8684647-d28fv" Oct 30 09:13:14.837527 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:14.837492 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ffc2a58-6cf8684647-d28fv" Oct 30 09:13:15.140442 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:15.140409 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2ffc2a58-6cf8684647-d28fv"] Oct 30 09:13:15.142047 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:13:15.142017 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ae2b5d9_beef_4d8b_9351_77be0eff9d0d.slice/crio-a9908f70487885c41e104178f938e064701dd7c98684b2f22457d3b6d61587ca WatchSource:0}: Error finding container a9908f70487885c41e104178f938e064701dd7c98684b2f22457d3b6d61587ca: Status 404 returned error can't find the container with id a9908f70487885c41e104178f938e064701dd7c98684b2f22457d3b6d61587ca Oct 30 09:13:15.320106 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:15.320072 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ffc2a58-6cf8684647-d28fv" event={"ID":"4ae2b5d9-beef-4d8b-9351-77be0eff9d0d","Type":"ContainerStarted","Data":"a9908f70487885c41e104178f938e064701dd7c98684b2f22457d3b6d61587ca"} Oct 30 09:13:15.812874 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:15.812839 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2"] Oct 30 09:13:15.818197 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:15.817407 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2" Oct 30 09:13:15.835084 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:15.835051 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2"] Oct 30 09:13:15.845475 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:15.845452 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-6cf9ebb1-openshift-default-dockercfg-q8m86\"" Oct 30 09:13:15.898883 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:15.898855 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4c806f9c-786bc89897-4qg69"] Oct 30 09:13:15.902575 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:15.902553 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c806f9c-786bc89897-4qg69" Oct 30 09:13:15.920714 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:15.920687 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4c806f9c-786bc89897-4qg69"] Oct 30 09:13:15.923510 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:15.923478 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba1039fc-94dc-490f-88b8-e789a117ff15-istio-envoy\") pod \"e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2\" (UID: \"ba1039fc-94dc-490f-88b8-e789a117ff15\") " pod="openshift-ingress/e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2" Oct 30 09:13:15.923684 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:15.923665 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba1039fc-94dc-490f-88b8-e789a117ff15-credential-socket\") pod \"e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2\" (UID: \"ba1039fc-94dc-490f-88b8-e789a117ff15\") " pod="openshift-ingress/e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2" Oct 30 09:13:15.923775 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:15.923715 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba1039fc-94dc-490f-88b8-e789a117ff15-workload-socket\") pod \"e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2\" (UID: \"ba1039fc-94dc-490f-88b8-e789a117ff15\") " pod="openshift-ingress/e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2" Oct 30 09:13:15.923820 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:15.923776 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba1039fc-94dc-490f-88b8-e789a117ff15-istiod-ca-cert\") pod \"e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2\" (UID: \"ba1039fc-94dc-490f-88b8-e789a117ff15\") " pod="openshift-ingress/e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2" Oct 30 09:13:15.923820 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:15.923806 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba1039fc-94dc-490f-88b8-e789a117ff15-istio-data\") pod \"e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2\" (UID: \"ba1039fc-94dc-490f-88b8-e789a117ff15\") " pod="openshift-ingress/e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2" Oct 30 09:13:15.923917 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:15.923830 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba1039fc-94dc-490f-88b8-e789a117ff15-istio-podinfo\") pod \"e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2\" (UID: \"ba1039fc-94dc-490f-88b8-e789a117ff15\") " pod="openshift-ingress/e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2" Oct 30 09:13:15.923917 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:15.923862 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba1039fc-94dc-490f-88b8-e789a117ff15-workload-certs\") pod \"e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2\" (UID: \"ba1039fc-94dc-490f-88b8-e789a117ff15\") " pod="openshift-ingress/e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2" Oct 30 09:13:15.924115 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:15.923995 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba1039fc-94dc-490f-88b8-e789a117ff15-istio-token\") pod \"e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2\" (UID: \"ba1039fc-94dc-490f-88b8-e789a117ff15\") " pod="openshift-ingress/e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2" Oct 30 09:13:15.924115 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:15.924107 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbhz5\" (UniqueName: \"kubernetes.io/projected/ba1039fc-94dc-490f-88b8-e789a117ff15-kube-api-access-nbhz5\") pod \"e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2\" (UID: \"ba1039fc-94dc-490f-88b8-e789a117ff15\") " pod="openshift-ingress/e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2" Oct 30 09:13:15.936560 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:15.936542 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-4c806f9c-serving-cert\"" Oct 30 09:13:16.024819 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:16.024784 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c6cc6dd-ac06-4a91-86e7-f08f0372284b-maas-api-tls\") pod \"maas-api-e2e-ait-4c806f9c-786bc89897-4qg69\" (UID: \"9c6cc6dd-ac06-4a91-86e7-f08f0372284b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c806f9c-786bc89897-4qg69" Oct 30 09:13:16.024980 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:16.024874 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba1039fc-94dc-490f-88b8-e789a117ff15-istio-envoy\") pod \"e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2\" (UID: \"ba1039fc-94dc-490f-88b8-e789a117ff15\") " pod="openshift-ingress/e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2" Oct 30 09:13:16.024980 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:16.024921 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz7x4\" (UniqueName: \"kubernetes.io/projected/9c6cc6dd-ac06-4a91-86e7-f08f0372284b-kube-api-access-hz7x4\") pod \"maas-api-e2e-ait-4c806f9c-786bc89897-4qg69\" (UID: \"9c6cc6dd-ac06-4a91-86e7-f08f0372284b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c806f9c-786bc89897-4qg69" Oct 30 09:13:16.025051 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:16.024987 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c6cc6dd-ac06-4a91-86e7-f08f0372284b-metrics-certs\") pod \"maas-api-e2e-ait-4c806f9c-786bc89897-4qg69\" (UID: \"9c6cc6dd-ac06-4a91-86e7-f08f0372284b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c806f9c-786bc89897-4qg69" Oct 30 09:13:16.025051 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:16.025037 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba1039fc-94dc-490f-88b8-e789a117ff15-credential-socket\") pod \"e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2\" (UID: \"ba1039fc-94dc-490f-88b8-e789a117ff15\") " pod="openshift-ingress/e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2" Oct 30 09:13:16.025136 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:16.025114 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c6cc6dd-ac06-4a91-86e7-f08f0372284b-openshift-service-ca\") pod \"maas-api-e2e-ait-4c806f9c-786bc89897-4qg69\" (UID: \"9c6cc6dd-ac06-4a91-86e7-f08f0372284b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c806f9c-786bc89897-4qg69" Oct 30 09:13:16.025184 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:16.025170 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba1039fc-94dc-490f-88b8-e789a117ff15-workload-socket\") pod \"e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2\" (UID: \"ba1039fc-94dc-490f-88b8-e789a117ff15\") " pod="openshift-ingress/e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2" Oct 30 09:13:16.025265 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:16.025251 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba1039fc-94dc-490f-88b8-e789a117ff15-istiod-ca-cert\") pod \"e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2\" (UID: \"ba1039fc-94dc-490f-88b8-e789a117ff15\") " pod="openshift-ingress/e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2" Oct 30 09:13:16.025302 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:16.025280 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba1039fc-94dc-490f-88b8-e789a117ff15-istio-data\") pod \"e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2\" (UID: \"ba1039fc-94dc-490f-88b8-e789a117ff15\") " pod="openshift-ingress/e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2" Oct 30 09:13:16.025302 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:16.025296 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba1039fc-94dc-490f-88b8-e789a117ff15-istio-podinfo\") pod \"e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2\" (UID: \"ba1039fc-94dc-490f-88b8-e789a117ff15\") " pod="openshift-ingress/e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2" Oct 30 09:13:16.025454 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:16.025326 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba1039fc-94dc-490f-88b8-e789a117ff15-workload-certs\") pod \"e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2\" (UID: \"ba1039fc-94dc-490f-88b8-e789a117ff15\") " pod="openshift-ingress/e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2" Oct 30 09:13:16.025454 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:16.025366 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba1039fc-94dc-490f-88b8-e789a117ff15-istio-token\") pod \"e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2\" (UID: \"ba1039fc-94dc-490f-88b8-e789a117ff15\") " pod="openshift-ingress/e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2" Oct 30 09:13:16.025454 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:16.025421 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbhz5\" (UniqueName: \"kubernetes.io/projected/ba1039fc-94dc-490f-88b8-e789a117ff15-kube-api-access-nbhz5\") pod \"e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2\" (UID: \"ba1039fc-94dc-490f-88b8-e789a117ff15\") " pod="openshift-ingress/e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2" Oct 30 09:13:16.025608 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:16.025453 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba1039fc-94dc-490f-88b8-e789a117ff15-credential-socket\") pod \"e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2\" (UID: \"ba1039fc-94dc-490f-88b8-e789a117ff15\") " pod="openshift-ingress/e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2" Oct 30 09:13:16.025608 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:16.025482 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba1039fc-94dc-490f-88b8-e789a117ff15-workload-socket\") pod \"e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2\" (UID: \"ba1039fc-94dc-490f-88b8-e789a117ff15\") " pod="openshift-ingress/e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2" Oct 30 09:13:16.025951 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:16.025919 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba1039fc-94dc-490f-88b8-e789a117ff15-istio-data\") pod \"e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2\" (UID: \"ba1039fc-94dc-490f-88b8-e789a117ff15\") " pod="openshift-ingress/e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2" Oct 30 09:13:16.026055 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:16.025934 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba1039fc-94dc-490f-88b8-e789a117ff15-workload-certs\") pod \"e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2\" (UID: \"ba1039fc-94dc-490f-88b8-e789a117ff15\") " pod="openshift-ingress/e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2" Oct 30 09:13:16.026055 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:16.025991 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba1039fc-94dc-490f-88b8-e789a117ff15-istiod-ca-cert\") pod \"e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2\" (UID: \"ba1039fc-94dc-490f-88b8-e789a117ff15\") " pod="openshift-ingress/e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2" Oct 30 09:13:16.027324 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:16.027303 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba1039fc-94dc-490f-88b8-e789a117ff15-istio-envoy\") pod \"e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2\" (UID: \"ba1039fc-94dc-490f-88b8-e789a117ff15\") " pod="openshift-ingress/e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2" Oct 30 09:13:16.027887 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:16.027869 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba1039fc-94dc-490f-88b8-e789a117ff15-istio-podinfo\") pod \"e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2\" (UID: \"ba1039fc-94dc-490f-88b8-e789a117ff15\") " pod="openshift-ingress/e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2" Oct 30 09:13:16.044568 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:16.044542 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba1039fc-94dc-490f-88b8-e789a117ff15-istio-token\") pod \"e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2\" (UID: \"ba1039fc-94dc-490f-88b8-e789a117ff15\") " pod="openshift-ingress/e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2" Oct 30 09:13:16.044684 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:16.044667 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbhz5\" (UniqueName: \"kubernetes.io/projected/ba1039fc-94dc-490f-88b8-e789a117ff15-kube-api-access-nbhz5\") pod \"e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2\" (UID: \"ba1039fc-94dc-490f-88b8-e789a117ff15\") " pod="openshift-ingress/e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2" Oct 30 09:13:16.131210 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:16.131123 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c6cc6dd-ac06-4a91-86e7-f08f0372284b-maas-api-tls\") pod \"maas-api-e2e-ait-4c806f9c-786bc89897-4qg69\" (UID: \"9c6cc6dd-ac06-4a91-86e7-f08f0372284b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c806f9c-786bc89897-4qg69" Oct 30 09:13:16.137784 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:16.132451 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz7x4\" (UniqueName: \"kubernetes.io/projected/9c6cc6dd-ac06-4a91-86e7-f08f0372284b-kube-api-access-hz7x4\") pod \"maas-api-e2e-ait-4c806f9c-786bc89897-4qg69\" (UID: \"9c6cc6dd-ac06-4a91-86e7-f08f0372284b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c806f9c-786bc89897-4qg69" Oct 30 09:13:16.165440 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:16.131795 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2" Oct 30 09:13:16.165918 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:16.165438 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c6cc6dd-ac06-4a91-86e7-f08f0372284b-metrics-certs\") pod \"maas-api-e2e-ait-4c806f9c-786bc89897-4qg69\" (UID: \"9c6cc6dd-ac06-4a91-86e7-f08f0372284b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c806f9c-786bc89897-4qg69" Oct 30 09:13:16.165918 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:16.165549 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c6cc6dd-ac06-4a91-86e7-f08f0372284b-openshift-service-ca\") pod \"maas-api-e2e-ait-4c806f9c-786bc89897-4qg69\" (UID: \"9c6cc6dd-ac06-4a91-86e7-f08f0372284b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c806f9c-786bc89897-4qg69" Oct 30 09:13:16.166129 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:16.138867 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c6cc6dd-ac06-4a91-86e7-f08f0372284b-maas-api-tls\") pod \"maas-api-e2e-ait-4c806f9c-786bc89897-4qg69\" (UID: \"9c6cc6dd-ac06-4a91-86e7-f08f0372284b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c806f9c-786bc89897-4qg69" Oct 30 09:13:16.166504 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:16.166477 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c6cc6dd-ac06-4a91-86e7-f08f0372284b-openshift-service-ca\") pod \"maas-api-e2e-ait-4c806f9c-786bc89897-4qg69\" (UID: \"9c6cc6dd-ac06-4a91-86e7-f08f0372284b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c806f9c-786bc89897-4qg69" Oct 30 09:13:16.177946 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:16.177808 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz7x4\" (UniqueName: \"kubernetes.io/projected/9c6cc6dd-ac06-4a91-86e7-f08f0372284b-kube-api-access-hz7x4\") pod \"maas-api-e2e-ait-4c806f9c-786bc89897-4qg69\" (UID: \"9c6cc6dd-ac06-4a91-86e7-f08f0372284b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c806f9c-786bc89897-4qg69" Oct 30 09:13:16.190531 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:16.190013 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c6cc6dd-ac06-4a91-86e7-f08f0372284b-metrics-certs\") pod \"maas-api-e2e-ait-4c806f9c-786bc89897-4qg69\" (UID: \"9c6cc6dd-ac06-4a91-86e7-f08f0372284b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c806f9c-786bc89897-4qg69" Oct 30 09:13:16.215534 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:16.215507 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c806f9c-786bc89897-4qg69" Oct 30 09:13:16.328095 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:16.328062 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ffc2a58-6cf8684647-d28fv" event={"ID":"4ae2b5d9-beef-4d8b-9351-77be0eff9d0d","Type":"ContainerStarted","Data":"819d4fe516cbe7048afcf9c5cdbd1f16fb62e4b4bdbfd1dbad52c2463d31db3b"} Oct 30 09:13:16.328253 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:16.328139 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ffc2a58-6cf8684647-d28fv" Oct 30 09:13:16.369989 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:16.369936 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ffc2a58-6cf8684647-d28fv" podStartSLOduration=3.018372728 podStartE2EDuration="3.369922263s" totalImagesPullingTime="351.549535ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:13:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:13:16.368634318 +0000 UTC m=+1156.630195306" watchObservedRunningTime="2026-10-30 09:13:16.369922263 +0000 UTC m=+1156.631483251" Oct 30 09:13:16.498191 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:16.498157 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2"] Oct 30 09:13:16.502345 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:13:16.502316 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba1039fc_94dc_490f_88b8_e789a117ff15.slice/crio-640ebfc5df0e6290ee6d0e1e8141cca81ce8da8255139d21d058ce247b6cf272 WatchSource:0}: Error finding container 640ebfc5df0e6290ee6d0e1e8141cca81ce8da8255139d21d058ce247b6cf272: Status 404 returned error can't find the container with id 640ebfc5df0e6290ee6d0e1e8141cca81ce8da8255139d21d058ce247b6cf272 Oct 30 09:13:16.504967 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:16.504936 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:13:16.505059 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:16.504999 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:13:16.505059 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:16.505030 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:13:16.539646 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:16.539615 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4c806f9c-786bc89897-4qg69"] Oct 30 09:13:17.335788 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:17.335752 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c806f9c-786bc89897-4qg69" event={"ID":"9c6cc6dd-ac06-4a91-86e7-f08f0372284b","Type":"ContainerStarted","Data":"ef261eb1c948bb31c6785e0937a1fe34d860872757109d3a72642d71dcc8cb75"} Oct 30 09:13:17.336238 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:17.335813 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c806f9c-786bc89897-4qg69" event={"ID":"9c6cc6dd-ac06-4a91-86e7-f08f0372284b","Type":"ContainerStarted","Data":"ddfc3ba0daf0eaa7a998d5a27821dadb13581c378f647fc6098c5a1ea72a6688"} Oct 30 09:13:17.336238 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:17.335902 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c806f9c-786bc89897-4qg69" Oct 30 09:13:17.337555 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:17.337529 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2" event={"ID":"ba1039fc-94dc-490f-88b8-e789a117ff15","Type":"ContainerStarted","Data":"49d960c157e0b63e33dcf77c14df7633f485038836b75346ebecb80628b577c9"} Oct 30 09:13:17.337678 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:17.337576 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2" event={"ID":"ba1039fc-94dc-490f-88b8-e789a117ff15","Type":"ContainerStarted","Data":"640ebfc5df0e6290ee6d0e1e8141cca81ce8da8255139d21d058ce247b6cf272"} Oct 30 09:13:17.378137 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:17.378083 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c806f9c-786bc89897-4qg69" podStartSLOduration=2.030127151 podStartE2EDuration="2.378066588s" totalImagesPullingTime="347.939437ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:13:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:13:17.376184335 +0000 UTC m=+1157.637745351" watchObservedRunningTime="2026-10-30 09:13:17.378066588 +0000 UTC m=+1157.639627579" Oct 30 09:13:17.419174 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:17.419121 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2" podStartSLOduration=2.419110481 podStartE2EDuration="2.419110481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:13:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:13:17.417034837 +0000 UTC m=+1157.678595824" watchObservedRunningTime="2026-10-30 09:13:17.419110481 +0000 UTC m=+1157.680671469" Oct 30 09:13:18.132017 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:18.131982 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2" Oct 30 09:13:18.137406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:18.137364 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2" Oct 30 09:13:18.342629 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:18.342595 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2" Oct 30 09:13:18.343446 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:18.343429 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2" Oct 30 09:13:21.225922 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:21.222665 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-6cf9ebb1-7f45694dd-vd6c6"] Oct 30 09:13:21.231537 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:21.231147 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-6cf9ebb1-7f45694dd-vd6c6" Oct 30 09:13:21.262405 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:21.248414 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-6cf9ebb1-7f45694dd-vd6c6"] Oct 30 09:13:21.291629 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:21.291601 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-6cf9ebb1\"" Oct 30 09:13:21.291922 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:21.291875 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-6cf9ebb1-dockercfg-5sm87\"" Oct 30 09:13:21.298272 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:21.298253 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6233a0dd-65487d7658-6t2d9"] Oct 30 09:13:21.301976 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:21.301956 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6233a0dd-65487d7658-6t2d9" Oct 30 09:13:21.319041 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:21.319016 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6233a0dd-65487d7658-6t2d9"] Oct 30 09:13:21.333089 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:21.333063 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8ed0455-6995-452e-b1b7-9c8d63ec640c-metrics-certs\") pod \"maas-api-e2e-migrate-6233a0dd-65487d7658-6t2d9\" (UID: \"f8ed0455-6995-452e-b1b7-9c8d63ec640c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6233a0dd-65487d7658-6t2d9" Oct 30 09:13:21.333205 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:21.333123 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8ed0455-6995-452e-b1b7-9c8d63ec640c-maas-api-tls\") pod \"maas-api-e2e-migrate-6233a0dd-65487d7658-6t2d9\" (UID: \"f8ed0455-6995-452e-b1b7-9c8d63ec640c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6233a0dd-65487d7658-6t2d9" Oct 30 09:13:21.333267 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:21.333202 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8ed0455-6995-452e-b1b7-9c8d63ec640c-openshift-service-ca\") pod \"maas-api-e2e-migrate-6233a0dd-65487d7658-6t2d9\" (UID: \"f8ed0455-6995-452e-b1b7-9c8d63ec640c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6233a0dd-65487d7658-6t2d9" Oct 30 09:13:21.333267 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:21.333243 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d47xw\" (UniqueName: \"kubernetes.io/projected/5cb20235-f9a6-49be-8f34-038d586cd2e6-kube-api-access-d47xw\") pod \"payload-processing-e2e-derive-6cf9ebb1-7f45694dd-vd6c6\" (UID: \"5cb20235-f9a6-49be-8f34-038d586cd2e6\") " pod="openshift-ingress/payload-processing-e2e-derive-6cf9ebb1-7f45694dd-vd6c6" Oct 30 09:13:21.333412 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:21.333305 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cb20235-f9a6-49be-8f34-038d586cd2e6-plugins-config-volume\") pod \"payload-processing-e2e-derive-6cf9ebb1-7f45694dd-vd6c6\" (UID: \"5cb20235-f9a6-49be-8f34-038d586cd2e6\") " pod="openshift-ingress/payload-processing-e2e-derive-6cf9ebb1-7f45694dd-vd6c6" Oct 30 09:13:21.333412 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:21.333333 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqmv4\" (UniqueName: \"kubernetes.io/projected/f8ed0455-6995-452e-b1b7-9c8d63ec640c-kube-api-access-fqmv4\") pod \"maas-api-e2e-migrate-6233a0dd-65487d7658-6t2d9\" (UID: \"f8ed0455-6995-452e-b1b7-9c8d63ec640c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6233a0dd-65487d7658-6t2d9" Oct 30 09:13:21.378689 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:21.378655 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-6233a0dd-serving-cert\"" Oct 30 09:13:21.434264 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:21.434233 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8ed0455-6995-452e-b1b7-9c8d63ec640c-openshift-service-ca\") pod \"maas-api-e2e-migrate-6233a0dd-65487d7658-6t2d9\" (UID: \"f8ed0455-6995-452e-b1b7-9c8d63ec640c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6233a0dd-65487d7658-6t2d9" Oct 30 09:13:21.434479 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:21.434270 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d47xw\" (UniqueName: \"kubernetes.io/projected/5cb20235-f9a6-49be-8f34-038d586cd2e6-kube-api-access-d47xw\") pod \"payload-processing-e2e-derive-6cf9ebb1-7f45694dd-vd6c6\" (UID: \"5cb20235-f9a6-49be-8f34-038d586cd2e6\") " pod="openshift-ingress/payload-processing-e2e-derive-6cf9ebb1-7f45694dd-vd6c6" Oct 30 09:13:21.434479 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:21.434390 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cb20235-f9a6-49be-8f34-038d586cd2e6-plugins-config-volume\") pod \"payload-processing-e2e-derive-6cf9ebb1-7f45694dd-vd6c6\" (UID: \"5cb20235-f9a6-49be-8f34-038d586cd2e6\") " pod="openshift-ingress/payload-processing-e2e-derive-6cf9ebb1-7f45694dd-vd6c6" Oct 30 09:13:21.434479 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:21.434414 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqmv4\" (UniqueName: \"kubernetes.io/projected/f8ed0455-6995-452e-b1b7-9c8d63ec640c-kube-api-access-fqmv4\") pod \"maas-api-e2e-migrate-6233a0dd-65487d7658-6t2d9\" (UID: \"f8ed0455-6995-452e-b1b7-9c8d63ec640c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6233a0dd-65487d7658-6t2d9" Oct 30 09:13:21.434662 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:21.434491 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8ed0455-6995-452e-b1b7-9c8d63ec640c-metrics-certs\") pod \"maas-api-e2e-migrate-6233a0dd-65487d7658-6t2d9\" (UID: \"f8ed0455-6995-452e-b1b7-9c8d63ec640c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6233a0dd-65487d7658-6t2d9" Oct 30 09:13:21.434662 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:21.434534 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8ed0455-6995-452e-b1b7-9c8d63ec640c-maas-api-tls\") pod \"maas-api-e2e-migrate-6233a0dd-65487d7658-6t2d9\" (UID: \"f8ed0455-6995-452e-b1b7-9c8d63ec640c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6233a0dd-65487d7658-6t2d9" Oct 30 09:13:21.434780 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:13:21.434675 4562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-6233a0dd-serving-cert: secret "maas-api-e2e-migrate-6233a0dd-serving-cert" not found Oct 30 09:13:21.434780 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:13:21.434746 4562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8ed0455-6995-452e-b1b7-9c8d63ec640c-maas-api-tls podName:f8ed0455-6995-452e-b1b7-9c8d63ec640c nodeName:}" failed. No retries permitted until 2026-10-30 09:13:21.934715072 +0000 UTC m=+1162.196276060 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f8ed0455-6995-452e-b1b7-9c8d63ec640c-maas-api-tls") pod "maas-api-e2e-migrate-6233a0dd-65487d7658-6t2d9" (UID: "f8ed0455-6995-452e-b1b7-9c8d63ec640c") : secret "maas-api-e2e-migrate-6233a0dd-serving-cert" not found Oct 30 09:13:21.435105 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:21.435080 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8ed0455-6995-452e-b1b7-9c8d63ec640c-openshift-service-ca\") pod \"maas-api-e2e-migrate-6233a0dd-65487d7658-6t2d9\" (UID: \"f8ed0455-6995-452e-b1b7-9c8d63ec640c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6233a0dd-65487d7658-6t2d9" Oct 30 09:13:21.435243 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:21.435225 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cb20235-f9a6-49be-8f34-038d586cd2e6-plugins-config-volume\") pod \"payload-processing-e2e-derive-6cf9ebb1-7f45694dd-vd6c6\" (UID: \"5cb20235-f9a6-49be-8f34-038d586cd2e6\") " pod="openshift-ingress/payload-processing-e2e-derive-6cf9ebb1-7f45694dd-vd6c6" Oct 30 09:13:21.437411 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:21.436878 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8ed0455-6995-452e-b1b7-9c8d63ec640c-metrics-certs\") pod \"maas-api-e2e-migrate-6233a0dd-65487d7658-6t2d9\" (UID: \"f8ed0455-6995-452e-b1b7-9c8d63ec640c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6233a0dd-65487d7658-6t2d9" Oct 30 09:13:21.466394 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:21.466341 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqmv4\" (UniqueName: \"kubernetes.io/projected/f8ed0455-6995-452e-b1b7-9c8d63ec640c-kube-api-access-fqmv4\") pod \"maas-api-e2e-migrate-6233a0dd-65487d7658-6t2d9\" (UID: \"f8ed0455-6995-452e-b1b7-9c8d63ec640c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6233a0dd-65487d7658-6t2d9" Oct 30 09:13:21.466595 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:21.466575 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d47xw\" (UniqueName: \"kubernetes.io/projected/5cb20235-f9a6-49be-8f34-038d586cd2e6-kube-api-access-d47xw\") pod \"payload-processing-e2e-derive-6cf9ebb1-7f45694dd-vd6c6\" (UID: \"5cb20235-f9a6-49be-8f34-038d586cd2e6\") " pod="openshift-ingress/payload-processing-e2e-derive-6cf9ebb1-7f45694dd-vd6c6" Oct 30 09:13:21.597487 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:21.597412 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-6cf9ebb1-7f45694dd-vd6c6" Oct 30 09:13:21.907874 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:13:21.907839 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cb20235_f9a6_49be_8f34_038d586cd2e6.slice/crio-87f17491ae268f4b2ac716d355c33139a31735cfbc4e6e90b197bb5acf22aa04 WatchSource:0}: Error finding container 87f17491ae268f4b2ac716d355c33139a31735cfbc4e6e90b197bb5acf22aa04: Status 404 returned error can't find the container with id 87f17491ae268f4b2ac716d355c33139a31735cfbc4e6e90b197bb5acf22aa04 Oct 30 09:13:21.908045 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:21.908022 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-6cf9ebb1-7f45694dd-vd6c6"] Oct 30 09:13:21.963407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:21.953962 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8ed0455-6995-452e-b1b7-9c8d63ec640c-maas-api-tls\") pod \"maas-api-e2e-migrate-6233a0dd-65487d7658-6t2d9\" (UID: \"f8ed0455-6995-452e-b1b7-9c8d63ec640c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6233a0dd-65487d7658-6t2d9" Oct 30 09:13:21.963407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:21.959997 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8ed0455-6995-452e-b1b7-9c8d63ec640c-maas-api-tls\") pod \"maas-api-e2e-migrate-6233a0dd-65487d7658-6t2d9\" (UID: \"f8ed0455-6995-452e-b1b7-9c8d63ec640c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6233a0dd-65487d7658-6t2d9" Oct 30 09:13:22.219058 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:22.219024 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6233a0dd-65487d7658-6t2d9" Oct 30 09:13:22.368267 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:22.368238 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ffc2a58-6cf8684647-d28fv" Oct 30 09:13:22.393525 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:22.393405 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-6cf9ebb1-7f45694dd-vd6c6" event={"ID":"5cb20235-f9a6-49be-8f34-038d586cd2e6","Type":"ContainerStarted","Data":"a9f2e8df46a5f709d2727e62bca4ad9461367bc01afbe0149d7fab11d944228f"} Oct 30 09:13:22.393525 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:22.393487 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-6cf9ebb1-7f45694dd-vd6c6" event={"ID":"5cb20235-f9a6-49be-8f34-038d586cd2e6","Type":"ContainerStarted","Data":"87f17491ae268f4b2ac716d355c33139a31735cfbc4e6e90b197bb5acf22aa04"} Oct 30 09:13:22.394114 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:22.394090 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-6cf9ebb1-7f45694dd-vd6c6" Oct 30 09:13:22.458346 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:22.458283 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-6cf9ebb1-7f45694dd-vd6c6" podStartSLOduration=1.4582613229999999 podStartE2EDuration="1.458261323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:13:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:13:22.456771003 +0000 UTC m=+1162.718332005" watchObservedRunningTime="2026-10-30 09:13:22.458261323 +0000 UTC m=+1162.719822313" Oct 30 09:13:22.550468 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:22.550126 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6233a0dd-65487d7658-6t2d9"] Oct 30 09:13:22.569123 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:13:22.569077 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8ed0455_6995_452e_b1b7_9c8d63ec640c.slice/crio-1730506031f5dd503ddf9a00de5f38d3c6025a5289434082db792c2db4408bd0 WatchSource:0}: Error finding container 1730506031f5dd503ddf9a00de5f38d3c6025a5289434082db792c2db4408bd0: Status 404 returned error can't find the container with id 1730506031f5dd503ddf9a00de5f38d3c6025a5289434082db792c2db4408bd0 Oct 30 09:13:23.364415 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:23.363864 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c806f9c-786bc89897-4qg69" Oct 30 09:13:23.399731 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:23.399697 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6233a0dd-65487d7658-6t2d9" event={"ID":"f8ed0455-6995-452e-b1b7-9c8d63ec640c","Type":"ContainerStarted","Data":"a659cf46abd54a63f3e8c0b092cba5144a41ed92d4c2953216caa59ab593da44"} Oct 30 09:13:23.400205 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:23.399753 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6233a0dd-65487d7658-6t2d9" event={"ID":"f8ed0455-6995-452e-b1b7-9c8d63ec640c","Type":"ContainerStarted","Data":"1730506031f5dd503ddf9a00de5f38d3c6025a5289434082db792c2db4408bd0"} Oct 30 09:13:23.400205 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:23.399796 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6233a0dd-65487d7658-6t2d9" Oct 30 09:13:23.473266 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:23.471544 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6233a0dd-65487d7658-6t2d9" podStartSLOduration=2.151757981 podStartE2EDuration="2.471528012s" totalImagesPullingTime="319.770031ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:13:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:13:23.447786044 +0000 UTC m=+1163.709347033" watchObservedRunningTime="2026-10-30 09:13:23.471528012 +0000 UTC m=+1163.733088999" Oct 30 09:13:28.344522 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:28.344483 4562 patch_prober.go:29] interesting pod/e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 09:13:28.345213 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:28.345178 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l" podUID="05198b8c-2250-4810-873c-55d1ef91077a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 09:13:28.596145 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:13:28.596067 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:13:29.412069 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:29.412038 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6233a0dd-65487d7658-6t2d9" Oct 30 09:13:30.435859 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:30.435807 4562 generic.go:356] "Generic (PLEG): container finished" podID="05198b8c-2250-4810-873c-55d1ef91077a" containerID="ab20d321378cfc63bd2f5389776c5b85efbc06a6503857c89f5e2034a18f8238" exitCode=0 Oct 30 09:13:30.436440 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:30.435879 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l" event={"ID":"05198b8c-2250-4810-873c-55d1ef91077a","Type":"ContainerDied","Data":"ab20d321378cfc63bd2f5389776c5b85efbc06a6503857c89f5e2034a18f8238"} Oct 30 09:13:30.436440 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:30.436283 4562 scope.go:122] "RemoveContainer" containerID="ab20d321378cfc63bd2f5389776c5b85efbc06a6503857c89f5e2034a18f8238" Oct 30 09:13:30.437680 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:30.437646 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:13:30.437817 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:30.437724 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:13:30.437817 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:30.437756 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:13:31.459405 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:31.450921 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l" event={"ID":"05198b8c-2250-4810-873c-55d1ef91077a","Type":"ContainerStarted","Data":"473dafb25cecd013cf06f48eac671576a570c83065587df50f3ff157919ca129"} Oct 30 09:13:32.206429 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:32.196792 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l" Oct 30 09:13:32.206429 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:32.196925 4562 patch_prober.go:29] interesting pod/e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 30 09:13:32.206429 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:32.196968 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l" podUID="05198b8c-2250-4810-873c-55d1ef91077a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 30 09:13:33.220406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:33.218840 4562 patch_prober.go:29] interesting pod/e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 30 09:13:33.220406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:33.218912 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l" podUID="05198b8c-2250-4810-873c-55d1ef91077a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 30 09:13:34.200409 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:34.200313 4562 patch_prober.go:29] interesting pod/e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 30 09:13:34.200409 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:34.200374 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l" podUID="05198b8c-2250-4810-873c-55d1ef91077a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 30 09:13:34.433937 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:34.433354 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-6cf9ebb1-7f45694dd-vd6c6" Oct 30 09:13:35.197198 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:35.196736 4562 patch_prober.go:29] interesting pod/e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 30 09:13:35.197198 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:35.196807 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l" podUID="05198b8c-2250-4810-873c-55d1ef91077a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 30 09:13:36.202406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:36.195887 4562 patch_prober.go:29] interesting pod/e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 30 09:13:36.202406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:36.195952 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l" podUID="05198b8c-2250-4810-873c-55d1ef91077a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 30 09:13:37.200330 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:37.200291 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l" Oct 30 09:13:37.200747 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:37.200726 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l" Oct 30 09:13:37.201574 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:37.201551 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l" Oct 30 09:13:41.299875 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:41.299842 4562 patch_prober.go:29] interesting pod/e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 30 09:13:41.300292 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:41.299896 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2" podUID="267dbedb-3d29-43ab-8619-2ef3ae10d6a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 30 09:13:41.332542 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:41.332505 4562 patch_prober.go:29] interesting pod/e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 09:13:41.332699 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:41.332576 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6" podUID="26fc1564-372f-491d-93aa-8daa11a251ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 09:13:41.704016 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:41.703989 4562 generic.go:356] "Generic (PLEG): container finished" podID="267dbedb-3d29-43ab-8619-2ef3ae10d6a7" containerID="4659424aca33e89e42602a460866f016785afc0b80c74c331b244ef159f719af" exitCode=0 Oct 30 09:13:41.704144 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:41.704062 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2" event={"ID":"267dbedb-3d29-43ab-8619-2ef3ae10d6a7","Type":"ContainerDied","Data":"4659424aca33e89e42602a460866f016785afc0b80c74c331b244ef159f719af"} Oct 30 09:13:41.704540 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:41.704520 4562 scope.go:122] "RemoveContainer" containerID="4659424aca33e89e42602a460866f016785afc0b80c74c331b244ef159f719af" Oct 30 09:13:41.705811 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:41.705794 4562 generic.go:356] "Generic (PLEG): container finished" podID="ba1039fc-94dc-490f-88b8-e789a117ff15" containerID="49d960c157e0b63e33dcf77c14df7633f485038836b75346ebecb80628b577c9" exitCode=0 Oct 30 09:13:41.705895 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:41.705871 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2" event={"ID":"ba1039fc-94dc-490f-88b8-e789a117ff15","Type":"ContainerDied","Data":"49d960c157e0b63e33dcf77c14df7633f485038836b75346ebecb80628b577c9"} Oct 30 09:13:41.706208 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:41.706191 4562 scope.go:122] "RemoveContainer" containerID="49d960c157e0b63e33dcf77c14df7633f485038836b75346ebecb80628b577c9" Oct 30 09:13:41.721608 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:41.721580 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:13:41.721700 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:41.721640 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:13:41.721748 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:41.721728 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:13:41.721808 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:41.721789 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:13:41.721862 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:41.721651 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:13:41.721915 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:41.721883 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:13:42.715757 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:42.714795 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2" event={"ID":"267dbedb-3d29-43ab-8619-2ef3ae10d6a7","Type":"ContainerStarted","Data":"ae830805e7f8e08946a3b63f477c766c710b81658c73a9df8eaafa593b781a29"} Oct 30 09:13:42.717267 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:42.717237 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2" event={"ID":"ba1039fc-94dc-490f-88b8-e789a117ff15","Type":"ContainerStarted","Data":"18e493083d8e0298695f999a5f3be2ac21b8953ece39fd1334ba49248d688618"} Oct 30 09:13:42.719106 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:42.719081 4562 generic.go:356] "Generic (PLEG): container finished" podID="26fc1564-372f-491d-93aa-8daa11a251ee" containerID="d1826dded4e2230f90b913cbdc18360b65fd3ce3ba7dda1f59e88d401a338369" exitCode=0 Oct 30 09:13:42.719239 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:42.719225 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6" event={"ID":"26fc1564-372f-491d-93aa-8daa11a251ee","Type":"ContainerDied","Data":"d1826dded4e2230f90b913cbdc18360b65fd3ce3ba7dda1f59e88d401a338369"} Oct 30 09:13:42.719600 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:42.719581 4562 scope.go:122] "RemoveContainer" containerID="d1826dded4e2230f90b913cbdc18360b65fd3ce3ba7dda1f59e88d401a338369" Oct 30 09:13:42.720839 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:42.720809 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:13:42.720934 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:42.720899 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:13:42.720990 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:42.720935 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:13:42.989864 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:42.989780 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2" Oct 30 09:13:42.996475 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:42.990196 4562 patch_prober.go:29] interesting pod/e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 30 09:13:42.996750 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:42.996715 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2" podUID="267dbedb-3d29-43ab-8619-2ef3ae10d6a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 30 09:13:43.132866 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:43.132833 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2" Oct 30 09:13:43.133116 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:43.133083 4562 patch_prober.go:29] interesting pod/e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 30 09:13:43.133245 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:43.133135 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2" podUID="ba1039fc-94dc-490f-88b8-e789a117ff15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 30 09:13:43.725640 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:43.725596 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6" event={"ID":"26fc1564-372f-491d-93aa-8daa11a251ee","Type":"ContainerStarted","Data":"dcf183019d200d3bdbb3a67a348b729b54c5c2749f3412874d64beb47d29c30d"} Oct 30 09:13:43.989908 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:43.989815 4562 patch_prober.go:29] interesting pod/e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 30 09:13:43.990179 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:43.990148 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2" podUID="267dbedb-3d29-43ab-8619-2ef3ae10d6a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 30 09:13:44.135406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:44.134270 4562 patch_prober.go:29] interesting pod/e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 30 09:13:44.135406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:44.134340 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2" podUID="ba1039fc-94dc-490f-88b8-e789a117ff15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 30 09:13:44.400879 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:44.400786 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6" Oct 30 09:13:44.401222 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:44.401194 4562 patch_prober.go:29] interesting pod/e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 30 09:13:44.401397 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:44.401355 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6" podUID="26fc1564-372f-491d-93aa-8daa11a251ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 30 09:13:45.001985 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:45.001948 4562 patch_prober.go:29] interesting pod/e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 30 09:13:45.002957 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:45.002922 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2" podUID="267dbedb-3d29-43ab-8619-2ef3ae10d6a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 30 09:13:45.133076 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:45.133039 4562 patch_prober.go:29] interesting pod/e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 30 09:13:45.133247 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:45.133088 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2" podUID="ba1039fc-94dc-490f-88b8-e789a117ff15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 30 09:13:45.393462 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:45.392797 4562 patch_prober.go:29] interesting pod/e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 30 09:13:45.393462 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:45.392859 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6" podUID="26fc1564-372f-491d-93aa-8daa11a251ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 30 09:13:45.996249 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:45.996211 4562 patch_prober.go:29] interesting pod/e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 30 09:13:45.996571 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:45.996537 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2" podUID="267dbedb-3d29-43ab-8619-2ef3ae10d6a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 30 09:13:46.183406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:46.144505 4562 patch_prober.go:29] interesting pod/e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 30 09:13:46.183406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:46.144586 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2" podUID="ba1039fc-94dc-490f-88b8-e789a117ff15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 30 09:13:46.219089 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:46.219048 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2ffc2a58-6cf8684647-d28fv"] Oct 30 09:13:46.219588 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:46.219533 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ffc2a58-6cf8684647-d28fv" podUID="4ae2b5d9-beef-4d8b-9351-77be0eff9d0d" containerName="maas-api" containerID="cri-o://819d4fe516cbe7048afcf9c5cdbd1f16fb62e4b4bdbfd1dbad52c2463d31db3b" gracePeriod=30 Oct 30 09:13:46.398594 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:46.398146 4562 patch_prober.go:29] interesting pod/e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 30 09:13:46.398594 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:46.398211 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6" podUID="26fc1564-372f-491d-93aa-8daa11a251ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 30 09:13:46.516828 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:46.516743 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4c806f9c-786bc89897-4qg69"] Oct 30 09:13:46.517291 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:46.517259 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c806f9c-786bc89897-4qg69" podUID="9c6cc6dd-ac06-4a91-86e7-f08f0372284b" containerName="maas-api" containerID="cri-o://ef261eb1c948bb31c6785e0937a1fe34d860872757109d3a72642d71dcc8cb75" gracePeriod=30 Oct 30 09:13:46.837831 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:46.835637 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c806f9c-786bc89897-4qg69" event={"ID":"9c6cc6dd-ac06-4a91-86e7-f08f0372284b","Type":"ContainerDied","Data":"ef261eb1c948bb31c6785e0937a1fe34d860872757109d3a72642d71dcc8cb75"} Oct 30 09:13:46.922418 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:46.835609 4562 generic.go:356] "Generic (PLEG): container finished" podID="9c6cc6dd-ac06-4a91-86e7-f08f0372284b" containerID="ef261eb1c948bb31c6785e0937a1fe34d860872757109d3a72642d71dcc8cb75" exitCode=0 Oct 30 09:13:47.011404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:47.008192 4562 patch_prober.go:29] interesting pod/e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 30 09:13:47.011404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:47.008268 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2" podUID="267dbedb-3d29-43ab-8619-2ef3ae10d6a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 30 09:13:47.166454 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:47.160690 4562 generic.go:356] "Generic (PLEG): container finished" podID="4ae2b5d9-beef-4d8b-9351-77be0eff9d0d" containerID="819d4fe516cbe7048afcf9c5cdbd1f16fb62e4b4bdbfd1dbad52c2463d31db3b" exitCode=0 Oct 30 09:13:47.166454 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:47.160787 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ffc2a58-6cf8684647-d28fv" event={"ID":"4ae2b5d9-beef-4d8b-9351-77be0eff9d0d","Type":"ContainerDied","Data":"819d4fe516cbe7048afcf9c5cdbd1f16fb62e4b4bdbfd1dbad52c2463d31db3b"} Oct 30 09:13:47.194405 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:47.191840 4562 patch_prober.go:29] interesting pod/e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 30 09:13:47.194405 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:47.191899 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2" podUID="ba1039fc-94dc-490f-88b8-e789a117ff15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 30 09:13:47.409404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:47.408459 4562 patch_prober.go:29] interesting pod/e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 30 09:13:47.409404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:47.408530 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6" podUID="26fc1564-372f-491d-93aa-8daa11a251ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 30 09:13:47.520411 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:47.520340 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ffc2a58-6cf8684647-d28fv" Oct 30 09:13:47.631459 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:47.624937 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ae2b5d9-beef-4d8b-9351-77be0eff9d0d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ae2b5d9-beef-4d8b-9351-77be0eff9d0d-maas-api-tls\") pod \"4ae2b5d9-beef-4d8b-9351-77be0eff9d0d\" (UID: \"4ae2b5d9-beef-4d8b-9351-77be0eff9d0d\") " Oct 30 09:13:47.631459 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:47.624995 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ae2b5d9-beef-4d8b-9351-77be0eff9d0d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ae2b5d9-beef-4d8b-9351-77be0eff9d0d-metrics-certs\") pod \"4ae2b5d9-beef-4d8b-9351-77be0eff9d0d\" (UID: \"4ae2b5d9-beef-4d8b-9351-77be0eff9d0d\") " Oct 30 09:13:47.631459 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:47.625040 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ae2b5d9-beef-4d8b-9351-77be0eff9d0d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ae2b5d9-beef-4d8b-9351-77be0eff9d0d-openshift-service-ca\") pod \"4ae2b5d9-beef-4d8b-9351-77be0eff9d0d\" (UID: \"4ae2b5d9-beef-4d8b-9351-77be0eff9d0d\") " Oct 30 09:13:47.631459 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:47.625214 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ae2b5d9-beef-4d8b-9351-77be0eff9d0d-kube-api-access-j5w54\" (UniqueName: \"kubernetes.io/projected/4ae2b5d9-beef-4d8b-9351-77be0eff9d0d-kube-api-access-j5w54\") pod \"4ae2b5d9-beef-4d8b-9351-77be0eff9d0d\" (UID: \"4ae2b5d9-beef-4d8b-9351-77be0eff9d0d\") " Oct 30 09:13:47.631459 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:47.629937 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ae2b5d9-beef-4d8b-9351-77be0eff9d0d-kube-api-access-j5w54" pod "4ae2b5d9-beef-4d8b-9351-77be0eff9d0d" (UID: "4ae2b5d9-beef-4d8b-9351-77be0eff9d0d"). InnerVolumeSpecName "kube-api-access-j5w54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:13:47.631459 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:47.630253 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ae2b5d9-beef-4d8b-9351-77be0eff9d0d-openshift-service-ca" pod "4ae2b5d9-beef-4d8b-9351-77be0eff9d0d" (UID: "4ae2b5d9-beef-4d8b-9351-77be0eff9d0d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:13:47.637375 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:47.637333 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ae2b5d9-beef-4d8b-9351-77be0eff9d0d-metrics-certs" pod "4ae2b5d9-beef-4d8b-9351-77be0eff9d0d" (UID: "4ae2b5d9-beef-4d8b-9351-77be0eff9d0d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:13:47.728447 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:47.728114 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5w54\" (UniqueName: \"kubernetes.io/projected/4ae2b5d9-beef-4d8b-9351-77be0eff9d0d-kube-api-access-j5w54\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:13:47.728447 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:47.728155 4562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ae2b5d9-beef-4d8b-9351-77be0eff9d0d-metrics-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:13:47.728447 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:47.728170 4562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ae2b5d9-beef-4d8b-9351-77be0eff9d0d-openshift-service-ca\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:13:47.738591 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:47.738532 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ae2b5d9-beef-4d8b-9351-77be0eff9d0d-maas-api-tls" pod "4ae2b5d9-beef-4d8b-9351-77be0eff9d0d" (UID: "4ae2b5d9-beef-4d8b-9351-77be0eff9d0d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:13:47.850415 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:47.846526 4562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ae2b5d9-beef-4d8b-9351-77be0eff9d0d-maas-api-tls\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:13:47.905348 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:47.905042 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c806f9c-786bc89897-4qg69" Oct 30 09:13:47.990107 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:47.989044 4562 patch_prober.go:29] interesting pod/e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 30 09:13:47.990107 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:47.989102 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2" podUID="267dbedb-3d29-43ab-8619-2ef3ae10d6a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 30 09:13:48.056333 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:48.056201 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c6cc6dd-ac06-4a91-86e7-f08f0372284b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c6cc6dd-ac06-4a91-86e7-f08f0372284b-metrics-certs\") pod \"9c6cc6dd-ac06-4a91-86e7-f08f0372284b\" (UID: \"9c6cc6dd-ac06-4a91-86e7-f08f0372284b\") " Oct 30 09:13:48.056630 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:48.056606 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c6cc6dd-ac06-4a91-86e7-f08f0372284b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c6cc6dd-ac06-4a91-86e7-f08f0372284b-maas-api-tls\") pod \"9c6cc6dd-ac06-4a91-86e7-f08f0372284b\" (UID: \"9c6cc6dd-ac06-4a91-86e7-f08f0372284b\") " Oct 30 09:13:48.056889 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:48.056869 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c6cc6dd-ac06-4a91-86e7-f08f0372284b-kube-api-access-hz7x4\" (UniqueName: \"kubernetes.io/projected/9c6cc6dd-ac06-4a91-86e7-f08f0372284b-kube-api-access-hz7x4\") pod \"9c6cc6dd-ac06-4a91-86e7-f08f0372284b\" (UID: \"9c6cc6dd-ac06-4a91-86e7-f08f0372284b\") " Oct 30 09:13:48.057033 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:48.057018 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c6cc6dd-ac06-4a91-86e7-f08f0372284b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c6cc6dd-ac06-4a91-86e7-f08f0372284b-openshift-service-ca\") pod \"9c6cc6dd-ac06-4a91-86e7-f08f0372284b\" (UID: \"9c6cc6dd-ac06-4a91-86e7-f08f0372284b\") " Oct 30 09:13:48.058340 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:48.058317 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c6cc6dd-ac06-4a91-86e7-f08f0372284b-openshift-service-ca" pod "9c6cc6dd-ac06-4a91-86e7-f08f0372284b" (UID: "9c6cc6dd-ac06-4a91-86e7-f08f0372284b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:13:48.076452 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:48.071721 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c6cc6dd-ac06-4a91-86e7-f08f0372284b-maas-api-tls" pod "9c6cc6dd-ac06-4a91-86e7-f08f0372284b" (UID: "9c6cc6dd-ac06-4a91-86e7-f08f0372284b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:13:48.108595 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:48.108556 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c6cc6dd-ac06-4a91-86e7-f08f0372284b-kube-api-access-hz7x4" pod "9c6cc6dd-ac06-4a91-86e7-f08f0372284b" (UID: "9c6cc6dd-ac06-4a91-86e7-f08f0372284b"). InnerVolumeSpecName "kube-api-access-hz7x4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:13:48.111950 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:48.111916 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c6cc6dd-ac06-4a91-86e7-f08f0372284b-metrics-certs" pod "9c6cc6dd-ac06-4a91-86e7-f08f0372284b" (UID: "9c6cc6dd-ac06-4a91-86e7-f08f0372284b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:13:48.161325 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:48.161285 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz7x4\" (UniqueName: \"kubernetes.io/projected/9c6cc6dd-ac06-4a91-86e7-f08f0372284b-kube-api-access-hz7x4\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:13:48.161601 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:48.161581 4562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c6cc6dd-ac06-4a91-86e7-f08f0372284b-openshift-service-ca\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:13:48.161708 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:48.161695 4562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c6cc6dd-ac06-4a91-86e7-f08f0372284b-metrics-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:13:48.161800 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:48.161790 4562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c6cc6dd-ac06-4a91-86e7-f08f0372284b-maas-api-tls\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:13:48.178412 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:48.178125 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ffc2a58-6cf8684647-d28fv" event={"ID":"4ae2b5d9-beef-4d8b-9351-77be0eff9d0d","Type":"ContainerDied","Data":"a9908f70487885c41e104178f938e064701dd7c98684b2f22457d3b6d61587ca"} Oct 30 09:13:48.178412 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:48.178181 4562 scope.go:122] "RemoveContainer" containerID="819d4fe516cbe7048afcf9c5cdbd1f16fb62e4b4bdbfd1dbad52c2463d31db3b" Oct 30 09:13:48.178412 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:48.178288 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ffc2a58-6cf8684647-d28fv" Oct 30 09:13:48.181997 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:48.181968 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c806f9c-786bc89897-4qg69" event={"ID":"9c6cc6dd-ac06-4a91-86e7-f08f0372284b","Type":"ContainerDied","Data":"ddfc3ba0daf0eaa7a998d5a27821dadb13581c378f647fc6098c5a1ea72a6688"} Oct 30 09:13:48.182124 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:48.182009 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c806f9c-786bc89897-4qg69" Oct 30 09:13:48.191894 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:48.191717 4562 scope.go:122] "RemoveContainer" containerID="ef261eb1c948bb31c6785e0937a1fe34d860872757109d3a72642d71dcc8cb75" Oct 30 09:13:48.231455 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:48.231425 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2ffc2a58-6cf8684647-d28fv"] Oct 30 09:13:48.233455 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:48.233430 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2ffc2a58-6cf8684647-d28fv"] Oct 30 09:13:48.256819 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:48.256729 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2" Oct 30 09:13:48.258327 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:48.258267 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2" Oct 30 09:13:48.271819 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:48.271778 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2" Oct 30 09:13:48.283286 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:48.283254 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4c806f9c-786bc89897-4qg69"] Oct 30 09:13:48.287375 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:48.287343 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4c806f9c-786bc89897-4qg69"] Oct 30 09:13:48.391785 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:48.391749 4562 patch_prober.go:29] interesting pod/e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 30 09:13:48.392008 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:48.391982 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6" podUID="26fc1564-372f-491d-93aa-8daa11a251ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 30 09:13:48.479139 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:48.478952 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l"] Oct 30 09:13:48.479296 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:48.479180 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l" podUID="05198b8c-2250-4810-873c-55d1ef91077a" containerName="istio-proxy" containerID="cri-o://473dafb25cecd013cf06f48eac671576a570c83065587df50f3ff157919ca129" gracePeriod=30 Oct 30 09:13:48.990333 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:48.990306 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2" Oct 30 09:13:48.990635 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:48.990611 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2" Oct 30 09:13:48.991411 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:48.991369 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2" Oct 30 09:13:49.410712 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:49.410625 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6" Oct 30 09:13:49.414811 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:49.412089 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6" Oct 30 09:13:49.414811 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:49.414542 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6" Oct 30 09:13:49.873112 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:49.873071 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ae2b5d9-beef-4d8b-9351-77be0eff9d0d" path="/var/lib/kubelet/pods/4ae2b5d9-beef-4d8b-9351-77be0eff9d0d/volumes" Oct 30 09:13:49.873563 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:49.873546 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c6cc6dd-ac06-4a91-86e7-f08f0372284b" path="/var/lib/kubelet/pods/9c6cc6dd-ac06-4a91-86e7-f08f0372284b/volumes" Oct 30 09:13:50.347998 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:50.347958 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ffc2a58-6cf8684647-d28fv" podUID="4ae2b5d9-beef-4d8b-9351-77be0eff9d0d" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.49:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:13:51.632370 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:51.632326 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6233a0dd-65487d7658-6t2d9"] Oct 30 09:13:51.632884 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:51.632774 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6233a0dd-65487d7658-6t2d9" podUID="f8ed0455-6995-452e-b1b7-9c8d63ec640c" containerName="maas-api" containerID="cri-o://a659cf46abd54a63f3e8c0b092cba5144a41ed92d4c2953216caa59ab593da44" gracePeriod=30 Oct 30 09:13:51.951988 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:51.951959 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6233a0dd-65487d7658-6t2d9" Oct 30 09:13:52.028964 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:52.028927 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8ed0455-6995-452e-b1b7-9c8d63ec640c-kube-api-access-fqmv4\" (UniqueName: \"kubernetes.io/projected/f8ed0455-6995-452e-b1b7-9c8d63ec640c-kube-api-access-fqmv4\") pod \"f8ed0455-6995-452e-b1b7-9c8d63ec640c\" (UID: \"f8ed0455-6995-452e-b1b7-9c8d63ec640c\") " Oct 30 09:13:52.029135 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:52.028997 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8ed0455-6995-452e-b1b7-9c8d63ec640c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8ed0455-6995-452e-b1b7-9c8d63ec640c-metrics-certs\") pod \"f8ed0455-6995-452e-b1b7-9c8d63ec640c\" (UID: \"f8ed0455-6995-452e-b1b7-9c8d63ec640c\") " Oct 30 09:13:52.029135 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:52.029101 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8ed0455-6995-452e-b1b7-9c8d63ec640c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8ed0455-6995-452e-b1b7-9c8d63ec640c-maas-api-tls\") pod \"f8ed0455-6995-452e-b1b7-9c8d63ec640c\" (UID: \"f8ed0455-6995-452e-b1b7-9c8d63ec640c\") " Oct 30 09:13:52.029304 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:52.029153 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8ed0455-6995-452e-b1b7-9c8d63ec640c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8ed0455-6995-452e-b1b7-9c8d63ec640c-openshift-service-ca\") pod \"f8ed0455-6995-452e-b1b7-9c8d63ec640c\" (UID: \"f8ed0455-6995-452e-b1b7-9c8d63ec640c\") " Oct 30 09:13:52.030016 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:52.029988 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8ed0455-6995-452e-b1b7-9c8d63ec640c-openshift-service-ca" pod "f8ed0455-6995-452e-b1b7-9c8d63ec640c" (UID: "f8ed0455-6995-452e-b1b7-9c8d63ec640c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:13:52.031941 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:52.031913 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8ed0455-6995-452e-b1b7-9c8d63ec640c-kube-api-access-fqmv4" pod "f8ed0455-6995-452e-b1b7-9c8d63ec640c" (UID: "f8ed0455-6995-452e-b1b7-9c8d63ec640c"). InnerVolumeSpecName "kube-api-access-fqmv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:13:52.032178 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:52.032143 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8ed0455-6995-452e-b1b7-9c8d63ec640c-maas-api-tls" pod "f8ed0455-6995-452e-b1b7-9c8d63ec640c" (UID: "f8ed0455-6995-452e-b1b7-9c8d63ec640c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:13:52.032486 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:52.032449 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8ed0455-6995-452e-b1b7-9c8d63ec640c-metrics-certs" pod "f8ed0455-6995-452e-b1b7-9c8d63ec640c" (UID: "f8ed0455-6995-452e-b1b7-9c8d63ec640c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:13:52.131212 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:52.131175 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqmv4\" (UniqueName: \"kubernetes.io/projected/f8ed0455-6995-452e-b1b7-9c8d63ec640c-kube-api-access-fqmv4\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:13:52.131212 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:52.131207 4562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8ed0455-6995-452e-b1b7-9c8d63ec640c-metrics-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:13:52.131212 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:52.131216 4562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8ed0455-6995-452e-b1b7-9c8d63ec640c-maas-api-tls\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:13:52.131466 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:52.131224 4562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8ed0455-6995-452e-b1b7-9c8d63ec640c-openshift-service-ca\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:13:52.203958 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:52.203917 4562 patch_prober.go:29] interesting pod/e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 09:13:52.204137 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:52.203996 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l" podUID="05198b8c-2250-4810-873c-55d1ef91077a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 09:13:52.220092 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:52.220063 4562 generic.go:356] "Generic (PLEG): container finished" podID="f8ed0455-6995-452e-b1b7-9c8d63ec640c" containerID="a659cf46abd54a63f3e8c0b092cba5144a41ed92d4c2953216caa59ab593da44" exitCode=0 Oct 30 09:13:52.220245 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:52.220109 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6233a0dd-65487d7658-6t2d9" Oct 30 09:13:52.220245 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:52.220112 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6233a0dd-65487d7658-6t2d9" event={"ID":"f8ed0455-6995-452e-b1b7-9c8d63ec640c","Type":"ContainerDied","Data":"a659cf46abd54a63f3e8c0b092cba5144a41ed92d4c2953216caa59ab593da44"} Oct 30 09:13:52.220245 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:52.220166 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6233a0dd-65487d7658-6t2d9" event={"ID":"f8ed0455-6995-452e-b1b7-9c8d63ec640c","Type":"ContainerDied","Data":"1730506031f5dd503ddf9a00de5f38d3c6025a5289434082db792c2db4408bd0"} Oct 30 09:13:52.220245 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:52.220198 4562 scope.go:122] "RemoveContainer" containerID="a659cf46abd54a63f3e8c0b092cba5144a41ed92d4c2953216caa59ab593da44" Oct 30 09:13:52.230156 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:52.230136 4562 scope.go:122] "RemoveContainer" containerID="a659cf46abd54a63f3e8c0b092cba5144a41ed92d4c2953216caa59ab593da44" Oct 30 09:13:52.230449 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:13:52.230417 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a659cf46abd54a63f3e8c0b092cba5144a41ed92d4c2953216caa59ab593da44\": container with ID starting with a659cf46abd54a63f3e8c0b092cba5144a41ed92d4c2953216caa59ab593da44 not found: ID does not exist" containerID="a659cf46abd54a63f3e8c0b092cba5144a41ed92d4c2953216caa59ab593da44" Oct 30 09:13:52.230544 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:52.230461 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a659cf46abd54a63f3e8c0b092cba5144a41ed92d4c2953216caa59ab593da44"} err="failed to get container status \"a659cf46abd54a63f3e8c0b092cba5144a41ed92d4c2953216caa59ab593da44\": rpc error: code = NotFound desc = could not find container \"a659cf46abd54a63f3e8c0b092cba5144a41ed92d4c2953216caa59ab593da44\": container with ID starting with a659cf46abd54a63f3e8c0b092cba5144a41ed92d4c2953216caa59ab593da44 not found: ID does not exist" Oct 30 09:13:52.272987 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:52.272963 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6233a0dd-65487d7658-6t2d9"] Oct 30 09:13:52.274941 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:52.274920 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6233a0dd-65487d7658-6t2d9"] Oct 30 09:13:52.490923 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:52.490831 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6"] Oct 30 09:13:52.491152 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:52.491115 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6" podUID="26fc1564-372f-491d-93aa-8daa11a251ee" containerName="istio-proxy" containerID="cri-o://dcf183019d200d3bdbb3a67a348b729b54c5c2749f3412874d64beb47d29c30d" gracePeriod=30 Oct 30 09:13:52.529958 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:52.529924 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2"] Oct 30 09:13:52.530279 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:52.530221 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2" podUID="267dbedb-3d29-43ab-8619-2ef3ae10d6a7" containerName="istio-proxy" containerID="cri-o://ae830805e7f8e08946a3b63f477c766c710b81658c73a9df8eaafa593b781a29" gracePeriod=30 Oct 30 09:13:53.742877 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:53.742821 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l" Oct 30 09:13:53.851832 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:53.851793 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05198b8c-2250-4810-873c-55d1ef91077a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05198b8c-2250-4810-873c-55d1ef91077a-credential-socket\") pod \"05198b8c-2250-4810-873c-55d1ef91077a\" (UID: \"05198b8c-2250-4810-873c-55d1ef91077a\") " Oct 30 09:13:53.852010 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:53.851852 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05198b8c-2250-4810-873c-55d1ef91077a-kube-api-access-xzzf7\" (UniqueName: \"kubernetes.io/projected/05198b8c-2250-4810-873c-55d1ef91077a-kube-api-access-xzzf7\") pod \"05198b8c-2250-4810-873c-55d1ef91077a\" (UID: \"05198b8c-2250-4810-873c-55d1ef91077a\") " Oct 30 09:13:53.852010 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:53.851887 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05198b8c-2250-4810-873c-55d1ef91077a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05198b8c-2250-4810-873c-55d1ef91077a-workload-socket\") pod \"05198b8c-2250-4810-873c-55d1ef91077a\" (UID: \"05198b8c-2250-4810-873c-55d1ef91077a\") " Oct 30 09:13:53.852010 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:53.851920 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05198b8c-2250-4810-873c-55d1ef91077a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05198b8c-2250-4810-873c-55d1ef91077a-workload-certs\") pod \"05198b8c-2250-4810-873c-55d1ef91077a\" (UID: \"05198b8c-2250-4810-873c-55d1ef91077a\") " Oct 30 09:13:53.852233 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:53.852122 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05198b8c-2250-4810-873c-55d1ef91077a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05198b8c-2250-4810-873c-55d1ef91077a-istio-podinfo\") pod \"05198b8c-2250-4810-873c-55d1ef91077a\" (UID: \"05198b8c-2250-4810-873c-55d1ef91077a\") " Oct 30 09:13:53.852233 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:53.852214 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05198b8c-2250-4810-873c-55d1ef91077a-istio-token\" (UniqueName: \"kubernetes.io/projected/05198b8c-2250-4810-873c-55d1ef91077a-istio-token\") pod \"05198b8c-2250-4810-873c-55d1ef91077a\" (UID: \"05198b8c-2250-4810-873c-55d1ef91077a\") " Oct 30 09:13:53.852339 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:53.852234 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05198b8c-2250-4810-873c-55d1ef91077a-credential-socket" pod "05198b8c-2250-4810-873c-55d1ef91077a" (UID: "05198b8c-2250-4810-873c-55d1ef91077a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:13:53.852339 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:53.852254 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05198b8c-2250-4810-873c-55d1ef91077a-workload-certs" pod "05198b8c-2250-4810-873c-55d1ef91077a" (UID: "05198b8c-2250-4810-873c-55d1ef91077a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:13:53.852339 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:53.852251 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05198b8c-2250-4810-873c-55d1ef91077a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05198b8c-2250-4810-873c-55d1ef91077a-istio-envoy\") pod \"05198b8c-2250-4810-873c-55d1ef91077a\" (UID: \"05198b8c-2250-4810-873c-55d1ef91077a\") " Oct 30 09:13:53.852530 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:53.852339 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05198b8c-2250-4810-873c-55d1ef91077a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05198b8c-2250-4810-873c-55d1ef91077a-istio-data\") pod \"05198b8c-2250-4810-873c-55d1ef91077a\" (UID: \"05198b8c-2250-4810-873c-55d1ef91077a\") " Oct 30 09:13:53.852530 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:53.852362 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05198b8c-2250-4810-873c-55d1ef91077a-workload-socket" pod "05198b8c-2250-4810-873c-55d1ef91077a" (UID: "05198b8c-2250-4810-873c-55d1ef91077a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:13:53.852530 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:53.852376 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05198b8c-2250-4810-873c-55d1ef91077a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05198b8c-2250-4810-873c-55d1ef91077a-istiod-ca-cert\") pod \"05198b8c-2250-4810-873c-55d1ef91077a\" (UID: \"05198b8c-2250-4810-873c-55d1ef91077a\") " Oct 30 09:13:53.852678 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:53.852584 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05198b8c-2250-4810-873c-55d1ef91077a-istio-data" pod "05198b8c-2250-4810-873c-55d1ef91077a" (UID: "05198b8c-2250-4810-873c-55d1ef91077a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:13:53.852946 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:53.852921 4562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05198b8c-2250-4810-873c-55d1ef91077a-istio-data\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:13:53.853038 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:53.852956 4562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05198b8c-2250-4810-873c-55d1ef91077a-credential-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:13:53.853038 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:53.852972 4562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05198b8c-2250-4810-873c-55d1ef91077a-workload-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:13:53.853038 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:53.852987 4562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05198b8c-2250-4810-873c-55d1ef91077a-workload-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:13:53.853038 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:53.852927 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05198b8c-2250-4810-873c-55d1ef91077a-istiod-ca-cert" pod "05198b8c-2250-4810-873c-55d1ef91077a" (UID: "05198b8c-2250-4810-873c-55d1ef91077a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:13:53.854738 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:53.854696 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05198b8c-2250-4810-873c-55d1ef91077a-istio-token" pod "05198b8c-2250-4810-873c-55d1ef91077a" (UID: "05198b8c-2250-4810-873c-55d1ef91077a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:13:53.854842 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:53.854798 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05198b8c-2250-4810-873c-55d1ef91077a-istio-podinfo" pod "05198b8c-2250-4810-873c-55d1ef91077a" (UID: "05198b8c-2250-4810-873c-55d1ef91077a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:13:53.854842 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:53.854829 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05198b8c-2250-4810-873c-55d1ef91077a-istio-envoy" pod "05198b8c-2250-4810-873c-55d1ef91077a" (UID: "05198b8c-2250-4810-873c-55d1ef91077a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:13:53.855460 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:53.855440 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05198b8c-2250-4810-873c-55d1ef91077a-kube-api-access-xzzf7" pod "05198b8c-2250-4810-873c-55d1ef91077a" (UID: "05198b8c-2250-4810-873c-55d1ef91077a"). InnerVolumeSpecName "kube-api-access-xzzf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:13:53.876563 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:53.876497 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8ed0455-6995-452e-b1b7-9c8d63ec640c" path="/var/lib/kubelet/pods/f8ed0455-6995-452e-b1b7-9c8d63ec640c/volumes" Oct 30 09:13:53.954548 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:53.954515 4562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05198b8c-2250-4810-873c-55d1ef91077a-istio-podinfo\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:13:53.954548 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:53.954547 4562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05198b8c-2250-4810-873c-55d1ef91077a-istio-token\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:13:53.954763 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:53.954562 4562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05198b8c-2250-4810-873c-55d1ef91077a-istio-envoy\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:13:53.954763 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:53.954575 4562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05198b8c-2250-4810-873c-55d1ef91077a-istiod-ca-cert\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:13:53.954763 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:53.954588 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzzf7\" (UniqueName: \"kubernetes.io/projected/05198b8c-2250-4810-873c-55d1ef91077a-kube-api-access-xzzf7\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:13:54.024978 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.024773 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw"] Oct 30 09:13:54.026779 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.025369 4562 cpu_manager.go:414] "containerMap: removing container" podUID="05198b8c-2250-4810-873c-55d1ef91077a" containerName="istio-proxy" Oct 30 09:13:54.026779 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.025406 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05198b8c-2250-4810-873c-55d1ef91077a" containerName="istio-proxy" Oct 30 09:13:54.026779 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.025437 4562 cpu_manager.go:414] "containerMap: removing container" podUID="9c6cc6dd-ac06-4a91-86e7-f08f0372284b" containerName="maas-api" Oct 30 09:13:54.026779 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.025445 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c6cc6dd-ac06-4a91-86e7-f08f0372284b" containerName="maas-api" Oct 30 09:13:54.026779 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.025462 4562 cpu_manager.go:414] "containerMap: removing container" podUID="4ae2b5d9-beef-4d8b-9351-77be0eff9d0d" containerName="maas-api" Oct 30 09:13:54.026779 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.025470 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ae2b5d9-beef-4d8b-9351-77be0eff9d0d" containerName="maas-api" Oct 30 09:13:54.026779 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.025487 4562 cpu_manager.go:414] "containerMap: removing container" podUID="f8ed0455-6995-452e-b1b7-9c8d63ec640c" containerName="maas-api" Oct 30 09:13:54.026779 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.025495 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8ed0455-6995-452e-b1b7-9c8d63ec640c" containerName="maas-api" Oct 30 09:13:54.026779 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.025596 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c6cc6dd-ac06-4a91-86e7-f08f0372284b" containerName="maas-api" Oct 30 09:13:54.026779 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.025610 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="05198b8c-2250-4810-873c-55d1ef91077a" containerName="istio-proxy" Oct 30 09:13:54.026779 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.025620 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ae2b5d9-beef-4d8b-9351-77be0eff9d0d" containerName="maas-api" Oct 30 09:13:54.026779 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.025630 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8ed0455-6995-452e-b1b7-9c8d63ec640c" containerName="maas-api" Oct 30 09:13:54.031817 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.031792 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw" Oct 30 09:13:54.054138 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.054073 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-176b6c-openshift-default-dockercfg-k67tz\"" Oct 30 09:13:54.056668 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.056143 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ba1f223-8fd5-499b-ac09-73fc50031a7d-workload-socket\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw\" (UID: \"0ba1f223-8fd5-499b-ac09-73fc50031a7d\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw" Oct 30 09:13:54.056668 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.056198 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ba1f223-8fd5-499b-ac09-73fc50031a7d-istio-podinfo\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw\" (UID: \"0ba1f223-8fd5-499b-ac09-73fc50031a7d\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw" Oct 30 09:13:54.056668 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.056249 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ba1f223-8fd5-499b-ac09-73fc50031a7d-istio-data\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw\" (UID: \"0ba1f223-8fd5-499b-ac09-73fc50031a7d\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw" Oct 30 09:13:54.056668 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.056279 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgvxm\" (UniqueName: \"kubernetes.io/projected/0ba1f223-8fd5-499b-ac09-73fc50031a7d-kube-api-access-qgvxm\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw\" (UID: \"0ba1f223-8fd5-499b-ac09-73fc50031a7d\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw" Oct 30 09:13:54.056668 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.056308 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ba1f223-8fd5-499b-ac09-73fc50031a7d-istio-token\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw\" (UID: \"0ba1f223-8fd5-499b-ac09-73fc50031a7d\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw" Oct 30 09:13:54.056668 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.056341 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ba1f223-8fd5-499b-ac09-73fc50031a7d-credential-socket\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw\" (UID: \"0ba1f223-8fd5-499b-ac09-73fc50031a7d\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw" Oct 30 09:13:54.056668 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.056363 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ba1f223-8fd5-499b-ac09-73fc50031a7d-istiod-ca-cert\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw\" (UID: \"0ba1f223-8fd5-499b-ac09-73fc50031a7d\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw" Oct 30 09:13:54.056668 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.056425 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ba1f223-8fd5-499b-ac09-73fc50031a7d-istio-envoy\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw\" (UID: \"0ba1f223-8fd5-499b-ac09-73fc50031a7d\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw" Oct 30 09:13:54.056668 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.056490 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ba1f223-8fd5-499b-ac09-73fc50031a7d-workload-certs\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw\" (UID: \"0ba1f223-8fd5-499b-ac09-73fc50031a7d\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw" Oct 30 09:13:54.065157 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.065132 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw"] Oct 30 09:13:54.157432 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.157359 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ba1f223-8fd5-499b-ac09-73fc50031a7d-istio-data\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw\" (UID: \"0ba1f223-8fd5-499b-ac09-73fc50031a7d\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw" Oct 30 09:13:54.157600 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.157454 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgvxm\" (UniqueName: \"kubernetes.io/projected/0ba1f223-8fd5-499b-ac09-73fc50031a7d-kube-api-access-qgvxm\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw\" (UID: \"0ba1f223-8fd5-499b-ac09-73fc50031a7d\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw" Oct 30 09:13:54.157600 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.157496 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ba1f223-8fd5-499b-ac09-73fc50031a7d-istio-token\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw\" (UID: \"0ba1f223-8fd5-499b-ac09-73fc50031a7d\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw" Oct 30 09:13:54.157600 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.157536 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ba1f223-8fd5-499b-ac09-73fc50031a7d-credential-socket\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw\" (UID: \"0ba1f223-8fd5-499b-ac09-73fc50031a7d\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw" Oct 30 09:13:54.157600 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.157560 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ba1f223-8fd5-499b-ac09-73fc50031a7d-istiod-ca-cert\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw\" (UID: \"0ba1f223-8fd5-499b-ac09-73fc50031a7d\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw" Oct 30 09:13:54.157828 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.157619 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ba1f223-8fd5-499b-ac09-73fc50031a7d-istio-envoy\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw\" (UID: \"0ba1f223-8fd5-499b-ac09-73fc50031a7d\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw" Oct 30 09:13:54.157828 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.157696 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ba1f223-8fd5-499b-ac09-73fc50031a7d-workload-certs\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw\" (UID: \"0ba1f223-8fd5-499b-ac09-73fc50031a7d\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw" Oct 30 09:13:54.157828 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.157755 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ba1f223-8fd5-499b-ac09-73fc50031a7d-workload-socket\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw\" (UID: \"0ba1f223-8fd5-499b-ac09-73fc50031a7d\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw" Oct 30 09:13:54.157828 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.157801 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ba1f223-8fd5-499b-ac09-73fc50031a7d-istio-podinfo\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw\" (UID: \"0ba1f223-8fd5-499b-ac09-73fc50031a7d\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw" Oct 30 09:13:54.157984 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.157799 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ba1f223-8fd5-499b-ac09-73fc50031a7d-istio-data\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw\" (UID: \"0ba1f223-8fd5-499b-ac09-73fc50031a7d\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw" Oct 30 09:13:54.158056 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.158035 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ba1f223-8fd5-499b-ac09-73fc50031a7d-workload-certs\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw\" (UID: \"0ba1f223-8fd5-499b-ac09-73fc50031a7d\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw" Oct 30 09:13:54.158268 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.158244 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ba1f223-8fd5-499b-ac09-73fc50031a7d-workload-socket\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw\" (UID: \"0ba1f223-8fd5-499b-ac09-73fc50031a7d\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw" Oct 30 09:13:54.158440 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.158413 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ba1f223-8fd5-499b-ac09-73fc50031a7d-credential-socket\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw\" (UID: \"0ba1f223-8fd5-499b-ac09-73fc50031a7d\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw" Oct 30 09:13:54.158500 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.158471 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ba1f223-8fd5-499b-ac09-73fc50031a7d-istiod-ca-cert\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw\" (UID: \"0ba1f223-8fd5-499b-ac09-73fc50031a7d\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw" Oct 30 09:13:54.160203 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.160186 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ba1f223-8fd5-499b-ac09-73fc50031a7d-istio-envoy\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw\" (UID: \"0ba1f223-8fd5-499b-ac09-73fc50031a7d\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw" Oct 30 09:13:54.160361 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.160345 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ba1f223-8fd5-499b-ac09-73fc50031a7d-istio-podinfo\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw\" (UID: \"0ba1f223-8fd5-499b-ac09-73fc50031a7d\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw" Oct 30 09:13:54.177124 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.177100 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgvxm\" (UniqueName: \"kubernetes.io/projected/0ba1f223-8fd5-499b-ac09-73fc50031a7d-kube-api-access-qgvxm\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw\" (UID: \"0ba1f223-8fd5-499b-ac09-73fc50031a7d\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw" Oct 30 09:13:54.177532 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.177515 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ba1f223-8fd5-499b-ac09-73fc50031a7d-istio-token\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw\" (UID: \"0ba1f223-8fd5-499b-ac09-73fc50031a7d\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw" Oct 30 09:13:54.242112 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.242075 4562 generic.go:356] "Generic (PLEG): container finished" podID="05198b8c-2250-4810-873c-55d1ef91077a" containerID="473dafb25cecd013cf06f48eac671576a570c83065587df50f3ff157919ca129" exitCode=0 Oct 30 09:13:54.242307 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.242141 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l" Oct 30 09:13:54.242307 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.242158 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l" event={"ID":"05198b8c-2250-4810-873c-55d1ef91077a","Type":"ContainerDied","Data":"473dafb25cecd013cf06f48eac671576a570c83065587df50f3ff157919ca129"} Oct 30 09:13:54.242307 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.242219 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l" event={"ID":"05198b8c-2250-4810-873c-55d1ef91077a","Type":"ContainerDied","Data":"a1a09c86c1c0e59947669a24de11f4478c8a2c46440a383299fe26e87ddfd448"} Oct 30 09:13:54.242307 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.242249 4562 scope.go:122] "RemoveContainer" containerID="473dafb25cecd013cf06f48eac671576a570c83065587df50f3ff157919ca129" Oct 30 09:13:54.254315 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.254251 4562 scope.go:122] "RemoveContainer" containerID="ab20d321378cfc63bd2f5389776c5b85efbc06a6503857c89f5e2034a18f8238" Oct 30 09:13:54.277250 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.277224 4562 scope.go:122] "RemoveContainer" containerID="473dafb25cecd013cf06f48eac671576a570c83065587df50f3ff157919ca129" Oct 30 09:13:54.277629 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:13:54.277603 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"473dafb25cecd013cf06f48eac671576a570c83065587df50f3ff157919ca129\": container with ID starting with 473dafb25cecd013cf06f48eac671576a570c83065587df50f3ff157919ca129 not found: ID does not exist" containerID="473dafb25cecd013cf06f48eac671576a570c83065587df50f3ff157919ca129" Oct 30 09:13:54.277752 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.277640 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"473dafb25cecd013cf06f48eac671576a570c83065587df50f3ff157919ca129"} err="failed to get container status \"473dafb25cecd013cf06f48eac671576a570c83065587df50f3ff157919ca129\": rpc error: code = NotFound desc = could not find container \"473dafb25cecd013cf06f48eac671576a570c83065587df50f3ff157919ca129\": container with ID starting with 473dafb25cecd013cf06f48eac671576a570c83065587df50f3ff157919ca129 not found: ID does not exist" Oct 30 09:13:54.277752 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.277662 4562 scope.go:122] "RemoveContainer" containerID="ab20d321378cfc63bd2f5389776c5b85efbc06a6503857c89f5e2034a18f8238" Oct 30 09:13:54.278020 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:13:54.277993 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab20d321378cfc63bd2f5389776c5b85efbc06a6503857c89f5e2034a18f8238\": container with ID starting with ab20d321378cfc63bd2f5389776c5b85efbc06a6503857c89f5e2034a18f8238 not found: ID does not exist" containerID="ab20d321378cfc63bd2f5389776c5b85efbc06a6503857c89f5e2034a18f8238" Oct 30 09:13:54.278093 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.278024 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab20d321378cfc63bd2f5389776c5b85efbc06a6503857c89f5e2034a18f8238"} err="failed to get container status \"ab20d321378cfc63bd2f5389776c5b85efbc06a6503857c89f5e2034a18f8238\": rpc error: code = NotFound desc = could not find container \"ab20d321378cfc63bd2f5389776c5b85efbc06a6503857c89f5e2034a18f8238\": container with ID starting with ab20d321378cfc63bd2f5389776c5b85efbc06a6503857c89f5e2034a18f8238 not found: ID does not exist" Oct 30 09:13:54.298966 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.298922 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l"] Oct 30 09:13:54.303884 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.303857 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-4c806f9c-openshift-default-54567cdd67-bp85l"] Oct 30 09:13:54.361360 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.361327 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw" Oct 30 09:13:54.610493 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.610422 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e3d2ff-7ffc858867-5wpzf"] Oct 30 09:13:54.611086 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.611069 4562 cpu_manager.go:414] "containerMap: removing container" podUID="05198b8c-2250-4810-873c-55d1ef91077a" containerName="istio-proxy" Oct 30 09:13:54.611140 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.611089 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05198b8c-2250-4810-873c-55d1ef91077a" containerName="istio-proxy" Oct 30 09:13:54.611208 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.611196 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="05198b8c-2250-4810-873c-55d1ef91077a" containerName="istio-proxy" Oct 30 09:13:54.614069 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.614042 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e3d2ff-7ffc858867-5wpzf" Oct 30 09:13:54.628290 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.628261 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e3d2ff-7ffc858867-5wpzf"] Oct 30 09:13:54.630025 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.630000 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-e3d2ff\"" Oct 30 09:13:54.630193 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.630178 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-e3d2ff-dockercfg-q7r2m\"" Oct 30 09:13:54.665472 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.663442 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2f602e8-2490-4eee-944b-f50c6ae747cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-e3d2ff-7ffc858867-5wpzf\" (UID: \"e2f602e8-2490-4eee-944b-f50c6ae747cc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e3d2ff-7ffc858867-5wpzf" Oct 30 09:13:54.665472 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.663565 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkl8k\" (UniqueName: \"kubernetes.io/projected/e2f602e8-2490-4eee-944b-f50c6ae747cc-kube-api-access-bkl8k\") pod \"payload-pre-processing-e2e-worker-w2-e3d2ff-7ffc858867-5wpzf\" (UID: \"e2f602e8-2490-4eee-944b-f50c6ae747cc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e3d2ff-7ffc858867-5wpzf" Oct 30 09:13:54.686404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.686059 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw"] Oct 30 09:13:54.690574 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.690536 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e3d2ff-cbb67b84b-j59gb"] Oct 30 09:13:54.693844 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.693693 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-e3d2ff-cbb67b84b-j59gb" Oct 30 09:13:54.703622 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.701880 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:13:54.703622 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.701947 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:13:54.703622 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.702073 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:13:54.738223 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.738197 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e3d2ff-cbb67b84b-j59gb"] Oct 30 09:13:54.765163 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.765137 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bab9e9b-9d5e-4dde-81e2-cd274b635f8e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-e3d2ff-cbb67b84b-j59gb\" (UID: \"0bab9e9b-9d5e-4dde-81e2-cd274b635f8e\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e3d2ff-cbb67b84b-j59gb" Oct 30 09:13:54.765486 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.765176 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkl8k\" (UniqueName: \"kubernetes.io/projected/e2f602e8-2490-4eee-944b-f50c6ae747cc-kube-api-access-bkl8k\") pod \"payload-pre-processing-e2e-worker-w2-e3d2ff-7ffc858867-5wpzf\" (UID: \"e2f602e8-2490-4eee-944b-f50c6ae747cc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e3d2ff-7ffc858867-5wpzf" Oct 30 09:13:54.765486 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.765205 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dglsk\" (UniqueName: \"kubernetes.io/projected/0bab9e9b-9d5e-4dde-81e2-cd274b635f8e-kube-api-access-dglsk\") pod \"payload-processing-e2e-worker-w2-e3d2ff-cbb67b84b-j59gb\" (UID: \"0bab9e9b-9d5e-4dde-81e2-cd274b635f8e\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e3d2ff-cbb67b84b-j59gb" Oct 30 09:13:54.765486 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.765347 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2f602e8-2490-4eee-944b-f50c6ae747cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-e3d2ff-7ffc858867-5wpzf\" (UID: \"e2f602e8-2490-4eee-944b-f50c6ae747cc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e3d2ff-7ffc858867-5wpzf" Oct 30 09:13:54.765947 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.765930 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2f602e8-2490-4eee-944b-f50c6ae747cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-e3d2ff-7ffc858867-5wpzf\" (UID: \"e2f602e8-2490-4eee-944b-f50c6ae747cc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e3d2ff-7ffc858867-5wpzf" Oct 30 09:13:54.784886 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.784865 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkl8k\" (UniqueName: \"kubernetes.io/projected/e2f602e8-2490-4eee-944b-f50c6ae747cc-kube-api-access-bkl8k\") pod \"payload-pre-processing-e2e-worker-w2-e3d2ff-7ffc858867-5wpzf\" (UID: \"e2f602e8-2490-4eee-944b-f50c6ae747cc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e3d2ff-7ffc858867-5wpzf" Oct 30 09:13:54.866148 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.866052 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dglsk\" (UniqueName: \"kubernetes.io/projected/0bab9e9b-9d5e-4dde-81e2-cd274b635f8e-kube-api-access-dglsk\") pod \"payload-processing-e2e-worker-w2-e3d2ff-cbb67b84b-j59gb\" (UID: \"0bab9e9b-9d5e-4dde-81e2-cd274b635f8e\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e3d2ff-cbb67b84b-j59gb" Oct 30 09:13:54.866305 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.866192 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bab9e9b-9d5e-4dde-81e2-cd274b635f8e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-e3d2ff-cbb67b84b-j59gb\" (UID: \"0bab9e9b-9d5e-4dde-81e2-cd274b635f8e\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e3d2ff-cbb67b84b-j59gb" Oct 30 09:13:54.866891 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.866866 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bab9e9b-9d5e-4dde-81e2-cd274b635f8e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-e3d2ff-cbb67b84b-j59gb\" (UID: \"0bab9e9b-9d5e-4dde-81e2-cd274b635f8e\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e3d2ff-cbb67b84b-j59gb" Oct 30 09:13:54.887699 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.887672 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dglsk\" (UniqueName: \"kubernetes.io/projected/0bab9e9b-9d5e-4dde-81e2-cd274b635f8e-kube-api-access-dglsk\") pod \"payload-processing-e2e-worker-w2-e3d2ff-cbb67b84b-j59gb\" (UID: \"0bab9e9b-9d5e-4dde-81e2-cd274b635f8e\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e3d2ff-cbb67b84b-j59gb" Oct 30 09:13:54.896341 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.896314 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-6cf9ebb1-7f45694dd-vd6c6"] Oct 30 09:13:54.926833 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:54.926805 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e3d2ff-7ffc858867-5wpzf" Oct 30 09:13:55.038194 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:55.038155 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-e3d2ff-cbb67b84b-j59gb" Oct 30 09:13:55.224424 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:13:55.224395 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2f602e8_2490_4eee_944b_f50c6ae747cc.slice/crio-6d8c91bf30799fd0fb2d4ebc50e80ef1c058ef22c767a6478c144b7d67ea41d5 WatchSource:0}: Error finding container 6d8c91bf30799fd0fb2d4ebc50e80ef1c058ef22c767a6478c144b7d67ea41d5: Status 404 returned error can't find the container with id 6d8c91bf30799fd0fb2d4ebc50e80ef1c058ef22c767a6478c144b7d67ea41d5 Oct 30 09:13:55.224713 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:55.224690 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e3d2ff-7ffc858867-5wpzf"] Oct 30 09:13:55.249241 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:55.248666 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw" event={"ID":"0ba1f223-8fd5-499b-ac09-73fc50031a7d","Type":"ContainerStarted","Data":"ed8d5f5580de3dbeb252ed0bb25fd1f08967b60724caa0a6e6ef3f6e4c964500"} Oct 30 09:13:55.249241 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:55.248730 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw" event={"ID":"0ba1f223-8fd5-499b-ac09-73fc50031a7d","Type":"ContainerStarted","Data":"ab840124578573b5cf7b6be2b51050bcb655b64b3bca0286fb9846c519c113c3"} Oct 30 09:13:55.251092 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:55.251069 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e3d2ff-7ffc858867-5wpzf" event={"ID":"e2f602e8-2490-4eee-944b-f50c6ae747cc","Type":"ContainerStarted","Data":"6d8c91bf30799fd0fb2d4ebc50e80ef1c058ef22c767a6478c144b7d67ea41d5"} Oct 30 09:13:55.295655 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:55.295607 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw" podStartSLOduration=2.295592796 podStartE2EDuration="2.295592796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:13:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:13:55.293437423 +0000 UTC m=+1195.554998410" watchObservedRunningTime="2026-10-30 09:13:55.295592796 +0000 UTC m=+1195.557153783" Oct 30 09:13:55.351358 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:13:55.351332 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bab9e9b_9d5e_4dde_81e2_cd274b635f8e.slice/crio-df96ea221efe0afdb24d9e4c97b2bd1d3302c2156faf5203a4da2491ab2d7c08 WatchSource:0}: Error finding container df96ea221efe0afdb24d9e4c97b2bd1d3302c2156faf5203a4da2491ab2d7c08: Status 404 returned error can't find the container with id df96ea221efe0afdb24d9e4c97b2bd1d3302c2156faf5203a4da2491ab2d7c08 Oct 30 09:13:55.352183 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:55.352151 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e3d2ff-cbb67b84b-j59gb"] Oct 30 09:13:55.361440 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:55.361418 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw" Oct 30 09:13:55.366640 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:55.366620 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw" Oct 30 09:13:55.874186 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:55.874154 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05198b8c-2250-4810-873c-55d1ef91077a" path="/var/lib/kubelet/pods/05198b8c-2250-4810-873c-55d1ef91077a/volumes" Oct 30 09:13:55.947805 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:55.947774 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-e3d2ff-78f84d76-w6jp2"] Oct 30 09:13:55.950681 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:55.950663 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e3d2ff-78f84d76-w6jp2" Oct 30 09:13:55.967943 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:55.967902 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-e3d2ff-78f84d76-w6jp2"] Oct 30 09:13:55.978563 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:55.978523 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-e3d2ff-serving-cert\"" Oct 30 09:13:55.978697 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:55.978600 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 09:13:55.978697 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:55.978675 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-hx9dk\"" Oct 30 09:13:55.980117 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:55.980089 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a70bbb8b-7d34-488e-bd32-fa463616f2e7-metrics-certs\") pod \"maas-api-e2e-worker-w2-e3d2ff-78f84d76-w6jp2\" (UID: \"a70bbb8b-7d34-488e-bd32-fa463616f2e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e3d2ff-78f84d76-w6jp2" Oct 30 09:13:55.980250 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:55.980170 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a70bbb8b-7d34-488e-bd32-fa463616f2e7-maas-api-tls\") pod \"maas-api-e2e-worker-w2-e3d2ff-78f84d76-w6jp2\" (UID: \"a70bbb8b-7d34-488e-bd32-fa463616f2e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e3d2ff-78f84d76-w6jp2" Oct 30 09:13:55.980250 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:55.980224 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvh6v\" (UniqueName: \"kubernetes.io/projected/a70bbb8b-7d34-488e-bd32-fa463616f2e7-kube-api-access-lvh6v\") pod \"maas-api-e2e-worker-w2-e3d2ff-78f84d76-w6jp2\" (UID: \"a70bbb8b-7d34-488e-bd32-fa463616f2e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e3d2ff-78f84d76-w6jp2" Oct 30 09:13:55.980341 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:55.980288 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a70bbb8b-7d34-488e-bd32-fa463616f2e7-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-e3d2ff-78f84d76-w6jp2\" (UID: \"a70bbb8b-7d34-488e-bd32-fa463616f2e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e3d2ff-78f84d76-w6jp2" Oct 30 09:13:56.081375 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:56.081338 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a70bbb8b-7d34-488e-bd32-fa463616f2e7-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-e3d2ff-78f84d76-w6jp2\" (UID: \"a70bbb8b-7d34-488e-bd32-fa463616f2e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e3d2ff-78f84d76-w6jp2" Oct 30 09:13:56.081558 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:56.081504 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a70bbb8b-7d34-488e-bd32-fa463616f2e7-metrics-certs\") pod \"maas-api-e2e-worker-w2-e3d2ff-78f84d76-w6jp2\" (UID: \"a70bbb8b-7d34-488e-bd32-fa463616f2e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e3d2ff-78f84d76-w6jp2" Oct 30 09:13:56.081558 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:56.081550 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a70bbb8b-7d34-488e-bd32-fa463616f2e7-maas-api-tls\") pod \"maas-api-e2e-worker-w2-e3d2ff-78f84d76-w6jp2\" (UID: \"a70bbb8b-7d34-488e-bd32-fa463616f2e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e3d2ff-78f84d76-w6jp2" Oct 30 09:13:56.081713 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:56.081687 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvh6v\" (UniqueName: \"kubernetes.io/projected/a70bbb8b-7d34-488e-bd32-fa463616f2e7-kube-api-access-lvh6v\") pod \"maas-api-e2e-worker-w2-e3d2ff-78f84d76-w6jp2\" (UID: \"a70bbb8b-7d34-488e-bd32-fa463616f2e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e3d2ff-78f84d76-w6jp2" Oct 30 09:13:56.082127 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:56.082101 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a70bbb8b-7d34-488e-bd32-fa463616f2e7-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-e3d2ff-78f84d76-w6jp2\" (UID: \"a70bbb8b-7d34-488e-bd32-fa463616f2e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e3d2ff-78f84d76-w6jp2" Oct 30 09:13:56.084083 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:56.084051 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a70bbb8b-7d34-488e-bd32-fa463616f2e7-maas-api-tls\") pod \"maas-api-e2e-worker-w2-e3d2ff-78f84d76-w6jp2\" (UID: \"a70bbb8b-7d34-488e-bd32-fa463616f2e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e3d2ff-78f84d76-w6jp2" Oct 30 09:13:56.084197 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:56.084153 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a70bbb8b-7d34-488e-bd32-fa463616f2e7-metrics-certs\") pod \"maas-api-e2e-worker-w2-e3d2ff-78f84d76-w6jp2\" (UID: \"a70bbb8b-7d34-488e-bd32-fa463616f2e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e3d2ff-78f84d76-w6jp2" Oct 30 09:13:56.101605 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:56.101579 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvh6v\" (UniqueName: \"kubernetes.io/projected/a70bbb8b-7d34-488e-bd32-fa463616f2e7-kube-api-access-lvh6v\") pod \"maas-api-e2e-worker-w2-e3d2ff-78f84d76-w6jp2\" (UID: \"a70bbb8b-7d34-488e-bd32-fa463616f2e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e3d2ff-78f84d76-w6jp2" Oct 30 09:13:56.258476 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:56.258442 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e3d2ff-7ffc858867-5wpzf" event={"ID":"e2f602e8-2490-4eee-944b-f50c6ae747cc","Type":"ContainerStarted","Data":"1b2d8fbd5b7920e8fb63db6c48c5ccf9efa1e8504a6a6da159b1a2751db5ed9b"} Oct 30 09:13:56.258709 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:56.258590 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e3d2ff-7ffc858867-5wpzf" Oct 30 09:13:56.260191 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:56.260161 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-e3d2ff-cbb67b84b-j59gb" event={"ID":"0bab9e9b-9d5e-4dde-81e2-cd274b635f8e","Type":"ContainerStarted","Data":"2bbf69aa8d522dd6e2fb9873498c00f12a7eb7ade596713c152f77db67616e38"} Oct 30 09:13:56.260306 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:56.260207 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-e3d2ff-cbb67b84b-j59gb" event={"ID":"0bab9e9b-9d5e-4dde-81e2-cd274b635f8e","Type":"ContainerStarted","Data":"df96ea221efe0afdb24d9e4c97b2bd1d3302c2156faf5203a4da2491ab2d7c08"} Oct 30 09:13:56.260306 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:56.260287 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-e3d2ff-cbb67b84b-j59gb" Oct 30 09:13:56.260428 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:56.260324 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw" Oct 30 09:13:56.261490 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:56.261469 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw" Oct 30 09:13:56.263612 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:56.263594 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e3d2ff-78f84d76-w6jp2" Oct 30 09:13:56.299269 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:56.299213 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e3d2ff-7ffc858867-5wpzf" podStartSLOduration=2.299198734 podStartE2EDuration="2.299198734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:13:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:13:56.296893806 +0000 UTC m=+1196.558454794" watchObservedRunningTime="2026-10-30 09:13:56.299198734 +0000 UTC m=+1196.560759721" Oct 30 09:13:56.377694 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:56.377635 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-e3d2ff-cbb67b84b-j59gb" podStartSLOduration=2.377618531 podStartE2EDuration="2.377618531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:13:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:13:56.376606348 +0000 UTC m=+1196.638167353" watchObservedRunningTime="2026-10-30 09:13:56.377618531 +0000 UTC m=+1196.639179518" Oct 30 09:13:56.557470 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:56.557432 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-e3d2ff-78f84d76-w6jp2"] Oct 30 09:13:56.557682 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:13:56.557660 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda70bbb8b_7d34_488e_bd32_fa463616f2e7.slice/crio-3c1e95a0d0a354e9f853c992f3395149ab9219305545fd631a9e251a62a61706 WatchSource:0}: Error finding container 3c1e95a0d0a354e9f853c992f3395149ab9219305545fd631a9e251a62a61706: Status 404 returned error can't find the container with id 3c1e95a0d0a354e9f853c992f3395149ab9219305545fd631a9e251a62a61706 Oct 30 09:13:57.266497 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:57.266409 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e3d2ff-78f84d76-w6jp2" event={"ID":"a70bbb8b-7d34-488e-bd32-fa463616f2e7","Type":"ContainerStarted","Data":"a7d2ccfe44e7c50d45db7e518245b48ccb2a192e868810bc37454627b691224f"} Oct 30 09:13:57.266497 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:57.266463 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e3d2ff-78f84d76-w6jp2" event={"ID":"a70bbb8b-7d34-488e-bd32-fa463616f2e7","Type":"ContainerStarted","Data":"3c1e95a0d0a354e9f853c992f3395149ab9219305545fd631a9e251a62a61706"} Oct 30 09:13:57.267027 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:57.266720 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e3d2ff-78f84d76-w6jp2" Oct 30 09:13:57.307212 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:57.307165 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e3d2ff-78f84d76-w6jp2" podStartSLOduration=1.901380568 podStartE2EDuration="2.307149694s" totalImagesPullingTime="405.769126ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:13:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:13:57.305415772 +0000 UTC m=+1197.566976761" watchObservedRunningTime="2026-10-30 09:13:57.307149694 +0000 UTC m=+1197.568710684" Oct 30 09:13:57.787216 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:57.787192 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6" Oct 30 09:13:57.820052 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:57.820029 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2" Oct 30 09:13:57.885949 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:57.885920 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-2f69c8-5dbb84f778-2gp27"] Oct 30 09:13:57.886303 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:57.886291 4562 cpu_manager.go:414] "containerMap: removing container" podUID="267dbedb-3d29-43ab-8619-2ef3ae10d6a7" containerName="istio-proxy" Oct 30 09:13:57.886363 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:57.886304 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="267dbedb-3d29-43ab-8619-2ef3ae10d6a7" containerName="istio-proxy" Oct 30 09:13:57.886363 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:57.886324 4562 cpu_manager.go:414] "containerMap: removing container" podUID="26fc1564-372f-491d-93aa-8daa11a251ee" containerName="istio-proxy" Oct 30 09:13:57.886363 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:57.886330 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26fc1564-372f-491d-93aa-8daa11a251ee" containerName="istio-proxy" Oct 30 09:13:57.886363 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:57.886344 4562 cpu_manager.go:414] "containerMap: removing container" podUID="26fc1564-372f-491d-93aa-8daa11a251ee" containerName="istio-proxy" Oct 30 09:13:57.886363 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:57.886349 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26fc1564-372f-491d-93aa-8daa11a251ee" containerName="istio-proxy" Oct 30 09:13:57.886535 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:57.886427 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="26fc1564-372f-491d-93aa-8daa11a251ee" containerName="istio-proxy" Oct 30 09:13:57.886535 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:57.886437 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="267dbedb-3d29-43ab-8619-2ef3ae10d6a7" containerName="istio-proxy" Oct 30 09:13:57.886535 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:57.886443 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="267dbedb-3d29-43ab-8619-2ef3ae10d6a7" containerName="istio-proxy" Oct 30 09:13:57.888418 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:57.888402 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2f69c8-5dbb84f778-2gp27" Oct 30 09:13:57.913038 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:57.911685 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-2f69c8-serving-cert\"" Oct 30 09:13:57.916764 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:57.916740 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-2f69c8-5dbb84f778-2gp27"] Oct 30 09:13:57.923830 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:57.923794 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/267dbedb-3d29-43ab-8619-2ef3ae10d6a7-istio-token\" (UniqueName: \"kubernetes.io/projected/267dbedb-3d29-43ab-8619-2ef3ae10d6a7-istio-token\") pod \"267dbedb-3d29-43ab-8619-2ef3ae10d6a7\" (UID: \"267dbedb-3d29-43ab-8619-2ef3ae10d6a7\") " Oct 30 09:13:57.923937 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:57.923839 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/267dbedb-3d29-43ab-8619-2ef3ae10d6a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/267dbedb-3d29-43ab-8619-2ef3ae10d6a7-istio-envoy\") pod \"267dbedb-3d29-43ab-8619-2ef3ae10d6a7\" (UID: \"267dbedb-3d29-43ab-8619-2ef3ae10d6a7\") " Oct 30 09:13:57.924037 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:57.924013 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26fc1564-372f-491d-93aa-8daa11a251ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26fc1564-372f-491d-93aa-8daa11a251ee-workload-socket\") pod \"26fc1564-372f-491d-93aa-8daa11a251ee\" (UID: \"26fc1564-372f-491d-93aa-8daa11a251ee\") " Oct 30 09:13:57.924130 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:57.924061 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/267dbedb-3d29-43ab-8619-2ef3ae10d6a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/267dbedb-3d29-43ab-8619-2ef3ae10d6a7-workload-certs\") pod \"267dbedb-3d29-43ab-8619-2ef3ae10d6a7\" (UID: \"267dbedb-3d29-43ab-8619-2ef3ae10d6a7\") " Oct 30 09:13:57.924130 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:57.924102 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/267dbedb-3d29-43ab-8619-2ef3ae10d6a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/267dbedb-3d29-43ab-8619-2ef3ae10d6a7-istio-data\") pod \"267dbedb-3d29-43ab-8619-2ef3ae10d6a7\" (UID: \"267dbedb-3d29-43ab-8619-2ef3ae10d6a7\") " Oct 30 09:13:57.924246 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:57.924139 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/267dbedb-3d29-43ab-8619-2ef3ae10d6a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/267dbedb-3d29-43ab-8619-2ef3ae10d6a7-istiod-ca-cert\") pod \"267dbedb-3d29-43ab-8619-2ef3ae10d6a7\" (UID: \"267dbedb-3d29-43ab-8619-2ef3ae10d6a7\") " Oct 30 09:13:57.924246 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:57.924216 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26fc1564-372f-491d-93aa-8daa11a251ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26fc1564-372f-491d-93aa-8daa11a251ee-istiod-ca-cert\") pod \"26fc1564-372f-491d-93aa-8daa11a251ee\" (UID: \"26fc1564-372f-491d-93aa-8daa11a251ee\") " Oct 30 09:13:57.924350 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:57.924285 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26fc1564-372f-491d-93aa-8daa11a251ee-kube-api-access-8cshr\" (UniqueName: \"kubernetes.io/projected/26fc1564-372f-491d-93aa-8daa11a251ee-kube-api-access-8cshr\") pod \"26fc1564-372f-491d-93aa-8daa11a251ee\" (UID: \"26fc1564-372f-491d-93aa-8daa11a251ee\") " Oct 30 09:13:57.924350 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:57.924336 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26fc1564-372f-491d-93aa-8daa11a251ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26fc1564-372f-491d-93aa-8daa11a251ee-workload-certs\") pod \"26fc1564-372f-491d-93aa-8daa11a251ee\" (UID: \"26fc1564-372f-491d-93aa-8daa11a251ee\") " Oct 30 09:13:57.924708 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:57.924344 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26fc1564-372f-491d-93aa-8daa11a251ee-workload-socket" pod "26fc1564-372f-491d-93aa-8daa11a251ee" (UID: "26fc1564-372f-491d-93aa-8daa11a251ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:13:57.924708 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:57.924365 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/267dbedb-3d29-43ab-8619-2ef3ae10d6a7-kube-api-access-rkcl9\" (UniqueName: \"kubernetes.io/projected/267dbedb-3d29-43ab-8619-2ef3ae10d6a7-kube-api-access-rkcl9\") pod \"267dbedb-3d29-43ab-8619-2ef3ae10d6a7\" (UID: \"267dbedb-3d29-43ab-8619-2ef3ae10d6a7\") " Oct 30 09:13:57.924708 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:57.924366 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/267dbedb-3d29-43ab-8619-2ef3ae10d6a7-istio-data" pod "267dbedb-3d29-43ab-8619-2ef3ae10d6a7" (UID: "267dbedb-3d29-43ab-8619-2ef3ae10d6a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:13:57.924708 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:57.924408 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/267dbedb-3d29-43ab-8619-2ef3ae10d6a7-workload-certs" pod "267dbedb-3d29-43ab-8619-2ef3ae10d6a7" (UID: "267dbedb-3d29-43ab-8619-2ef3ae10d6a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:13:57.924708 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:57.924419 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/267dbedb-3d29-43ab-8619-2ef3ae10d6a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/267dbedb-3d29-43ab-8619-2ef3ae10d6a7-workload-socket\") pod \"267dbedb-3d29-43ab-8619-2ef3ae10d6a7\" (UID: \"267dbedb-3d29-43ab-8619-2ef3ae10d6a7\") " Oct 30 09:13:57.924708 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:57.924452 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26fc1564-372f-491d-93aa-8daa11a251ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26fc1564-372f-491d-93aa-8daa11a251ee-istio-data\") pod \"26fc1564-372f-491d-93aa-8daa11a251ee\" (UID: \"26fc1564-372f-491d-93aa-8daa11a251ee\") " Oct 30 09:13:57.924708 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:57.924485 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/267dbedb-3d29-43ab-8619-2ef3ae10d6a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/267dbedb-3d29-43ab-8619-2ef3ae10d6a7-istio-podinfo\") pod \"267dbedb-3d29-43ab-8619-2ef3ae10d6a7\" (UID: \"267dbedb-3d29-43ab-8619-2ef3ae10d6a7\") " Oct 30 09:13:57.924708 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:57.924514 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26fc1564-372f-491d-93aa-8daa11a251ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26fc1564-372f-491d-93aa-8daa11a251ee-credential-socket\") pod \"26fc1564-372f-491d-93aa-8daa11a251ee\" (UID: \"26fc1564-372f-491d-93aa-8daa11a251ee\") " Oct 30 09:13:57.924708 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:57.924588 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/267dbedb-3d29-43ab-8619-2ef3ae10d6a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/267dbedb-3d29-43ab-8619-2ef3ae10d6a7-credential-socket\") pod \"267dbedb-3d29-43ab-8619-2ef3ae10d6a7\" (UID: \"267dbedb-3d29-43ab-8619-2ef3ae10d6a7\") " Oct 30 09:13:57.924708 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:57.924653 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26fc1564-372f-491d-93aa-8daa11a251ee-istio-token\" (UniqueName: \"kubernetes.io/projected/26fc1564-372f-491d-93aa-8daa11a251ee-istio-token\") pod \"26fc1564-372f-491d-93aa-8daa11a251ee\" (UID: \"26fc1564-372f-491d-93aa-8daa11a251ee\") " Oct 30 09:13:57.924708 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:57.924685 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/267dbedb-3d29-43ab-8619-2ef3ae10d6a7-istiod-ca-cert" pod "267dbedb-3d29-43ab-8619-2ef3ae10d6a7" (UID: "267dbedb-3d29-43ab-8619-2ef3ae10d6a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:13:57.924708 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:57.924687 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/26fc1564-372f-491d-93aa-8daa11a251ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26fc1564-372f-491d-93aa-8daa11a251ee-istio-podinfo\") pod \"26fc1564-372f-491d-93aa-8daa11a251ee\" (UID: \"26fc1564-372f-491d-93aa-8daa11a251ee\") " Oct 30 09:13:57.925344 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:57.924742 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26fc1564-372f-491d-93aa-8daa11a251ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26fc1564-372f-491d-93aa-8daa11a251ee-istio-envoy\") pod \"26fc1564-372f-491d-93aa-8daa11a251ee\" (UID: \"26fc1564-372f-491d-93aa-8daa11a251ee\") " Oct 30 09:13:57.925344 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:57.925106 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26fc1564-372f-491d-93aa-8daa11a251ee-workload-certs" pod "26fc1564-372f-491d-93aa-8daa11a251ee" (UID: "26fc1564-372f-491d-93aa-8daa11a251ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:13:57.925344 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:57.925190 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26fc1564-372f-491d-93aa-8daa11a251ee-istiod-ca-cert" pod "26fc1564-372f-491d-93aa-8daa11a251ee" (UID: "26fc1564-372f-491d-93aa-8daa11a251ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:13:57.925344 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:57.925314 4562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/267dbedb-3d29-43ab-8619-2ef3ae10d6a7-istiod-ca-cert\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:13:57.925344 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:57.925331 4562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26fc1564-372f-491d-93aa-8daa11a251ee-istiod-ca-cert\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:13:57.925344 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:57.925344 4562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26fc1564-372f-491d-93aa-8daa11a251ee-workload-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:13:57.925650 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:57.925356 4562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26fc1564-372f-491d-93aa-8daa11a251ee-workload-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:13:57.925650 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:57.925371 4562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/267dbedb-3d29-43ab-8619-2ef3ae10d6a7-workload-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:13:57.925650 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:57.925402 4562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/267dbedb-3d29-43ab-8619-2ef3ae10d6a7-istio-data\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:13:57.925650 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:57.925616 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/267dbedb-3d29-43ab-8619-2ef3ae10d6a7-credential-socket" pod "267dbedb-3d29-43ab-8619-2ef3ae10d6a7" (UID: "267dbedb-3d29-43ab-8619-2ef3ae10d6a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:13:57.925949 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:57.925794 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26fc1564-372f-491d-93aa-8daa11a251ee-credential-socket" pod "26fc1564-372f-491d-93aa-8daa11a251ee" (UID: "26fc1564-372f-491d-93aa-8daa11a251ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:13:57.926016 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:57.925959 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/267dbedb-3d29-43ab-8619-2ef3ae10d6a7-workload-socket" pod "267dbedb-3d29-43ab-8619-2ef3ae10d6a7" (UID: "267dbedb-3d29-43ab-8619-2ef3ae10d6a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:13:57.927787 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:57.927757 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26fc1564-372f-491d-93aa-8daa11a251ee-kube-api-access-8cshr" pod "26fc1564-372f-491d-93aa-8daa11a251ee" (UID: "26fc1564-372f-491d-93aa-8daa11a251ee"). InnerVolumeSpecName "kube-api-access-8cshr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:13:57.928013 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:57.927994 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26fc1564-372f-491d-93aa-8daa11a251ee-istio-data" pod "26fc1564-372f-491d-93aa-8daa11a251ee" (UID: "26fc1564-372f-491d-93aa-8daa11a251ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:13:57.928105 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:57.928080 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/267dbedb-3d29-43ab-8619-2ef3ae10d6a7-istio-token" pod "267dbedb-3d29-43ab-8619-2ef3ae10d6a7" (UID: "267dbedb-3d29-43ab-8619-2ef3ae10d6a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:13:57.928776 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:57.928693 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26fc1564-372f-491d-93aa-8daa11a251ee-istio-token" pod "26fc1564-372f-491d-93aa-8daa11a251ee" (UID: "26fc1564-372f-491d-93aa-8daa11a251ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:13:57.928776 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:57.928695 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26fc1564-372f-491d-93aa-8daa11a251ee-istio-podinfo" pod "26fc1564-372f-491d-93aa-8daa11a251ee" (UID: "26fc1564-372f-491d-93aa-8daa11a251ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:13:57.928776 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:57.928748 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26fc1564-372f-491d-93aa-8daa11a251ee-istio-envoy" pod "26fc1564-372f-491d-93aa-8daa11a251ee" (UID: "26fc1564-372f-491d-93aa-8daa11a251ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:13:57.928954 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:57.928823 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/267dbedb-3d29-43ab-8619-2ef3ae10d6a7-istio-envoy" pod "267dbedb-3d29-43ab-8619-2ef3ae10d6a7" (UID: "267dbedb-3d29-43ab-8619-2ef3ae10d6a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:13:57.930439 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:57.930404 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/267dbedb-3d29-43ab-8619-2ef3ae10d6a7-istio-podinfo" pod "267dbedb-3d29-43ab-8619-2ef3ae10d6a7" (UID: "267dbedb-3d29-43ab-8619-2ef3ae10d6a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:13:57.930891 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:57.930613 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/267dbedb-3d29-43ab-8619-2ef3ae10d6a7-kube-api-access-rkcl9" pod "267dbedb-3d29-43ab-8619-2ef3ae10d6a7" (UID: "267dbedb-3d29-43ab-8619-2ef3ae10d6a7"). InnerVolumeSpecName "kube-api-access-rkcl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:13:58.025946 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:58.025917 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wxqz\" (UniqueName: \"kubernetes.io/projected/48d959d9-0791-48dd-84e9-78da9baaaea3-kube-api-access-4wxqz\") pod \"maas-api-e2e-worker-w0-2f69c8-5dbb84f778-2gp27\" (UID: \"48d959d9-0791-48dd-84e9-78da9baaaea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2f69c8-5dbb84f778-2gp27" Oct 30 09:13:58.026108 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:58.025954 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48d959d9-0791-48dd-84e9-78da9baaaea3-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-2f69c8-5dbb84f778-2gp27\" (UID: \"48d959d9-0791-48dd-84e9-78da9baaaea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2f69c8-5dbb84f778-2gp27" Oct 30 09:13:58.026108 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:58.025985 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48d959d9-0791-48dd-84e9-78da9baaaea3-maas-api-tls\") pod \"maas-api-e2e-worker-w0-2f69c8-5dbb84f778-2gp27\" (UID: \"48d959d9-0791-48dd-84e9-78da9baaaea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2f69c8-5dbb84f778-2gp27" Oct 30 09:13:58.026187 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:58.026138 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48d959d9-0791-48dd-84e9-78da9baaaea3-metrics-certs\") pod \"maas-api-e2e-worker-w0-2f69c8-5dbb84f778-2gp27\" (UID: \"48d959d9-0791-48dd-84e9-78da9baaaea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2f69c8-5dbb84f778-2gp27" Oct 30 09:13:58.026238 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:58.026192 4562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/267dbedb-3d29-43ab-8619-2ef3ae10d6a7-credential-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:13:58.026238 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:58.026202 4562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26fc1564-372f-491d-93aa-8daa11a251ee-istio-token\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:13:58.026238 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:58.026212 4562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26fc1564-372f-491d-93aa-8daa11a251ee-istio-podinfo\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:13:58.026238 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:58.026224 4562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26fc1564-372f-491d-93aa-8daa11a251ee-istio-envoy\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:13:58.026238 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:58.026233 4562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/267dbedb-3d29-43ab-8619-2ef3ae10d6a7-istio-token\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:13:58.026238 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:58.026240 4562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/267dbedb-3d29-43ab-8619-2ef3ae10d6a7-istio-envoy\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:13:58.026434 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:58.026249 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cshr\" (UniqueName: \"kubernetes.io/projected/26fc1564-372f-491d-93aa-8daa11a251ee-kube-api-access-8cshr\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:13:58.026434 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:58.026257 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkcl9\" (UniqueName: \"kubernetes.io/projected/267dbedb-3d29-43ab-8619-2ef3ae10d6a7-kube-api-access-rkcl9\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:13:58.026434 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:58.026266 4562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/267dbedb-3d29-43ab-8619-2ef3ae10d6a7-workload-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:13:58.026434 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:58.026298 4562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26fc1564-372f-491d-93aa-8daa11a251ee-istio-data\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:13:58.026434 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:58.026323 4562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/267dbedb-3d29-43ab-8619-2ef3ae10d6a7-istio-podinfo\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:13:58.026434 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:58.026334 4562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26fc1564-372f-491d-93aa-8daa11a251ee-credential-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:13:58.127094 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:58.126984 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48d959d9-0791-48dd-84e9-78da9baaaea3-metrics-certs\") pod \"maas-api-e2e-worker-w0-2f69c8-5dbb84f778-2gp27\" (UID: \"48d959d9-0791-48dd-84e9-78da9baaaea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2f69c8-5dbb84f778-2gp27" Oct 30 09:13:58.127255 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:58.127138 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wxqz\" (UniqueName: \"kubernetes.io/projected/48d959d9-0791-48dd-84e9-78da9baaaea3-kube-api-access-4wxqz\") pod \"maas-api-e2e-worker-w0-2f69c8-5dbb84f778-2gp27\" (UID: \"48d959d9-0791-48dd-84e9-78da9baaaea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2f69c8-5dbb84f778-2gp27" Oct 30 09:13:58.127255 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:58.127162 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48d959d9-0791-48dd-84e9-78da9baaaea3-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-2f69c8-5dbb84f778-2gp27\" (UID: \"48d959d9-0791-48dd-84e9-78da9baaaea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2f69c8-5dbb84f778-2gp27" Oct 30 09:13:58.127255 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:58.127193 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48d959d9-0791-48dd-84e9-78da9baaaea3-maas-api-tls\") pod \"maas-api-e2e-worker-w0-2f69c8-5dbb84f778-2gp27\" (UID: \"48d959d9-0791-48dd-84e9-78da9baaaea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2f69c8-5dbb84f778-2gp27" Oct 30 09:13:58.127859 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:58.127829 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48d959d9-0791-48dd-84e9-78da9baaaea3-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-2f69c8-5dbb84f778-2gp27\" (UID: \"48d959d9-0791-48dd-84e9-78da9baaaea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2f69c8-5dbb84f778-2gp27" Oct 30 09:13:58.129560 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:58.129538 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48d959d9-0791-48dd-84e9-78da9baaaea3-maas-api-tls\") pod \"maas-api-e2e-worker-w0-2f69c8-5dbb84f778-2gp27\" (UID: \"48d959d9-0791-48dd-84e9-78da9baaaea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2f69c8-5dbb84f778-2gp27" Oct 30 09:13:58.129630 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:58.129605 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48d959d9-0791-48dd-84e9-78da9baaaea3-metrics-certs\") pod \"maas-api-e2e-worker-w0-2f69c8-5dbb84f778-2gp27\" (UID: \"48d959d9-0791-48dd-84e9-78da9baaaea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2f69c8-5dbb84f778-2gp27" Oct 30 09:13:58.147065 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:58.147043 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wxqz\" (UniqueName: \"kubernetes.io/projected/48d959d9-0791-48dd-84e9-78da9baaaea3-kube-api-access-4wxqz\") pod \"maas-api-e2e-worker-w0-2f69c8-5dbb84f778-2gp27\" (UID: \"48d959d9-0791-48dd-84e9-78da9baaaea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2f69c8-5dbb84f778-2gp27" Oct 30 09:13:58.200819 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:58.200783 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2f69c8-5dbb84f778-2gp27" Oct 30 09:13:58.274286 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:58.274252 4562 generic.go:356] "Generic (PLEG): container finished" podID="267dbedb-3d29-43ab-8619-2ef3ae10d6a7" containerID="ae830805e7f8e08946a3b63f477c766c710b81658c73a9df8eaafa593b781a29" exitCode=0 Oct 30 09:13:58.274818 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:58.274333 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2" Oct 30 09:13:58.274818 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:58.274336 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2" event={"ID":"267dbedb-3d29-43ab-8619-2ef3ae10d6a7","Type":"ContainerDied","Data":"ae830805e7f8e08946a3b63f477c766c710b81658c73a9df8eaafa593b781a29"} Oct 30 09:13:58.274818 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:58.274423 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2" event={"ID":"267dbedb-3d29-43ab-8619-2ef3ae10d6a7","Type":"ContainerDied","Data":"30a341c867b15d05ca1eda311fbd61c037525f5694e7d15e6c23d54b9027ba1c"} Oct 30 09:13:58.274818 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:58.274444 4562 scope.go:122] "RemoveContainer" containerID="ae830805e7f8e08946a3b63f477c766c710b81658c73a9df8eaafa593b781a29" Oct 30 09:13:58.276714 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:58.276682 4562 generic.go:356] "Generic (PLEG): container finished" podID="26fc1564-372f-491d-93aa-8daa11a251ee" containerID="dcf183019d200d3bdbb3a67a348b729b54c5c2749f3412874d64beb47d29c30d" exitCode=0 Oct 30 09:13:58.277495 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:58.277476 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6" Oct 30 09:13:58.278211 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:58.278111 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6" event={"ID":"26fc1564-372f-491d-93aa-8daa11a251ee","Type":"ContainerDied","Data":"dcf183019d200d3bdbb3a67a348b729b54c5c2749f3412874d64beb47d29c30d"} Oct 30 09:13:58.278211 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:58.278160 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6" event={"ID":"26fc1564-372f-491d-93aa-8daa11a251ee","Type":"ContainerDied","Data":"128b6f2a89fc14c490b1de1316d21f65de30b93b40c6784e6a13c3893a745ab4"} Oct 30 09:13:58.288517 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:58.286496 4562 scope.go:122] "RemoveContainer" containerID="4659424aca33e89e42602a460866f016785afc0b80c74c331b244ef159f719af" Oct 30 09:13:58.303924 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:58.303901 4562 scope.go:122] "RemoveContainer" containerID="ae830805e7f8e08946a3b63f477c766c710b81658c73a9df8eaafa593b781a29" Oct 30 09:13:58.304457 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:13:58.304424 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae830805e7f8e08946a3b63f477c766c710b81658c73a9df8eaafa593b781a29\": container with ID starting with ae830805e7f8e08946a3b63f477c766c710b81658c73a9df8eaafa593b781a29 not found: ID does not exist" containerID="ae830805e7f8e08946a3b63f477c766c710b81658c73a9df8eaafa593b781a29" Oct 30 09:13:58.304551 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:58.304461 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae830805e7f8e08946a3b63f477c766c710b81658c73a9df8eaafa593b781a29"} err="failed to get container status \"ae830805e7f8e08946a3b63f477c766c710b81658c73a9df8eaafa593b781a29\": rpc error: code = NotFound desc = could not find container \"ae830805e7f8e08946a3b63f477c766c710b81658c73a9df8eaafa593b781a29\": container with ID starting with ae830805e7f8e08946a3b63f477c766c710b81658c73a9df8eaafa593b781a29 not found: ID does not exist" Oct 30 09:13:58.304551 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:58.304485 4562 scope.go:122] "RemoveContainer" containerID="4659424aca33e89e42602a460866f016785afc0b80c74c331b244ef159f719af" Oct 30 09:13:58.304921 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:13:58.304894 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4659424aca33e89e42602a460866f016785afc0b80c74c331b244ef159f719af\": container with ID starting with 4659424aca33e89e42602a460866f016785afc0b80c74c331b244ef159f719af not found: ID does not exist" containerID="4659424aca33e89e42602a460866f016785afc0b80c74c331b244ef159f719af" Oct 30 09:13:58.305019 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:58.304931 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4659424aca33e89e42602a460866f016785afc0b80c74c331b244ef159f719af"} err="failed to get container status \"4659424aca33e89e42602a460866f016785afc0b80c74c331b244ef159f719af\": rpc error: code = NotFound desc = could not find container \"4659424aca33e89e42602a460866f016785afc0b80c74c331b244ef159f719af\": container with ID starting with 4659424aca33e89e42602a460866f016785afc0b80c74c331b244ef159f719af not found: ID does not exist" Oct 30 09:13:58.305019 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:58.304952 4562 scope.go:122] "RemoveContainer" containerID="dcf183019d200d3bdbb3a67a348b729b54c5c2749f3412874d64beb47d29c30d" Oct 30 09:13:58.313642 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:58.313621 4562 scope.go:122] "RemoveContainer" containerID="d1826dded4e2230f90b913cbdc18360b65fd3ce3ba7dda1f59e88d401a338369" Oct 30 09:13:58.322606 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:58.322583 4562 scope.go:122] "RemoveContainer" containerID="dcf183019d200d3bdbb3a67a348b729b54c5c2749f3412874d64beb47d29c30d" Oct 30 09:13:58.322923 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:13:58.322900 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcf183019d200d3bdbb3a67a348b729b54c5c2749f3412874d64beb47d29c30d\": container with ID starting with dcf183019d200d3bdbb3a67a348b729b54c5c2749f3412874d64beb47d29c30d not found: ID does not exist" containerID="dcf183019d200d3bdbb3a67a348b729b54c5c2749f3412874d64beb47d29c30d" Oct 30 09:13:58.323011 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:58.322929 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcf183019d200d3bdbb3a67a348b729b54c5c2749f3412874d64beb47d29c30d"} err="failed to get container status \"dcf183019d200d3bdbb3a67a348b729b54c5c2749f3412874d64beb47d29c30d\": rpc error: code = NotFound desc = could not find container \"dcf183019d200d3bdbb3a67a348b729b54c5c2749f3412874d64beb47d29c30d\": container with ID starting with dcf183019d200d3bdbb3a67a348b729b54c5c2749f3412874d64beb47d29c30d not found: ID does not exist" Oct 30 09:13:58.323011 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:58.322946 4562 scope.go:122] "RemoveContainer" containerID="d1826dded4e2230f90b913cbdc18360b65fd3ce3ba7dda1f59e88d401a338369" Oct 30 09:13:58.323207 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:13:58.323187 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1826dded4e2230f90b913cbdc18360b65fd3ce3ba7dda1f59e88d401a338369\": container with ID starting with d1826dded4e2230f90b913cbdc18360b65fd3ce3ba7dda1f59e88d401a338369 not found: ID does not exist" containerID="d1826dded4e2230f90b913cbdc18360b65fd3ce3ba7dda1f59e88d401a338369" Oct 30 09:13:58.323250 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:58.323215 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1826dded4e2230f90b913cbdc18360b65fd3ce3ba7dda1f59e88d401a338369"} err="failed to get container status \"d1826dded4e2230f90b913cbdc18360b65fd3ce3ba7dda1f59e88d401a338369\": rpc error: code = NotFound desc = could not find container \"d1826dded4e2230f90b913cbdc18360b65fd3ce3ba7dda1f59e88d401a338369\": container with ID starting with d1826dded4e2230f90b913cbdc18360b65fd3ce3ba7dda1f59e88d401a338369 not found: ID does not exist" Oct 30 09:13:58.344505 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:58.344478 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2"] Oct 30 09:13:58.348042 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:58.348020 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-6233a0dd-gw-openshift-default-84fc9f8d9-ktxz2"] Oct 30 09:13:58.384283 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:58.384208 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6"] Oct 30 09:13:58.396593 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:58.396562 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-70374d4e-openshift-default-b455f5bd5-h7bn6"] Oct 30 09:13:58.527149 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:13:58.527121 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48d959d9_0791_48dd_84e9_78da9baaaea3.slice/crio-db9b6ded7fc4139b39d74185fd654f2b355605376c721d5e0479eed48b513409 WatchSource:0}: Error finding container db9b6ded7fc4139b39d74185fd654f2b355605376c721d5e0479eed48b513409: Status 404 returned error can't find the container with id db9b6ded7fc4139b39d74185fd654f2b355605376c721d5e0479eed48b513409 Oct 30 09:13:58.527926 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:58.527900 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-2f69c8-5dbb84f778-2gp27"] Oct 30 09:13:58.598165 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:13:58.598131 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:13:59.285731 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:59.285680 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2f69c8-5dbb84f778-2gp27" event={"ID":"48d959d9-0791-48dd-84e9-78da9baaaea3","Type":"ContainerStarted","Data":"9b7b2f1ec9845637e9ea3d56f608542e3a06e9569434daa701710bb2b9c15ad8"} Oct 30 09:13:59.286117 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:59.285752 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2f69c8-5dbb84f778-2gp27" event={"ID":"48d959d9-0791-48dd-84e9-78da9baaaea3","Type":"ContainerStarted","Data":"db9b6ded7fc4139b39d74185fd654f2b355605376c721d5e0479eed48b513409"} Oct 30 09:13:59.286117 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:59.285787 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2f69c8-5dbb84f778-2gp27" Oct 30 09:13:59.326085 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:59.326035 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2f69c8-5dbb84f778-2gp27" podStartSLOduration=1.973012605 podStartE2EDuration="2.326022704s" totalImagesPullingTime="353.010099ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:13:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:13:59.324808844 +0000 UTC m=+1199.586369845" watchObservedRunningTime="2026-10-30 09:13:59.326022704 +0000 UTC m=+1199.587583691" Oct 30 09:13:59.589247 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:59.589172 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3a61e1-689c7f9675-6b84c"] Oct 30 09:13:59.589680 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:59.589665 4562 cpu_manager.go:414] "containerMap: removing container" podUID="267dbedb-3d29-43ab-8619-2ef3ae10d6a7" containerName="istio-proxy" Oct 30 09:13:59.589743 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:59.589682 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="267dbedb-3d29-43ab-8619-2ef3ae10d6a7" containerName="istio-proxy" Oct 30 09:13:59.589780 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:59.589769 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="26fc1564-372f-491d-93aa-8daa11a251ee" containerName="istio-proxy" Oct 30 09:13:59.592700 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:59.592685 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3a61e1-689c7f9675-6b84c" Oct 30 09:13:59.609636 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:59.609613 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-3a61e1\"" Oct 30 09:13:59.609769 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:59.609655 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-3a61e1-dockercfg-lq9w2\"" Oct 30 09:13:59.611849 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:59.611827 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3a61e1-689c7f9675-6b84c"] Oct 30 09:13:59.652365 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:59.652337 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62f4c\" (UniqueName: \"kubernetes.io/projected/d1195ec8-c914-4d63-aca9-85e8b74b5d44-kube-api-access-62f4c\") pod \"payload-pre-processing-e2e-worker-w4-3a61e1-689c7f9675-6b84c\" (UID: \"d1195ec8-c914-4d63-aca9-85e8b74b5d44\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3a61e1-689c7f9675-6b84c" Oct 30 09:13:59.652519 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:59.652375 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1195ec8-c914-4d63-aca9-85e8b74b5d44-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-3a61e1-689c7f9675-6b84c\" (UID: \"d1195ec8-c914-4d63-aca9-85e8b74b5d44\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3a61e1-689c7f9675-6b84c" Oct 30 09:13:59.707173 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:59.707144 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-3a61e1-6489f9fdfb-lt57n"] Oct 30 09:13:59.710497 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:59.710481 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-3a61e1-6489f9fdfb-lt57n" Oct 30 09:13:59.726117 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:59.726093 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-3a61e1-6489f9fdfb-lt57n"] Oct 30 09:13:59.753921 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:59.753883 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62f4c\" (UniqueName: \"kubernetes.io/projected/d1195ec8-c914-4d63-aca9-85e8b74b5d44-kube-api-access-62f4c\") pod \"payload-pre-processing-e2e-worker-w4-3a61e1-689c7f9675-6b84c\" (UID: \"d1195ec8-c914-4d63-aca9-85e8b74b5d44\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3a61e1-689c7f9675-6b84c" Oct 30 09:13:59.754163 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:59.754142 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1195ec8-c914-4d63-aca9-85e8b74b5d44-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-3a61e1-689c7f9675-6b84c\" (UID: \"d1195ec8-c914-4d63-aca9-85e8b74b5d44\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3a61e1-689c7f9675-6b84c" Oct 30 09:13:59.755304 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:59.755155 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1195ec8-c914-4d63-aca9-85e8b74b5d44-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-3a61e1-689c7f9675-6b84c\" (UID: \"d1195ec8-c914-4d63-aca9-85e8b74b5d44\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3a61e1-689c7f9675-6b84c" Oct 30 09:13:59.781434 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:59.780115 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62f4c\" (UniqueName: \"kubernetes.io/projected/d1195ec8-c914-4d63-aca9-85e8b74b5d44-kube-api-access-62f4c\") pod \"payload-pre-processing-e2e-worker-w4-3a61e1-689c7f9675-6b84c\" (UID: \"d1195ec8-c914-4d63-aca9-85e8b74b5d44\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3a61e1-689c7f9675-6b84c" Oct 30 09:13:59.856440 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:59.856356 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30a13eb2-b7e5-42ff-b5e9-993a20d588aa-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-3a61e1-6489f9fdfb-lt57n\" (UID: \"30a13eb2-b7e5-42ff-b5e9-993a20d588aa\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-3a61e1-6489f9fdfb-lt57n" Oct 30 09:13:59.856610 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:59.856591 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpbcq\" (UniqueName: \"kubernetes.io/projected/30a13eb2-b7e5-42ff-b5e9-993a20d588aa-kube-api-access-tpbcq\") pod \"payload-processing-e2e-worker-w4-3a61e1-6489f9fdfb-lt57n\" (UID: \"30a13eb2-b7e5-42ff-b5e9-993a20d588aa\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-3a61e1-6489f9fdfb-lt57n" Oct 30 09:13:59.874083 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:59.874045 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="267dbedb-3d29-43ab-8619-2ef3ae10d6a7" path="/var/lib/kubelet/pods/267dbedb-3d29-43ab-8619-2ef3ae10d6a7/volumes" Oct 30 09:13:59.874646 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:59.874628 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26fc1564-372f-491d-93aa-8daa11a251ee" path="/var/lib/kubelet/pods/26fc1564-372f-491d-93aa-8daa11a251ee/volumes" Oct 30 09:13:59.897105 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:59.897073 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-6cf9ebb1-7f45694dd-vd6c6" podUID="5cb20235-f9a6-49be-8f34-038d586cd2e6" containerName="payload-processing" containerID="cri-o://a9f2e8df46a5f709d2727e62bca4ad9461367bc01afbe0149d7fab11d944228f" gracePeriod=25 Oct 30 09:13:59.919061 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:59.919038 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-3a61e1-dockercfg-lq9w2\"" Oct 30 09:13:59.924972 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:59.924940 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3a61e1-689c7f9675-6b84c" Oct 30 09:13:59.958227 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:59.957361 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpbcq\" (UniqueName: \"kubernetes.io/projected/30a13eb2-b7e5-42ff-b5e9-993a20d588aa-kube-api-access-tpbcq\") pod \"payload-processing-e2e-worker-w4-3a61e1-6489f9fdfb-lt57n\" (UID: \"30a13eb2-b7e5-42ff-b5e9-993a20d588aa\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-3a61e1-6489f9fdfb-lt57n" Oct 30 09:13:59.958227 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:59.958040 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30a13eb2-b7e5-42ff-b5e9-993a20d588aa-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-3a61e1-6489f9fdfb-lt57n\" (UID: \"30a13eb2-b7e5-42ff-b5e9-993a20d588aa\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-3a61e1-6489f9fdfb-lt57n" Oct 30 09:13:59.958934 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:59.958906 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30a13eb2-b7e5-42ff-b5e9-993a20d588aa-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-3a61e1-6489f9fdfb-lt57n\" (UID: \"30a13eb2-b7e5-42ff-b5e9-993a20d588aa\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-3a61e1-6489f9fdfb-lt57n" Oct 30 09:13:59.988516 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:59.988484 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6q7n_1f51db5a-5720-4dea-ad02-f336b35d43b1/ovn-acl-logging/0.log" Oct 30 09:13:59.995230 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:59.994887 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6q7n_1f51db5a-5720-4dea-ad02-f336b35d43b1/ovn-acl-logging/0.log" Oct 30 09:13:59.997960 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:13:59.997921 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpbcq\" (UniqueName: \"kubernetes.io/projected/30a13eb2-b7e5-42ff-b5e9-993a20d588aa-kube-api-access-tpbcq\") pod \"payload-processing-e2e-worker-w4-3a61e1-6489f9fdfb-lt57n\" (UID: \"30a13eb2-b7e5-42ff-b5e9-993a20d588aa\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-3a61e1-6489f9fdfb-lt57n" Oct 30 09:14:00.222943 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:00.222915 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-6cf9ebb1-7f45694dd-vd6c6" Oct 30 09:14:00.274866 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:00.274841 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3a61e1-689c7f9675-6b84c"] Oct 30 09:14:00.275001 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:14:00.274973 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1195ec8_c914_4d63_aca9_85e8b74b5d44.slice/crio-046161ad3843e1e9bbd19be909781e2cfd5257b97ef50eb88920d1a14f817d62 WatchSource:0}: Error finding container 046161ad3843e1e9bbd19be909781e2cfd5257b97ef50eb88920d1a14f817d62: Status 404 returned error can't find the container with id 046161ad3843e1e9bbd19be909781e2cfd5257b97ef50eb88920d1a14f817d62 Oct 30 09:14:00.294090 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:00.294054 4562 generic.go:356] "Generic (PLEG): container finished" podID="5cb20235-f9a6-49be-8f34-038d586cd2e6" containerID="a9f2e8df46a5f709d2727e62bca4ad9461367bc01afbe0149d7fab11d944228f" exitCode=0 Oct 30 09:14:00.294750 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:00.294181 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-6cf9ebb1-7f45694dd-vd6c6" Oct 30 09:14:00.294750 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:00.294212 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-6cf9ebb1-7f45694dd-vd6c6" event={"ID":"5cb20235-f9a6-49be-8f34-038d586cd2e6","Type":"ContainerDied","Data":"a9f2e8df46a5f709d2727e62bca4ad9461367bc01afbe0149d7fab11d944228f"} Oct 30 09:14:00.294750 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:00.294337 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-6cf9ebb1-7f45694dd-vd6c6" event={"ID":"5cb20235-f9a6-49be-8f34-038d586cd2e6","Type":"ContainerDied","Data":"87f17491ae268f4b2ac716d355c33139a31735cfbc4e6e90b197bb5acf22aa04"} Oct 30 09:14:00.294750 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:00.294459 4562 scope.go:122] "RemoveContainer" containerID="a9f2e8df46a5f709d2727e62bca4ad9461367bc01afbe0149d7fab11d944228f" Oct 30 09:14:00.298046 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:00.298023 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3a61e1-689c7f9675-6b84c" event={"ID":"d1195ec8-c914-4d63-aca9-85e8b74b5d44","Type":"ContainerStarted","Data":"046161ad3843e1e9bbd19be909781e2cfd5257b97ef50eb88920d1a14f817d62"} Oct 30 09:14:00.313918 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:00.313891 4562 scope.go:122] "RemoveContainer" containerID="a9f2e8df46a5f709d2727e62bca4ad9461367bc01afbe0149d7fab11d944228f" Oct 30 09:14:00.314279 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:14:00.314252 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9f2e8df46a5f709d2727e62bca4ad9461367bc01afbe0149d7fab11d944228f\": container with ID starting with a9f2e8df46a5f709d2727e62bca4ad9461367bc01afbe0149d7fab11d944228f not found: ID does not exist" containerID="a9f2e8df46a5f709d2727e62bca4ad9461367bc01afbe0149d7fab11d944228f" Oct 30 09:14:00.314413 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:00.314293 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9f2e8df46a5f709d2727e62bca4ad9461367bc01afbe0149d7fab11d944228f"} err="failed to get container status \"a9f2e8df46a5f709d2727e62bca4ad9461367bc01afbe0149d7fab11d944228f\": rpc error: code = NotFound desc = could not find container \"a9f2e8df46a5f709d2727e62bca4ad9461367bc01afbe0149d7fab11d944228f\": container with ID starting with a9f2e8df46a5f709d2727e62bca4ad9461367bc01afbe0149d7fab11d944228f not found: ID does not exist" Oct 30 09:14:00.323932 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:00.323912 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-3a61e1-6489f9fdfb-lt57n" Oct 30 09:14:00.366152 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:00.366119 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cb20235-f9a6-49be-8f34-038d586cd2e6-kube-api-access-d47xw\" (UniqueName: \"kubernetes.io/projected/5cb20235-f9a6-49be-8f34-038d586cd2e6-kube-api-access-d47xw\") pod \"5cb20235-f9a6-49be-8f34-038d586cd2e6\" (UID: \"5cb20235-f9a6-49be-8f34-038d586cd2e6\") " Oct 30 09:14:00.366334 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:00.366317 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cb20235-f9a6-49be-8f34-038d586cd2e6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cb20235-f9a6-49be-8f34-038d586cd2e6-plugins-config-volume\") pod \"5cb20235-f9a6-49be-8f34-038d586cd2e6\" (UID: \"5cb20235-f9a6-49be-8f34-038d586cd2e6\") " Oct 30 09:14:00.366759 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:00.366729 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cb20235-f9a6-49be-8f34-038d586cd2e6-plugins-config-volume" pod "5cb20235-f9a6-49be-8f34-038d586cd2e6" (UID: "5cb20235-f9a6-49be-8f34-038d586cd2e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:14:00.368366 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:00.368335 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cb20235-f9a6-49be-8f34-038d586cd2e6-kube-api-access-d47xw" pod "5cb20235-f9a6-49be-8f34-038d586cd2e6" (UID: "5cb20235-f9a6-49be-8f34-038d586cd2e6"). InnerVolumeSpecName "kube-api-access-d47xw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:14:00.467316 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:00.467283 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d47xw\" (UniqueName: \"kubernetes.io/projected/5cb20235-f9a6-49be-8f34-038d586cd2e6-kube-api-access-d47xw\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:14:00.467316 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:00.467318 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cb20235-f9a6-49be-8f34-038d586cd2e6-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:14:00.515401 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:00.515002 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2"] Oct 30 09:14:00.515401 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:00.515257 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2" podUID="ba1039fc-94dc-490f-88b8-e789a117ff15" containerName="istio-proxy" containerID="cri-o://18e493083d8e0298695f999a5f3be2ac21b8953ece39fd1334ba49248d688618" gracePeriod=30 Oct 30 09:14:00.654903 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:00.654526 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-3a61e1-6489f9fdfb-lt57n"] Oct 30 09:14:00.660140 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:00.660106 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-6cf9ebb1-7f45694dd-vd6c6"] Oct 30 09:14:00.661696 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:00.661675 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-6cf9ebb1-7f45694dd-vd6c6"] Oct 30 09:14:01.305545 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:01.305499 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-3a61e1-6489f9fdfb-lt57n" event={"ID":"30a13eb2-b7e5-42ff-b5e9-993a20d588aa","Type":"ContainerStarted","Data":"ce60eae137911ea5c9203c6cfc80f5938a12a48479ff47c86ff38f63d6757cc6"} Oct 30 09:14:01.306001 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:01.305559 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-3a61e1-6489f9fdfb-lt57n" event={"ID":"30a13eb2-b7e5-42ff-b5e9-993a20d588aa","Type":"ContainerStarted","Data":"7003f94e05bd38f7906ffb21f46b445fad024f56a572afd8c83086250bb978e7"} Oct 30 09:14:01.306001 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:01.305606 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-3a61e1-6489f9fdfb-lt57n" Oct 30 09:14:01.307245 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:01.307221 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3a61e1-689c7f9675-6b84c" event={"ID":"d1195ec8-c914-4d63-aca9-85e8b74b5d44","Type":"ContainerStarted","Data":"d9a285e72bb8d4fff6dff6fb999172014528e8505fc6faa6f129a7b80ca37644"} Oct 30 09:14:01.307365 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:01.307354 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3a61e1-689c7f9675-6b84c" Oct 30 09:14:01.362011 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:01.361958 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-3a61e1-6489f9fdfb-lt57n" podStartSLOduration=2.361945588 podStartE2EDuration="2.361945588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:13:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:14:01.360423176 +0000 UTC m=+1201.621984161" watchObservedRunningTime="2026-10-30 09:14:01.361945588 +0000 UTC m=+1201.623506575" Oct 30 09:14:01.411491 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:01.411444 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3a61e1-689c7f9675-6b84c" podStartSLOduration=2.411430939 podStartE2EDuration="2.411430939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:13:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:14:01.410166726 +0000 UTC m=+1201.671727726" watchObservedRunningTime="2026-10-30 09:14:01.411430939 +0000 UTC m=+1201.672991927" Oct 30 09:14:01.481903 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:01.481878 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-176b6c-6bf94b8cd6-sq67c"] Oct 30 09:14:01.482305 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:01.482293 4562 cpu_manager.go:414] "containerMap: removing container" podUID="5cb20235-f9a6-49be-8f34-038d586cd2e6" containerName="payload-processing" Oct 30 09:14:01.482343 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:01.482306 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cb20235-f9a6-49be-8f34-038d586cd2e6" containerName="payload-processing" Oct 30 09:14:01.482406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:01.482375 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cb20235-f9a6-49be-8f34-038d586cd2e6" containerName="payload-processing" Oct 30 09:14:01.485543 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:01.485528 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-176b6c-6bf94b8cd6-sq67c" Oct 30 09:14:01.505909 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:01.505884 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-176b6c-6bf94b8cd6-sq67c"] Oct 30 09:14:01.542572 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:01.542488 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-176b6c\"" Oct 30 09:14:01.553729 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:01.553695 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-176b6c-dockercfg-g6z2j\"" Oct 30 09:14:01.579505 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:01.579410 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a347b84-639e-4f96-9794-4463ee2b8af6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-176b6c-6bf94b8cd6-sq67c\" (UID: \"6a347b84-639e-4f96-9794-4463ee2b8af6\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-176b6c-6bf94b8cd6-sq67c" Oct 30 09:14:01.579505 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:01.579485 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4srh\" (UniqueName: \"kubernetes.io/projected/6a347b84-639e-4f96-9794-4463ee2b8af6-kube-api-access-k4srh\") pod \"payload-processing-e2e-worker-w3-176b6c-6bf94b8cd6-sq67c\" (UID: \"6a347b84-639e-4f96-9794-4463ee2b8af6\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-176b6c-6bf94b8cd6-sq67c" Oct 30 09:14:01.680958 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:01.680922 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a347b84-639e-4f96-9794-4463ee2b8af6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-176b6c-6bf94b8cd6-sq67c\" (UID: \"6a347b84-639e-4f96-9794-4463ee2b8af6\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-176b6c-6bf94b8cd6-sq67c" Oct 30 09:14:01.680958 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:01.680965 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4srh\" (UniqueName: \"kubernetes.io/projected/6a347b84-639e-4f96-9794-4463ee2b8af6-kube-api-access-k4srh\") pod \"payload-processing-e2e-worker-w3-176b6c-6bf94b8cd6-sq67c\" (UID: \"6a347b84-639e-4f96-9794-4463ee2b8af6\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-176b6c-6bf94b8cd6-sq67c" Oct 30 09:14:01.681655 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:01.681634 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a347b84-639e-4f96-9794-4463ee2b8af6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-176b6c-6bf94b8cd6-sq67c\" (UID: \"6a347b84-639e-4f96-9794-4463ee2b8af6\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-176b6c-6bf94b8cd6-sq67c" Oct 30 09:14:01.700574 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:01.700546 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4srh\" (UniqueName: \"kubernetes.io/projected/6a347b84-639e-4f96-9794-4463ee2b8af6-kube-api-access-k4srh\") pod \"payload-processing-e2e-worker-w3-176b6c-6bf94b8cd6-sq67c\" (UID: \"6a347b84-639e-4f96-9794-4463ee2b8af6\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-176b6c-6bf94b8cd6-sq67c" Oct 30 09:14:01.798196 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:01.798163 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-176b6c-6bf94b8cd6-sq67c" Oct 30 09:14:01.876503 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:01.876468 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cb20235-f9a6-49be-8f34-038d586cd2e6" path="/var/lib/kubelet/pods/5cb20235-f9a6-49be-8f34-038d586cd2e6/volumes" Oct 30 09:14:02.106595 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:14:02.106562 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a347b84_639e_4f96_9794_4463ee2b8af6.slice/crio-a2e4a0a002a462a907c563c5fa4aedef367d7b595a48c0a15ff10a6fd182e735 WatchSource:0}: Error finding container a2e4a0a002a462a907c563c5fa4aedef367d7b595a48c0a15ff10a6fd182e735: Status 404 returned error can't find the container with id a2e4a0a002a462a907c563c5fa4aedef367d7b595a48c0a15ff10a6fd182e735 Oct 30 09:14:02.107203 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:02.107174 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-176b6c-6bf94b8cd6-sq67c"] Oct 30 09:14:02.313495 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:02.313451 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-176b6c-6bf94b8cd6-sq67c" event={"ID":"6a347b84-639e-4f96-9794-4463ee2b8af6","Type":"ContainerStarted","Data":"1cdedacca131e43ad38a08faefe34643e9f70f57568b184304b2f18b7876424d"} Oct 30 09:14:02.313973 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:02.313514 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-176b6c-6bf94b8cd6-sq67c" event={"ID":"6a347b84-639e-4f96-9794-4463ee2b8af6","Type":"ContainerStarted","Data":"a2e4a0a002a462a907c563c5fa4aedef367d7b595a48c0a15ff10a6fd182e735"} Oct 30 09:14:02.313973 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:02.313923 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-176b6c-6bf94b8cd6-sq67c" Oct 30 09:14:02.354809 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:02.354758 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-176b6c-6bf94b8cd6-sq67c" podStartSLOduration=1.354747284 podStartE2EDuration="1.354747284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:14:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:14:02.352250585 +0000 UTC m=+1202.613811573" watchObservedRunningTime="2026-10-30 09:14:02.354747284 +0000 UTC m=+1202.616308271" Oct 30 09:14:03.260920 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:03.260879 4562 patch_prober.go:29] interesting pod/e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 09:14:03.261097 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:03.260951 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2" podUID="ba1039fc-94dc-490f-88b8-e789a117ff15" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 09:14:03.284613 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:03.284579 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e3d2ff-78f84d76-w6jp2" Oct 30 09:14:05.029701 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:05.029666 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-176b6c-65457966fb-55v76"] Oct 30 09:14:05.033628 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:05.033608 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-176b6c-65457966fb-55v76" Oct 30 09:14:05.064394 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:05.064360 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-176b6c-serving-cert\"" Oct 30 09:14:05.066660 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:05.066631 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-176b6c-65457966fb-55v76"] Oct 30 09:14:05.127352 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:05.127318 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd2e3358-d08c-4e6d-b0b4-1825157b40fc-metrics-certs\") pod \"maas-api-e2e-worker-w3-176b6c-65457966fb-55v76\" (UID: \"bd2e3358-d08c-4e6d-b0b4-1825157b40fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-176b6c-65457966fb-55v76" Oct 30 09:14:05.127352 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:05.127356 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd2e3358-d08c-4e6d-b0b4-1825157b40fc-maas-api-tls\") pod \"maas-api-e2e-worker-w3-176b6c-65457966fb-55v76\" (UID: \"bd2e3358-d08c-4e6d-b0b4-1825157b40fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-176b6c-65457966fb-55v76" Oct 30 09:14:05.127565 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:05.127427 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd2e3358-d08c-4e6d-b0b4-1825157b40fc-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-176b6c-65457966fb-55v76\" (UID: \"bd2e3358-d08c-4e6d-b0b4-1825157b40fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-176b6c-65457966fb-55v76" Oct 30 09:14:05.127565 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:05.127464 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvf46\" (UniqueName: \"kubernetes.io/projected/bd2e3358-d08c-4e6d-b0b4-1825157b40fc-kube-api-access-pvf46\") pod \"maas-api-e2e-worker-w3-176b6c-65457966fb-55v76\" (UID: \"bd2e3358-d08c-4e6d-b0b4-1825157b40fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-176b6c-65457966fb-55v76" Oct 30 09:14:05.229310 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:05.229272 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd2e3358-d08c-4e6d-b0b4-1825157b40fc-metrics-certs\") pod \"maas-api-e2e-worker-w3-176b6c-65457966fb-55v76\" (UID: \"bd2e3358-d08c-4e6d-b0b4-1825157b40fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-176b6c-65457966fb-55v76" Oct 30 09:14:05.229494 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:05.229325 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd2e3358-d08c-4e6d-b0b4-1825157b40fc-maas-api-tls\") pod \"maas-api-e2e-worker-w3-176b6c-65457966fb-55v76\" (UID: \"bd2e3358-d08c-4e6d-b0b4-1825157b40fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-176b6c-65457966fb-55v76" Oct 30 09:14:05.229494 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:05.229365 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd2e3358-d08c-4e6d-b0b4-1825157b40fc-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-176b6c-65457966fb-55v76\" (UID: \"bd2e3358-d08c-4e6d-b0b4-1825157b40fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-176b6c-65457966fb-55v76" Oct 30 09:14:05.229494 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:05.229417 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvf46\" (UniqueName: \"kubernetes.io/projected/bd2e3358-d08c-4e6d-b0b4-1825157b40fc-kube-api-access-pvf46\") pod \"maas-api-e2e-worker-w3-176b6c-65457966fb-55v76\" (UID: \"bd2e3358-d08c-4e6d-b0b4-1825157b40fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-176b6c-65457966fb-55v76" Oct 30 09:14:05.229693 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:14:05.229673 4562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-176b6c-serving-cert: secret "maas-api-e2e-worker-w3-176b6c-serving-cert" not found Oct 30 09:14:05.229762 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:14:05.229750 4562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd2e3358-d08c-4e6d-b0b4-1825157b40fc-maas-api-tls podName:bd2e3358-d08c-4e6d-b0b4-1825157b40fc nodeName:}" failed. No retries permitted until 2026-10-30 09:14:05.729730586 +0000 UTC m=+1205.991291568 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bd2e3358-d08c-4e6d-b0b4-1825157b40fc-maas-api-tls") pod "maas-api-e2e-worker-w3-176b6c-65457966fb-55v76" (UID: "bd2e3358-d08c-4e6d-b0b4-1825157b40fc") : secret "maas-api-e2e-worker-w3-176b6c-serving-cert" not found Oct 30 09:14:05.230598 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:05.230571 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd2e3358-d08c-4e6d-b0b4-1825157b40fc-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-176b6c-65457966fb-55v76\" (UID: \"bd2e3358-d08c-4e6d-b0b4-1825157b40fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-176b6c-65457966fb-55v76" Oct 30 09:14:05.233085 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:05.233049 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd2e3358-d08c-4e6d-b0b4-1825157b40fc-metrics-certs\") pod \"maas-api-e2e-worker-w3-176b6c-65457966fb-55v76\" (UID: \"bd2e3358-d08c-4e6d-b0b4-1825157b40fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-176b6c-65457966fb-55v76" Oct 30 09:14:05.249889 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:05.249865 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvf46\" (UniqueName: \"kubernetes.io/projected/bd2e3358-d08c-4e6d-b0b4-1825157b40fc-kube-api-access-pvf46\") pod \"maas-api-e2e-worker-w3-176b6c-65457966fb-55v76\" (UID: \"bd2e3358-d08c-4e6d-b0b4-1825157b40fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-176b6c-65457966fb-55v76" Oct 30 09:14:05.303397 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:05.303312 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2f69c8-5dbb84f778-2gp27" Oct 30 09:14:05.734972 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:05.734943 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd2e3358-d08c-4e6d-b0b4-1825157b40fc-maas-api-tls\") pod \"maas-api-e2e-worker-w3-176b6c-65457966fb-55v76\" (UID: \"bd2e3358-d08c-4e6d-b0b4-1825157b40fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-176b6c-65457966fb-55v76" Oct 30 09:14:05.737205 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:05.737181 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd2e3358-d08c-4e6d-b0b4-1825157b40fc-maas-api-tls\") pod \"maas-api-e2e-worker-w3-176b6c-65457966fb-55v76\" (UID: \"bd2e3358-d08c-4e6d-b0b4-1825157b40fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-176b6c-65457966fb-55v76" Oct 30 09:14:05.947179 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:05.947135 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-176b6c-65457966fb-55v76" Oct 30 09:14:06.161297 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:06.161274 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2" Oct 30 09:14:06.261771 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:14:06.261740 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd2e3358_d08c_4e6d_b0b4_1825157b40fc.slice/crio-6c1cf211080cb515c1dff27e197fc7260af1b70065d313350d15a4fd96590ecb WatchSource:0}: Error finding container 6c1cf211080cb515c1dff27e197fc7260af1b70065d313350d15a4fd96590ecb: Status 404 returned error can't find the container with id 6c1cf211080cb515c1dff27e197fc7260af1b70065d313350d15a4fd96590ecb Oct 30 09:14:06.262374 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:06.262348 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-176b6c-65457966fb-55v76"] Oct 30 09:14:06.335082 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:06.335041 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-176b6c-65457966fb-55v76" event={"ID":"bd2e3358-d08c-4e6d-b0b4-1825157b40fc","Type":"ContainerStarted","Data":"6c1cf211080cb515c1dff27e197fc7260af1b70065d313350d15a4fd96590ecb"} Oct 30 09:14:06.336793 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:06.336764 4562 generic.go:356] "Generic (PLEG): container finished" podID="ba1039fc-94dc-490f-88b8-e789a117ff15" containerID="18e493083d8e0298695f999a5f3be2ac21b8953ece39fd1334ba49248d688618" exitCode=0 Oct 30 09:14:06.336918 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:06.336823 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2" Oct 30 09:14:06.336918 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:06.336852 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2" event={"ID":"ba1039fc-94dc-490f-88b8-e789a117ff15","Type":"ContainerDied","Data":"18e493083d8e0298695f999a5f3be2ac21b8953ece39fd1334ba49248d688618"} Oct 30 09:14:06.336918 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:06.336912 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2" event={"ID":"ba1039fc-94dc-490f-88b8-e789a117ff15","Type":"ContainerDied","Data":"640ebfc5df0e6290ee6d0e1e8141cca81ce8da8255139d21d058ce247b6cf272"} Oct 30 09:14:06.337089 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:06.336945 4562 scope.go:122] "RemoveContainer" containerID="18e493083d8e0298695f999a5f3be2ac21b8953ece39fd1334ba49248d688618" Oct 30 09:14:06.340211 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:06.340188 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba1039fc-94dc-490f-88b8-e789a117ff15-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba1039fc-94dc-490f-88b8-e789a117ff15-credential-socket\") pod \"ba1039fc-94dc-490f-88b8-e789a117ff15\" (UID: \"ba1039fc-94dc-490f-88b8-e789a117ff15\") " Oct 30 09:14:06.340339 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:06.340217 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba1039fc-94dc-490f-88b8-e789a117ff15-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba1039fc-94dc-490f-88b8-e789a117ff15-istio-podinfo\") pod \"ba1039fc-94dc-490f-88b8-e789a117ff15\" (UID: \"ba1039fc-94dc-490f-88b8-e789a117ff15\") " Oct 30 09:14:06.340339 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:06.340328 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba1039fc-94dc-490f-88b8-e789a117ff15-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba1039fc-94dc-490f-88b8-e789a117ff15-workload-socket\") pod \"ba1039fc-94dc-490f-88b8-e789a117ff15\" (UID: \"ba1039fc-94dc-490f-88b8-e789a117ff15\") " Oct 30 09:14:06.340476 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:06.340413 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba1039fc-94dc-490f-88b8-e789a117ff15-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba1039fc-94dc-490f-88b8-e789a117ff15-istio-data\") pod \"ba1039fc-94dc-490f-88b8-e789a117ff15\" (UID: \"ba1039fc-94dc-490f-88b8-e789a117ff15\") " Oct 30 09:14:06.340476 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:06.340441 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba1039fc-94dc-490f-88b8-e789a117ff15-istio-token\" (UniqueName: \"kubernetes.io/projected/ba1039fc-94dc-490f-88b8-e789a117ff15-istio-token\") pod \"ba1039fc-94dc-490f-88b8-e789a117ff15\" (UID: \"ba1039fc-94dc-490f-88b8-e789a117ff15\") " Oct 30 09:14:06.340579 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:06.340478 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba1039fc-94dc-490f-88b8-e789a117ff15-credential-socket" pod "ba1039fc-94dc-490f-88b8-e789a117ff15" (UID: "ba1039fc-94dc-490f-88b8-e789a117ff15"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:14:06.340579 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:06.340499 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba1039fc-94dc-490f-88b8-e789a117ff15-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba1039fc-94dc-490f-88b8-e789a117ff15-workload-certs\") pod \"ba1039fc-94dc-490f-88b8-e789a117ff15\" (UID: \"ba1039fc-94dc-490f-88b8-e789a117ff15\") " Oct 30 09:14:06.340579 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:06.340566 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba1039fc-94dc-490f-88b8-e789a117ff15-kube-api-access-nbhz5\" (UniqueName: \"kubernetes.io/projected/ba1039fc-94dc-490f-88b8-e789a117ff15-kube-api-access-nbhz5\") pod \"ba1039fc-94dc-490f-88b8-e789a117ff15\" (UID: \"ba1039fc-94dc-490f-88b8-e789a117ff15\") " Oct 30 09:14:06.340732 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:06.340614 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba1039fc-94dc-490f-88b8-e789a117ff15-workload-socket" pod "ba1039fc-94dc-490f-88b8-e789a117ff15" (UID: "ba1039fc-94dc-490f-88b8-e789a117ff15"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:14:06.340732 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:06.340631 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba1039fc-94dc-490f-88b8-e789a117ff15-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba1039fc-94dc-490f-88b8-e789a117ff15-istio-envoy\") pod \"ba1039fc-94dc-490f-88b8-e789a117ff15\" (UID: \"ba1039fc-94dc-490f-88b8-e789a117ff15\") " Oct 30 09:14:06.340732 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:06.340679 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba1039fc-94dc-490f-88b8-e789a117ff15-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba1039fc-94dc-490f-88b8-e789a117ff15-istiod-ca-cert\") pod \"ba1039fc-94dc-490f-88b8-e789a117ff15\" (UID: \"ba1039fc-94dc-490f-88b8-e789a117ff15\") " Oct 30 09:14:06.340732 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:06.340664 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba1039fc-94dc-490f-88b8-e789a117ff15-istio-data" pod "ba1039fc-94dc-490f-88b8-e789a117ff15" (UID: "ba1039fc-94dc-490f-88b8-e789a117ff15"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:14:06.340732 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:06.340719 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba1039fc-94dc-490f-88b8-e789a117ff15-workload-certs" pod "ba1039fc-94dc-490f-88b8-e789a117ff15" (UID: "ba1039fc-94dc-490f-88b8-e789a117ff15"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:14:06.341109 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:06.341082 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba1039fc-94dc-490f-88b8-e789a117ff15-istiod-ca-cert" pod "ba1039fc-94dc-490f-88b8-e789a117ff15" (UID: "ba1039fc-94dc-490f-88b8-e789a117ff15"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:14:06.341186 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:06.341094 4562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba1039fc-94dc-490f-88b8-e789a117ff15-workload-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:14:06.341186 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:06.341130 4562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba1039fc-94dc-490f-88b8-e789a117ff15-credential-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:14:06.341186 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:06.341146 4562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba1039fc-94dc-490f-88b8-e789a117ff15-workload-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:14:06.341186 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:06.341160 4562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba1039fc-94dc-490f-88b8-e789a117ff15-istio-data\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:14:06.342763 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:06.342740 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba1039fc-94dc-490f-88b8-e789a117ff15-kube-api-access-nbhz5" pod "ba1039fc-94dc-490f-88b8-e789a117ff15" (UID: "ba1039fc-94dc-490f-88b8-e789a117ff15"). InnerVolumeSpecName "kube-api-access-nbhz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:14:06.342903 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:06.342888 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba1039fc-94dc-490f-88b8-e789a117ff15-istio-podinfo" pod "ba1039fc-94dc-490f-88b8-e789a117ff15" (UID: "ba1039fc-94dc-490f-88b8-e789a117ff15"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:14:06.343055 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:06.343027 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba1039fc-94dc-490f-88b8-e789a117ff15-istio-envoy" pod "ba1039fc-94dc-490f-88b8-e789a117ff15" (UID: "ba1039fc-94dc-490f-88b8-e789a117ff15"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:14:06.343140 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:06.343038 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba1039fc-94dc-490f-88b8-e789a117ff15-istio-token" pod "ba1039fc-94dc-490f-88b8-e789a117ff15" (UID: "ba1039fc-94dc-490f-88b8-e789a117ff15"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:14:06.347536 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:06.347518 4562 scope.go:122] "RemoveContainer" containerID="49d960c157e0b63e33dcf77c14df7633f485038836b75346ebecb80628b577c9" Oct 30 09:14:06.363630 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:06.363614 4562 scope.go:122] "RemoveContainer" containerID="18e493083d8e0298695f999a5f3be2ac21b8953ece39fd1334ba49248d688618" Oct 30 09:14:06.363864 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:14:06.363846 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18e493083d8e0298695f999a5f3be2ac21b8953ece39fd1334ba49248d688618\": container with ID starting with 18e493083d8e0298695f999a5f3be2ac21b8953ece39fd1334ba49248d688618 not found: ID does not exist" containerID="18e493083d8e0298695f999a5f3be2ac21b8953ece39fd1334ba49248d688618" Oct 30 09:14:06.363923 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:06.363879 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18e493083d8e0298695f999a5f3be2ac21b8953ece39fd1334ba49248d688618"} err="failed to get container status \"18e493083d8e0298695f999a5f3be2ac21b8953ece39fd1334ba49248d688618\": rpc error: code = NotFound desc = could not find container \"18e493083d8e0298695f999a5f3be2ac21b8953ece39fd1334ba49248d688618\": container with ID starting with 18e493083d8e0298695f999a5f3be2ac21b8953ece39fd1334ba49248d688618 not found: ID does not exist" Oct 30 09:14:06.363923 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:06.363893 4562 scope.go:122] "RemoveContainer" containerID="49d960c157e0b63e33dcf77c14df7633f485038836b75346ebecb80628b577c9" Oct 30 09:14:06.364109 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:14:06.364090 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49d960c157e0b63e33dcf77c14df7633f485038836b75346ebecb80628b577c9\": container with ID starting with 49d960c157e0b63e33dcf77c14df7633f485038836b75346ebecb80628b577c9 not found: ID does not exist" containerID="49d960c157e0b63e33dcf77c14df7633f485038836b75346ebecb80628b577c9" Oct 30 09:14:06.364159 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:06.364115 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49d960c157e0b63e33dcf77c14df7633f485038836b75346ebecb80628b577c9"} err="failed to get container status \"49d960c157e0b63e33dcf77c14df7633f485038836b75346ebecb80628b577c9\": rpc error: code = NotFound desc = could not find container \"49d960c157e0b63e33dcf77c14df7633f485038836b75346ebecb80628b577c9\": container with ID starting with 49d960c157e0b63e33dcf77c14df7633f485038836b75346ebecb80628b577c9 not found: ID does not exist" Oct 30 09:14:06.442049 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:06.442027 4562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba1039fc-94dc-490f-88b8-e789a117ff15-istio-token\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:14:06.442049 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:06.442050 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbhz5\" (UniqueName: \"kubernetes.io/projected/ba1039fc-94dc-490f-88b8-e789a117ff15-kube-api-access-nbhz5\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:14:06.442196 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:06.442060 4562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba1039fc-94dc-490f-88b8-e789a117ff15-istio-envoy\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:14:06.442196 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:06.442069 4562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba1039fc-94dc-490f-88b8-e789a117ff15-istiod-ca-cert\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:14:06.442196 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:06.442077 4562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba1039fc-94dc-490f-88b8-e789a117ff15-istio-podinfo\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:14:06.687754 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:06.687720 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2"] Oct 30 09:14:06.699529 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:06.699498 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-6cf9ebb1-openshift-default-7c78d64d4f-zklj2"] Oct 30 09:14:07.269993 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:07.269962 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-e3d2ff-cbb67b84b-j59gb" Oct 30 09:14:07.270377 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:07.270042 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e3d2ff-7ffc858867-5wpzf" Oct 30 09:14:07.345453 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:07.345421 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-176b6c-65457966fb-55v76" event={"ID":"bd2e3358-d08c-4e6d-b0b4-1825157b40fc","Type":"ContainerStarted","Data":"8fe4fb58282d6b704766ee1376c893e237d884440471d19bbeb1c8349edd998d"} Oct 30 09:14:07.345599 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:07.345548 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-176b6c-65457966fb-55v76" Oct 30 09:14:07.391908 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:07.391852 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-176b6c-65457966fb-55v76" podStartSLOduration=2.05870441 podStartE2EDuration="2.391840072s" totalImagesPullingTime="333.135662ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:14:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:14:07.390935234 +0000 UTC m=+1207.652496222" watchObservedRunningTime="2026-10-30 09:14:07.391840072 +0000 UTC m=+1207.653401059" Oct 30 09:14:07.873130 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:07.873098 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba1039fc-94dc-490f-88b8-e789a117ff15" path="/var/lib/kubelet/pods/ba1039fc-94dc-490f-88b8-e789a117ff15/volumes" Oct 30 09:14:07.983957 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:07.983926 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-c5447d-7bb465d9f8-89mpq"] Oct 30 09:14:07.984377 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:07.984365 4562 cpu_manager.go:414] "containerMap: removing container" podUID="ba1039fc-94dc-490f-88b8-e789a117ff15" containerName="istio-proxy" Oct 30 09:14:07.984445 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:07.984390 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba1039fc-94dc-490f-88b8-e789a117ff15" containerName="istio-proxy" Oct 30 09:14:07.984480 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:07.984471 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba1039fc-94dc-490f-88b8-e789a117ff15" containerName="istio-proxy" Oct 30 09:14:07.984518 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:07.984481 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba1039fc-94dc-490f-88b8-e789a117ff15" containerName="istio-proxy" Oct 30 09:14:07.989074 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:07.989044 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-c5447d-7bb465d9f8-89mpq" Oct 30 09:14:08.005310 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:08.005285 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-c5447d\"" Oct 30 09:14:08.006540 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:08.006520 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-c5447d-7bb465d9f8-89mpq"] Oct 30 09:14:08.017245 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:08.017221 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-c5447d-dockercfg-7nqqq\"" Oct 30 09:14:08.159861 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:08.159788 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e125f79-421e-4ac0-8f37-8bbd5f430c9a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-c5447d-7bb465d9f8-89mpq\" (UID: \"3e125f79-421e-4ac0-8f37-8bbd5f430c9a\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-c5447d-7bb465d9f8-89mpq" Oct 30 09:14:08.160027 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:08.159951 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnr5c\" (UniqueName: \"kubernetes.io/projected/3e125f79-421e-4ac0-8f37-8bbd5f430c9a-kube-api-access-lnr5c\") pod \"payload-processing-e2e-worker-w1-c5447d-7bb465d9f8-89mpq\" (UID: \"3e125f79-421e-4ac0-8f37-8bbd5f430c9a\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-c5447d-7bb465d9f8-89mpq" Oct 30 09:14:08.260761 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:08.260719 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnr5c\" (UniqueName: \"kubernetes.io/projected/3e125f79-421e-4ac0-8f37-8bbd5f430c9a-kube-api-access-lnr5c\") pod \"payload-processing-e2e-worker-w1-c5447d-7bb465d9f8-89mpq\" (UID: \"3e125f79-421e-4ac0-8f37-8bbd5f430c9a\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-c5447d-7bb465d9f8-89mpq" Oct 30 09:14:08.260914 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:08.260788 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e125f79-421e-4ac0-8f37-8bbd5f430c9a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-c5447d-7bb465d9f8-89mpq\" (UID: \"3e125f79-421e-4ac0-8f37-8bbd5f430c9a\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-c5447d-7bb465d9f8-89mpq" Oct 30 09:14:08.261487 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:08.261465 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e125f79-421e-4ac0-8f37-8bbd5f430c9a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-c5447d-7bb465d9f8-89mpq\" (UID: \"3e125f79-421e-4ac0-8f37-8bbd5f430c9a\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-c5447d-7bb465d9f8-89mpq" Oct 30 09:14:08.280797 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:08.280759 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnr5c\" (UniqueName: \"kubernetes.io/projected/3e125f79-421e-4ac0-8f37-8bbd5f430c9a-kube-api-access-lnr5c\") pod \"payload-processing-e2e-worker-w1-c5447d-7bb465d9f8-89mpq\" (UID: \"3e125f79-421e-4ac0-8f37-8bbd5f430c9a\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-c5447d-7bb465d9f8-89mpq" Oct 30 09:14:08.304483 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:08.304448 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-c5447d-7bb465d9f8-89mpq" Oct 30 09:14:08.612644 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:14:08.612595 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e125f79_421e_4ac0_8f37_8bbd5f430c9a.slice/crio-74b8bf0b1db1fdbb647be777f62b784951276e406f1c6be7934a2c3dbd3bffef WatchSource:0}: Error finding container 74b8bf0b1db1fdbb647be777f62b784951276e406f1c6be7934a2c3dbd3bffef: Status 404 returned error can't find the container with id 74b8bf0b1db1fdbb647be777f62b784951276e406f1c6be7934a2c3dbd3bffef Oct 30 09:14:08.614278 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:08.614236 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-c5447d-7bb465d9f8-89mpq"] Oct 30 09:14:09.367565 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:09.367520 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-c5447d-7bb465d9f8-89mpq" event={"ID":"3e125f79-421e-4ac0-8f37-8bbd5f430c9a","Type":"ContainerStarted","Data":"a5a857705f309283aa0246600a32e3f0faac2040dbb6855e9854236da8ff5e6d"} Oct 30 09:14:09.368034 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:09.367591 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-c5447d-7bb465d9f8-89mpq" event={"ID":"3e125f79-421e-4ac0-8f37-8bbd5f430c9a","Type":"ContainerStarted","Data":"74b8bf0b1db1fdbb647be777f62b784951276e406f1c6be7934a2c3dbd3bffef"} Oct 30 09:14:09.368485 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:09.368463 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-c5447d-7bb465d9f8-89mpq" Oct 30 09:14:09.406468 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:09.406416 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-c5447d-7bb465d9f8-89mpq" podStartSLOduration=2.406401683 podStartE2EDuration="2.406401683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:14:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:14:09.406223863 +0000 UTC m=+1209.667784851" watchObservedRunningTime="2026-10-30 09:14:09.406401683 +0000 UTC m=+1209.667962677" Oct 30 09:14:12.316670 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:12.316644 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3a61e1-689c7f9675-6b84c" Oct 30 09:14:12.317069 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:12.316717 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-3a61e1-6489f9fdfb-lt57n" Oct 30 09:14:13.322200 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:13.322125 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-176b6c-6bf94b8cd6-sq67c" Oct 30 09:14:13.368359 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:13.368326 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-176b6c-65457966fb-55v76" Oct 30 09:14:13.708821 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:13.708784 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-c5447d-5c56c8576b-zpvxb"] Oct 30 09:14:13.709233 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:13.709220 4562 cpu_manager.go:414] "containerMap: removing container" podUID="ba1039fc-94dc-490f-88b8-e789a117ff15" containerName="istio-proxy" Oct 30 09:14:13.709275 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:13.709234 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba1039fc-94dc-490f-88b8-e789a117ff15" containerName="istio-proxy" Oct 30 09:14:13.712842 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:13.712822 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c5447d-5c56c8576b-zpvxb" Oct 30 09:14:13.728708 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:13.728681 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-c5447d-serving-cert\"" Oct 30 09:14:13.730093 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:13.730074 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-c5447d-5c56c8576b-zpvxb"] Oct 30 09:14:13.813157 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:13.813121 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18ca1016-fd69-495d-9398-320aa16c48cc-metrics-certs\") pod \"maas-api-e2e-worker-w1-c5447d-5c56c8576b-zpvxb\" (UID: \"18ca1016-fd69-495d-9398-320aa16c48cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c5447d-5c56c8576b-zpvxb" Oct 30 09:14:13.813316 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:13.813211 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg5l5\" (UniqueName: \"kubernetes.io/projected/18ca1016-fd69-495d-9398-320aa16c48cc-kube-api-access-lg5l5\") pod \"maas-api-e2e-worker-w1-c5447d-5c56c8576b-zpvxb\" (UID: \"18ca1016-fd69-495d-9398-320aa16c48cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c5447d-5c56c8576b-zpvxb" Oct 30 09:14:13.813420 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:13.813313 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18ca1016-fd69-495d-9398-320aa16c48cc-maas-api-tls\") pod \"maas-api-e2e-worker-w1-c5447d-5c56c8576b-zpvxb\" (UID: \"18ca1016-fd69-495d-9398-320aa16c48cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c5447d-5c56c8576b-zpvxb" Oct 30 09:14:13.813420 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:13.813403 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18ca1016-fd69-495d-9398-320aa16c48cc-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-c5447d-5c56c8576b-zpvxb\" (UID: \"18ca1016-fd69-495d-9398-320aa16c48cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c5447d-5c56c8576b-zpvxb" Oct 30 09:14:13.914077 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:13.914036 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18ca1016-fd69-495d-9398-320aa16c48cc-metrics-certs\") pod \"maas-api-e2e-worker-w1-c5447d-5c56c8576b-zpvxb\" (UID: \"18ca1016-fd69-495d-9398-320aa16c48cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c5447d-5c56c8576b-zpvxb" Oct 30 09:14:13.914246 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:13.914096 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg5l5\" (UniqueName: \"kubernetes.io/projected/18ca1016-fd69-495d-9398-320aa16c48cc-kube-api-access-lg5l5\") pod \"maas-api-e2e-worker-w1-c5447d-5c56c8576b-zpvxb\" (UID: \"18ca1016-fd69-495d-9398-320aa16c48cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c5447d-5c56c8576b-zpvxb" Oct 30 09:14:13.914246 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:13.914158 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18ca1016-fd69-495d-9398-320aa16c48cc-maas-api-tls\") pod \"maas-api-e2e-worker-w1-c5447d-5c56c8576b-zpvxb\" (UID: \"18ca1016-fd69-495d-9398-320aa16c48cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c5447d-5c56c8576b-zpvxb" Oct 30 09:14:13.914246 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:13.914239 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18ca1016-fd69-495d-9398-320aa16c48cc-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-c5447d-5c56c8576b-zpvxb\" (UID: \"18ca1016-fd69-495d-9398-320aa16c48cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c5447d-5c56c8576b-zpvxb" Oct 30 09:14:13.914929 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:13.914895 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18ca1016-fd69-495d-9398-320aa16c48cc-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-c5447d-5c56c8576b-zpvxb\" (UID: \"18ca1016-fd69-495d-9398-320aa16c48cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c5447d-5c56c8576b-zpvxb" Oct 30 09:14:13.916697 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:13.916671 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18ca1016-fd69-495d-9398-320aa16c48cc-maas-api-tls\") pod \"maas-api-e2e-worker-w1-c5447d-5c56c8576b-zpvxb\" (UID: \"18ca1016-fd69-495d-9398-320aa16c48cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c5447d-5c56c8576b-zpvxb" Oct 30 09:14:13.916799 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:13.916679 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18ca1016-fd69-495d-9398-320aa16c48cc-metrics-certs\") pod \"maas-api-e2e-worker-w1-c5447d-5c56c8576b-zpvxb\" (UID: \"18ca1016-fd69-495d-9398-320aa16c48cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c5447d-5c56c8576b-zpvxb" Oct 30 09:14:13.937143 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:13.937111 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg5l5\" (UniqueName: \"kubernetes.io/projected/18ca1016-fd69-495d-9398-320aa16c48cc-kube-api-access-lg5l5\") pod \"maas-api-e2e-worker-w1-c5447d-5c56c8576b-zpvxb\" (UID: \"18ca1016-fd69-495d-9398-320aa16c48cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c5447d-5c56c8576b-zpvxb" Oct 30 09:14:14.025750 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:14.025673 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c5447d-5c56c8576b-zpvxb" Oct 30 09:14:14.372503 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:14.371888 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-c5447d-5c56c8576b-zpvxb"] Oct 30 09:14:14.395727 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:14.395695 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c5447d-5c56c8576b-zpvxb" event={"ID":"18ca1016-fd69-495d-9398-320aa16c48cc","Type":"ContainerStarted","Data":"3c5639d0cb09fe979099fa0d9ad4f8d78a5395a1a678c1ea545588d3aa063067"} Oct 30 09:14:15.405288 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:15.405244 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c5447d-5c56c8576b-zpvxb" event={"ID":"18ca1016-fd69-495d-9398-320aa16c48cc","Type":"ContainerStarted","Data":"1cda1433af6840decad3edacb0763b0d317a98501b7112767b1d4cc4b89a233e"} Oct 30 09:14:15.405639 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:15.405377 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c5447d-5c56c8576b-zpvxb" Oct 30 09:14:15.448294 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:15.448234 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c5447d-5c56c8576b-zpvxb" podStartSLOduration=2.0493052880000002 podStartE2EDuration="2.448220488s" totalImagesPullingTime="398.9152ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:14:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:14:15.446586536 +0000 UTC m=+1215.708147548" watchObservedRunningTime="2026-10-30 09:14:15.448220488 +0000 UTC m=+1215.709781476" Oct 30 09:14:19.430023 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:19.429990 4562 generic.go:356] "Generic (PLEG): container finished" podID="0ba1f223-8fd5-499b-ac09-73fc50031a7d" containerID="ed8d5f5580de3dbeb252ed0bb25fd1f08967b60724caa0a6e6ef3f6e4c964500" exitCode=0 Oct 30 09:14:19.430452 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:19.430070 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw" event={"ID":"0ba1f223-8fd5-499b-ac09-73fc50031a7d","Type":"ContainerDied","Data":"ed8d5f5580de3dbeb252ed0bb25fd1f08967b60724caa0a6e6ef3f6e4c964500"} Oct 30 09:14:19.430632 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:19.430613 4562 scope.go:122] "RemoveContainer" containerID="ed8d5f5580de3dbeb252ed0bb25fd1f08967b60724caa0a6e6ef3f6e4c964500" Oct 30 09:14:19.432297 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:19.432259 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:14:19.432510 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:19.432375 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:14:19.432510 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:19.432451 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:14:20.437036 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:20.437000 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw" event={"ID":"0ba1f223-8fd5-499b-ac09-73fc50031a7d","Type":"ContainerStarted","Data":"d3577f1fdce308122eee270b739d549f3ab220909353ec1bad3092800432050e"} Oct 30 09:14:21.363601 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:21.363559 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw" Oct 30 09:14:21.364112 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:21.364078 4562 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 30 09:14:21.364247 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:21.364171 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw" podUID="0ba1f223-8fd5-499b-ac09-73fc50031a7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 30 09:14:21.380507 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:21.380486 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-c5447d-7bb465d9f8-89mpq" Oct 30 09:14:21.415788 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:21.415766 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c5447d-5c56c8576b-zpvxb" Oct 30 09:14:22.363845 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:22.363798 4562 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 30 09:14:22.364347 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:22.363863 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw" podUID="0ba1f223-8fd5-499b-ac09-73fc50031a7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 30 09:14:23.363758 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:23.363720 4562 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 30 09:14:23.363940 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:23.363774 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw" podUID="0ba1f223-8fd5-499b-ac09-73fc50031a7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 30 09:14:24.361829 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:24.361789 4562 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 30 09:14:24.362151 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:24.362121 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw" podUID="0ba1f223-8fd5-499b-ac09-73fc50031a7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 30 09:14:25.362425 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:25.362377 4562 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 30 09:14:25.362869 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:25.362473 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw" podUID="0ba1f223-8fd5-499b-ac09-73fc50031a7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 30 09:14:26.260346 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:26.260317 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw" Oct 30 09:14:26.365735 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:26.365707 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw" Oct 30 09:14:26.501485 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:14:26.501459 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw" Oct 30 09:14:28.595753 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:14:28.595721 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:14:58.600480 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:14:58.600440 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:15:28.595267 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:15:28.595231 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:15:48.736788 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:48.736707 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5"] Oct 30 09:15:48.741066 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:48.741043 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5" Oct 30 09:15:48.756063 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:48.756039 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-e3d2ff-openshift-default-dockercfg-j9w6q\"" Oct 30 09:15:48.771650 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:48.771623 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5"] Oct 30 09:15:48.876889 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:48.876855 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkfzz\" (UniqueName: \"kubernetes.io/projected/4c0ac365-a552-4514-a497-748845006a2e-kube-api-access-rkfzz\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5\" (UID: \"4c0ac365-a552-4514-a497-748845006a2e\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5" Oct 30 09:15:48.877068 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:48.876900 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c0ac365-a552-4514-a497-748845006a2e-istio-podinfo\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5\" (UID: \"4c0ac365-a552-4514-a497-748845006a2e\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5" Oct 30 09:15:48.877068 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:48.876925 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c0ac365-a552-4514-a497-748845006a2e-workload-certs\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5\" (UID: \"4c0ac365-a552-4514-a497-748845006a2e\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5" Oct 30 09:15:48.877068 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:48.876948 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c0ac365-a552-4514-a497-748845006a2e-istio-envoy\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5\" (UID: \"4c0ac365-a552-4514-a497-748845006a2e\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5" Oct 30 09:15:48.877209 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:48.877105 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c0ac365-a552-4514-a497-748845006a2e-istiod-ca-cert\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5\" (UID: \"4c0ac365-a552-4514-a497-748845006a2e\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5" Oct 30 09:15:48.877209 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:48.877167 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c0ac365-a552-4514-a497-748845006a2e-istio-token\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5\" (UID: \"4c0ac365-a552-4514-a497-748845006a2e\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5" Oct 30 09:15:48.877320 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:48.877272 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c0ac365-a552-4514-a497-748845006a2e-workload-socket\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5\" (UID: \"4c0ac365-a552-4514-a497-748845006a2e\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5" Oct 30 09:15:48.877394 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:48.877366 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c0ac365-a552-4514-a497-748845006a2e-istio-data\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5\" (UID: \"4c0ac365-a552-4514-a497-748845006a2e\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5" Oct 30 09:15:48.877430 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:48.877402 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c0ac365-a552-4514-a497-748845006a2e-credential-socket\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5\" (UID: \"4c0ac365-a552-4514-a497-748845006a2e\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5" Oct 30 09:15:48.978891 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:48.978855 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c0ac365-a552-4514-a497-748845006a2e-istiod-ca-cert\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5\" (UID: \"4c0ac365-a552-4514-a497-748845006a2e\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5" Oct 30 09:15:48.979071 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:48.978913 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c0ac365-a552-4514-a497-748845006a2e-istio-token\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5\" (UID: \"4c0ac365-a552-4514-a497-748845006a2e\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5" Oct 30 09:15:48.979071 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:48.978971 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c0ac365-a552-4514-a497-748845006a2e-workload-socket\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5\" (UID: \"4c0ac365-a552-4514-a497-748845006a2e\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5" Oct 30 09:15:48.979071 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:48.979028 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c0ac365-a552-4514-a497-748845006a2e-istio-data\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5\" (UID: \"4c0ac365-a552-4514-a497-748845006a2e\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5" Oct 30 09:15:48.979071 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:48.979062 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c0ac365-a552-4514-a497-748845006a2e-credential-socket\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5\" (UID: \"4c0ac365-a552-4514-a497-748845006a2e\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5" Oct 30 09:15:48.979285 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:48.979151 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkfzz\" (UniqueName: \"kubernetes.io/projected/4c0ac365-a552-4514-a497-748845006a2e-kube-api-access-rkfzz\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5\" (UID: \"4c0ac365-a552-4514-a497-748845006a2e\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5" Oct 30 09:15:48.979285 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:48.979192 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c0ac365-a552-4514-a497-748845006a2e-istio-podinfo\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5\" (UID: \"4c0ac365-a552-4514-a497-748845006a2e\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5" Oct 30 09:15:48.979285 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:48.979231 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c0ac365-a552-4514-a497-748845006a2e-workload-certs\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5\" (UID: \"4c0ac365-a552-4514-a497-748845006a2e\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5" Oct 30 09:15:48.979285 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:48.979272 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c0ac365-a552-4514-a497-748845006a2e-istio-envoy\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5\" (UID: \"4c0ac365-a552-4514-a497-748845006a2e\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5" Oct 30 09:15:48.979713 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:48.979683 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c0ac365-a552-4514-a497-748845006a2e-credential-socket\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5\" (UID: \"4c0ac365-a552-4514-a497-748845006a2e\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5" Oct 30 09:15:48.979827 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:48.979717 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c0ac365-a552-4514-a497-748845006a2e-istiod-ca-cert\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5\" (UID: \"4c0ac365-a552-4514-a497-748845006a2e\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5" Oct 30 09:15:48.980198 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:48.980172 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c0ac365-a552-4514-a497-748845006a2e-workload-socket\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5\" (UID: \"4c0ac365-a552-4514-a497-748845006a2e\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5" Oct 30 09:15:48.980427 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:48.980194 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c0ac365-a552-4514-a497-748845006a2e-istio-data\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5\" (UID: \"4c0ac365-a552-4514-a497-748845006a2e\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5" Oct 30 09:15:48.980515 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:48.980277 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c0ac365-a552-4514-a497-748845006a2e-workload-certs\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5\" (UID: \"4c0ac365-a552-4514-a497-748845006a2e\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5" Oct 30 09:15:48.981981 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:48.981955 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c0ac365-a552-4514-a497-748845006a2e-istio-envoy\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5\" (UID: \"4c0ac365-a552-4514-a497-748845006a2e\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5" Oct 30 09:15:48.982289 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:48.982257 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c0ac365-a552-4514-a497-748845006a2e-istio-podinfo\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5\" (UID: \"4c0ac365-a552-4514-a497-748845006a2e\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5" Oct 30 09:15:48.998907 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:48.998834 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c0ac365-a552-4514-a497-748845006a2e-istio-token\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5\" (UID: \"4c0ac365-a552-4514-a497-748845006a2e\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5" Oct 30 09:15:48.999082 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:48.999062 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkfzz\" (UniqueName: \"kubernetes.io/projected/4c0ac365-a552-4514-a497-748845006a2e-kube-api-access-rkfzz\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5\" (UID: \"4c0ac365-a552-4514-a497-748845006a2e\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5" Oct 30 09:15:49.054611 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:49.054586 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5" Oct 30 09:15:49.373280 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:49.373256 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5"] Oct 30 09:15:49.384632 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:49.383308 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:15:49.384632 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:49.383411 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:15:49.384632 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:49.383462 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:15:49.952405 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:49.941494 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5" event={"ID":"4c0ac365-a552-4514-a497-748845006a2e","Type":"ContainerStarted","Data":"74e7d1511b8a823cd994d874dd42544ce8e579f0b80778677098c7ecaaac1e64"} Oct 30 09:15:49.952405 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:49.941558 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5" event={"ID":"4c0ac365-a552-4514-a497-748845006a2e","Type":"ContainerStarted","Data":"34bf5cf7a25fd9b826b51372f079c5d5a597d2b16096d70e88baa62e90220045"} Oct 30 09:15:50.063406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:50.063028 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5" Oct 30 09:15:50.066213 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:50.066181 4562 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 09:15:50.066459 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:50.066429 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5" podUID="4c0ac365-a552-4514-a497-748845006a2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 09:15:51.059556 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:51.058495 4562 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 09:15:51.059556 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:51.058552 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5" podUID="4c0ac365-a552-4514-a497-748845006a2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 09:15:52.070482 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:52.070438 4562 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 09:15:52.070985 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:52.070519 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5" podUID="4c0ac365-a552-4514-a497-748845006a2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 09:15:53.062405 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:53.061339 4562 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 09:15:53.062405 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:53.061432 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5" podUID="4c0ac365-a552-4514-a497-748845006a2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 09:15:54.057406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:54.057272 4562 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 09:15:54.057406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:54.057339 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5" podUID="4c0ac365-a552-4514-a497-748845006a2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 09:15:54.233353 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:54.233292 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5" podStartSLOduration=6.233276028 podStartE2EDuration="6.233276028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:15:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:15:49.993543318 +0000 UTC m=+1310.255104307" watchObservedRunningTime="2026-10-30 09:15:54.233276028 +0000 UTC m=+1314.494837016" Oct 30 09:15:54.233905 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:54.233878 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf"] Oct 30 09:15:54.258404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:54.251032 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf" Oct 30 09:15:54.290977 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:54.290935 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf"] Oct 30 09:15:54.293591 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:54.293566 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-3a61e1-openshift-default-dockercfg-vvdt5\"" Oct 30 09:15:54.391108 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:54.391017 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4-istio-token\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf\" (UID: \"cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf" Oct 30 09:15:54.391108 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:54.391073 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4-istio-data\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf\" (UID: \"cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf" Oct 30 09:15:54.391345 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:54.391110 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4-istio-envoy\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf\" (UID: \"cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf" Oct 30 09:15:54.391345 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:54.391154 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4-workload-certs\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf\" (UID: \"cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf" Oct 30 09:15:54.391345 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:54.391211 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4-workload-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf\" (UID: \"cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf" Oct 30 09:15:54.391345 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:54.391235 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4-istio-podinfo\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf\" (UID: \"cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf" Oct 30 09:15:54.391345 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:54.391265 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4-istiod-ca-cert\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf\" (UID: \"cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf" Oct 30 09:15:54.391345 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:54.391296 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr6s5\" (UniqueName: \"kubernetes.io/projected/cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4-kube-api-access-tr6s5\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf\" (UID: \"cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf" Oct 30 09:15:54.391345 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:54.391344 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4-credential-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf\" (UID: \"cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf" Oct 30 09:15:54.511145 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:54.492233 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4-workload-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf\" (UID: \"cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf" Oct 30 09:15:54.511145 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:54.492281 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4-istio-podinfo\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf\" (UID: \"cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf" Oct 30 09:15:54.511145 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:54.492316 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4-istiod-ca-cert\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf\" (UID: \"cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf" Oct 30 09:15:54.511145 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:54.492347 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr6s5\" (UniqueName: \"kubernetes.io/projected/cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4-kube-api-access-tr6s5\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf\" (UID: \"cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf" Oct 30 09:15:54.511145 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:54.492406 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4-credential-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf\" (UID: \"cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf" Oct 30 09:15:54.511145 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:54.492498 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4-istio-token\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf\" (UID: \"cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf" Oct 30 09:15:54.511145 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:54.492543 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4-istio-data\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf\" (UID: \"cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf" Oct 30 09:15:54.511145 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:54.496889 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4-istio-envoy\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf\" (UID: \"cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf" Oct 30 09:15:54.511145 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:54.496993 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4-workload-certs\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf\" (UID: \"cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf" Oct 30 09:15:54.511145 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:54.497568 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4-workload-certs\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf\" (UID: \"cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf" Oct 30 09:15:54.511145 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:54.497814 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4-workload-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf\" (UID: \"cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf" Oct 30 09:15:54.511145 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:54.498497 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4-credential-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf\" (UID: \"cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf" Oct 30 09:15:54.511145 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:54.499093 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4-istiod-ca-cert\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf\" (UID: \"cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf" Oct 30 09:15:54.511145 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:54.499635 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4-istio-data\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf\" (UID: \"cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf" Oct 30 09:15:54.511145 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:54.502659 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4-istio-envoy\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf\" (UID: \"cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf" Oct 30 09:15:54.511145 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:54.506273 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4-istio-podinfo\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf\" (UID: \"cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf" Oct 30 09:15:54.521147 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:54.521125 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4-istio-token\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf\" (UID: \"cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf" Oct 30 09:15:54.522197 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:54.522179 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr6s5\" (UniqueName: \"kubernetes.io/projected/cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4-kube-api-access-tr6s5\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf\" (UID: \"cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf" Oct 30 09:15:54.595839 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:54.595804 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf" Oct 30 09:15:54.930817 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:54.930789 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf"] Oct 30 09:15:54.930920 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:15:54.930870 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb592c8a_aca2_4ba5_9de6_51d9cfc3bfb4.slice/crio-b1470b2d2cf35b59b3096dacae068966b3e530fb0d59eafe0daedca9c3f19c6a WatchSource:0}: Error finding container b1470b2d2cf35b59b3096dacae068966b3e530fb0d59eafe0daedca9c3f19c6a: Status 404 returned error can't find the container with id b1470b2d2cf35b59b3096dacae068966b3e530fb0d59eafe0daedca9c3f19c6a Oct 30 09:15:54.932956 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:54.932928 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:15:54.933024 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:54.932986 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:15:54.933060 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:54.933022 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:15:55.003481 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:55.003456 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf" event={"ID":"cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4","Type":"ContainerStarted","Data":"b1470b2d2cf35b59b3096dacae068966b3e530fb0d59eafe0daedca9c3f19c6a"} Oct 30 09:15:55.066405 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:55.065469 4562 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 09:15:55.066405 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:55.065527 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5" podUID="4c0ac365-a552-4514-a497-748845006a2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 09:15:56.010721 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:56.010676 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf" event={"ID":"cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4","Type":"ContainerStarted","Data":"5a1aca49e4c943a5bbfff5b7d725110f9c3f989317e27bcfd21290f5f3ef7fc5"} Oct 30 09:15:56.086734 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:56.086665 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf" podStartSLOduration=2.086647299 podStartE2EDuration="2.086647299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:15:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:15:56.071797416 +0000 UTC m=+1316.333358404" watchObservedRunningTime="2026-10-30 09:15:56.086647299 +0000 UTC m=+1316.348208289" Oct 30 09:15:56.596733 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:56.596687 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf" Oct 30 09:15:56.597879 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:56.597849 4562 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 09:15:56.598031 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:56.597911 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf" podUID="cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 09:15:56.912242 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:56.912153 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5" Oct 30 09:15:56.912567 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:56.912540 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5" Oct 30 09:15:56.913594 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:56.913569 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5" Oct 30 09:15:57.597066 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:57.597030 4562 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 09:15:57.597459 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:57.597082 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf" podUID="cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 09:15:58.595754 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:15:58.595719 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:15:58.597100 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:58.597080 4562 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 09:15:58.597411 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:58.597132 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf" podUID="cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 09:15:59.596811 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:59.596773 4562 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 09:15:59.596979 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:15:59.596828 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf" podUID="cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 09:16:00.596653 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:00.596614 4562 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 09:16:00.597059 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:00.596667 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf" podUID="cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 09:16:02.594680 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:02.594628 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf" Oct 30 09:16:02.595435 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:02.595415 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf" Oct 30 09:16:02.604898 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:02.604863 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf" Oct 30 09:16:20.176252 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:20.176215 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd"] Oct 30 09:16:20.181173 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:20.181152 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd" Oct 30 09:16:20.198356 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:20.198325 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd"] Oct 30 09:16:20.211693 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:20.211662 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-2f69c8-openshift-default-dockercfg-hmrnb\"" Oct 30 09:16:20.322988 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:20.322932 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eeb2baa-0243-4032-94c4-30be54e2cef5-istio-data\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd\" (UID: \"1eeb2baa-0243-4032-94c4-30be54e2cef5\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd" Oct 30 09:16:20.323164 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:20.323027 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8jkv\" (UniqueName: \"kubernetes.io/projected/1eeb2baa-0243-4032-94c4-30be54e2cef5-kube-api-access-x8jkv\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd\" (UID: \"1eeb2baa-0243-4032-94c4-30be54e2cef5\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd" Oct 30 09:16:20.323164 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:20.323053 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eeb2baa-0243-4032-94c4-30be54e2cef5-workload-socket\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd\" (UID: \"1eeb2baa-0243-4032-94c4-30be54e2cef5\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd" Oct 30 09:16:20.323286 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:20.323151 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eeb2baa-0243-4032-94c4-30be54e2cef5-credential-socket\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd\" (UID: \"1eeb2baa-0243-4032-94c4-30be54e2cef5\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd" Oct 30 09:16:20.323286 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:20.323213 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eeb2baa-0243-4032-94c4-30be54e2cef5-istio-podinfo\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd\" (UID: \"1eeb2baa-0243-4032-94c4-30be54e2cef5\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd" Oct 30 09:16:20.323399 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:20.323364 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1eeb2baa-0243-4032-94c4-30be54e2cef5-istio-token\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd\" (UID: \"1eeb2baa-0243-4032-94c4-30be54e2cef5\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd" Oct 30 09:16:20.323465 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:20.323448 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eeb2baa-0243-4032-94c4-30be54e2cef5-istio-envoy\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd\" (UID: \"1eeb2baa-0243-4032-94c4-30be54e2cef5\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd" Oct 30 09:16:20.323516 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:20.323485 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eeb2baa-0243-4032-94c4-30be54e2cef5-workload-certs\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd\" (UID: \"1eeb2baa-0243-4032-94c4-30be54e2cef5\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd" Oct 30 09:16:20.323622 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:20.323599 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eeb2baa-0243-4032-94c4-30be54e2cef5-istiod-ca-cert\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd\" (UID: \"1eeb2baa-0243-4032-94c4-30be54e2cef5\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd" Oct 30 09:16:20.406368 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:20.406329 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-176b6c/e2e-conflict-a-3231e37d-kserve-65b67bbd57-655l7"] Oct 30 09:16:20.410036 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:20.410021 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-176b6c/e2e-conflict-a-3231e37d-kserve-65b67bbd57-655l7" Oct 30 09:16:20.424636 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:20.424584 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eeb2baa-0243-4032-94c4-30be54e2cef5-istiod-ca-cert\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd\" (UID: \"1eeb2baa-0243-4032-94c4-30be54e2cef5\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd" Oct 30 09:16:20.424867 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:20.424729 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eeb2baa-0243-4032-94c4-30be54e2cef5-istio-data\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd\" (UID: \"1eeb2baa-0243-4032-94c4-30be54e2cef5\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd" Oct 30 09:16:20.424867 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:20.424823 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8jkv\" (UniqueName: \"kubernetes.io/projected/1eeb2baa-0243-4032-94c4-30be54e2cef5-kube-api-access-x8jkv\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd\" (UID: \"1eeb2baa-0243-4032-94c4-30be54e2cef5\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd" Oct 30 09:16:20.425052 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:20.424864 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eeb2baa-0243-4032-94c4-30be54e2cef5-workload-socket\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd\" (UID: \"1eeb2baa-0243-4032-94c4-30be54e2cef5\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd" Oct 30 09:16:20.425052 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:20.424908 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eeb2baa-0243-4032-94c4-30be54e2cef5-credential-socket\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd\" (UID: \"1eeb2baa-0243-4032-94c4-30be54e2cef5\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd" Oct 30 09:16:20.425052 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:20.424931 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eeb2baa-0243-4032-94c4-30be54e2cef5-istio-podinfo\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd\" (UID: \"1eeb2baa-0243-4032-94c4-30be54e2cef5\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd" Oct 30 09:16:20.425052 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:20.424971 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1eeb2baa-0243-4032-94c4-30be54e2cef5-istio-token\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd\" (UID: \"1eeb2baa-0243-4032-94c4-30be54e2cef5\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd" Oct 30 09:16:20.425052 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:20.425018 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eeb2baa-0243-4032-94c4-30be54e2cef5-istio-envoy\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd\" (UID: \"1eeb2baa-0243-4032-94c4-30be54e2cef5\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd" Oct 30 09:16:20.425052 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:20.425050 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eeb2baa-0243-4032-94c4-30be54e2cef5-workload-certs\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd\" (UID: \"1eeb2baa-0243-4032-94c4-30be54e2cef5\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd" Oct 30 09:16:20.425456 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:20.425434 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eeb2baa-0243-4032-94c4-30be54e2cef5-workload-certs\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd\" (UID: \"1eeb2baa-0243-4032-94c4-30be54e2cef5\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd" Oct 30 09:16:20.425518 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:20.425434 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eeb2baa-0243-4032-94c4-30be54e2cef5-workload-socket\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd\" (UID: \"1eeb2baa-0243-4032-94c4-30be54e2cef5\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd" Oct 30 09:16:20.425726 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:20.425703 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eeb2baa-0243-4032-94c4-30be54e2cef5-credential-socket\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd\" (UID: \"1eeb2baa-0243-4032-94c4-30be54e2cef5\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd" Oct 30 09:16:20.426100 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:20.426073 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eeb2baa-0243-4032-94c4-30be54e2cef5-istiod-ca-cert\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd\" (UID: \"1eeb2baa-0243-4032-94c4-30be54e2cef5\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd" Oct 30 09:16:20.426773 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:20.426704 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eeb2baa-0243-4032-94c4-30be54e2cef5-istio-data\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd\" (UID: \"1eeb2baa-0243-4032-94c4-30be54e2cef5\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd" Oct 30 09:16:20.430530 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:20.430034 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-176b6c\"/\"openshift-service-ca.crt\"" Oct 30 09:16:20.430530 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:20.430164 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-176b6c\"/\"kube-root-ca.crt\"" Oct 30 09:16:20.430530 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:20.430275 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-176b6c\"/\"e2e-conflict-a-3231e37d-kserve-self-signed-certs\"" Oct 30 09:16:20.430530 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:20.430425 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-176b6c\"/\"default-dockercfg-z2c5q\"" Oct 30 09:16:20.433107 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:20.433074 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eeb2baa-0243-4032-94c4-30be54e2cef5-istio-envoy\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd\" (UID: \"1eeb2baa-0243-4032-94c4-30be54e2cef5\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd" Oct 30 09:16:20.433235 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:20.433216 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-176b6c/e2e-conflict-a-3231e37d-kserve-65b67bbd57-655l7"] Oct 30 09:16:20.450796 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:20.440994 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eeb2baa-0243-4032-94c4-30be54e2cef5-istio-podinfo\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd\" (UID: \"1eeb2baa-0243-4032-94c4-30be54e2cef5\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd" Oct 30 09:16:20.450796 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:20.446185 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8jkv\" (UniqueName: \"kubernetes.io/projected/1eeb2baa-0243-4032-94c4-30be54e2cef5-kube-api-access-x8jkv\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd\" (UID: \"1eeb2baa-0243-4032-94c4-30be54e2cef5\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd" Oct 30 09:16:20.450796 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:20.446195 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1eeb2baa-0243-4032-94c4-30be54e2cef5-istio-token\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd\" (UID: \"1eeb2baa-0243-4032-94c4-30be54e2cef5\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd" Oct 30 09:16:20.494488 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:20.494455 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd" Oct 30 09:16:20.525902 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:20.525857 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73629ad6-4888-4633-8dac-cf3aa331bca2-model-cache\") pod \"e2e-conflict-a-3231e37d-kserve-65b67bbd57-655l7\" (UID: \"73629ad6-4888-4633-8dac-cf3aa331bca2\") " pod="e2e-models-e2e-worker-w3-176b6c/e2e-conflict-a-3231e37d-kserve-65b67bbd57-655l7" Oct 30 09:16:20.526474 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:20.526026 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps92z\" (UniqueName: \"kubernetes.io/projected/73629ad6-4888-4633-8dac-cf3aa331bca2-kube-api-access-ps92z\") pod \"e2e-conflict-a-3231e37d-kserve-65b67bbd57-655l7\" (UID: \"73629ad6-4888-4633-8dac-cf3aa331bca2\") " pod="e2e-models-e2e-worker-w3-176b6c/e2e-conflict-a-3231e37d-kserve-65b67bbd57-655l7" Oct 30 09:16:20.526474 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:20.526064 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73629ad6-4888-4633-8dac-cf3aa331bca2-home\") pod \"e2e-conflict-a-3231e37d-kserve-65b67bbd57-655l7\" (UID: \"73629ad6-4888-4633-8dac-cf3aa331bca2\") " pod="e2e-models-e2e-worker-w3-176b6c/e2e-conflict-a-3231e37d-kserve-65b67bbd57-655l7" Oct 30 09:16:20.526474 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:20.526192 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73629ad6-4888-4633-8dac-cf3aa331bca2-dshm\") pod \"e2e-conflict-a-3231e37d-kserve-65b67bbd57-655l7\" (UID: \"73629ad6-4888-4633-8dac-cf3aa331bca2\") " pod="e2e-models-e2e-worker-w3-176b6c/e2e-conflict-a-3231e37d-kserve-65b67bbd57-655l7" Oct 30 09:16:20.526474 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:20.526255 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73629ad6-4888-4633-8dac-cf3aa331bca2-tmp-dir\") pod \"e2e-conflict-a-3231e37d-kserve-65b67bbd57-655l7\" (UID: \"73629ad6-4888-4633-8dac-cf3aa331bca2\") " pod="e2e-models-e2e-worker-w3-176b6c/e2e-conflict-a-3231e37d-kserve-65b67bbd57-655l7" Oct 30 09:16:20.526474 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:20.526395 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73629ad6-4888-4633-8dac-cf3aa331bca2-tls-certs\") pod \"e2e-conflict-a-3231e37d-kserve-65b67bbd57-655l7\" (UID: \"73629ad6-4888-4633-8dac-cf3aa331bca2\") " pod="e2e-models-e2e-worker-w3-176b6c/e2e-conflict-a-3231e37d-kserve-65b67bbd57-655l7" Oct 30 09:16:20.629948 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:20.629910 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73629ad6-4888-4633-8dac-cf3aa331bca2-tls-certs\") pod \"e2e-conflict-a-3231e37d-kserve-65b67bbd57-655l7\" (UID: \"73629ad6-4888-4633-8dac-cf3aa331bca2\") " pod="e2e-models-e2e-worker-w3-176b6c/e2e-conflict-a-3231e37d-kserve-65b67bbd57-655l7" Oct 30 09:16:20.630106 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:20.629980 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73629ad6-4888-4633-8dac-cf3aa331bca2-model-cache\") pod \"e2e-conflict-a-3231e37d-kserve-65b67bbd57-655l7\" (UID: \"73629ad6-4888-4633-8dac-cf3aa331bca2\") " pod="e2e-models-e2e-worker-w3-176b6c/e2e-conflict-a-3231e37d-kserve-65b67bbd57-655l7" Oct 30 09:16:20.630106 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:20.630024 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps92z\" (UniqueName: \"kubernetes.io/projected/73629ad6-4888-4633-8dac-cf3aa331bca2-kube-api-access-ps92z\") pod \"e2e-conflict-a-3231e37d-kserve-65b67bbd57-655l7\" (UID: \"73629ad6-4888-4633-8dac-cf3aa331bca2\") " pod="e2e-models-e2e-worker-w3-176b6c/e2e-conflict-a-3231e37d-kserve-65b67bbd57-655l7" Oct 30 09:16:20.630106 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:20.630045 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73629ad6-4888-4633-8dac-cf3aa331bca2-home\") pod \"e2e-conflict-a-3231e37d-kserve-65b67bbd57-655l7\" (UID: \"73629ad6-4888-4633-8dac-cf3aa331bca2\") " pod="e2e-models-e2e-worker-w3-176b6c/e2e-conflict-a-3231e37d-kserve-65b67bbd57-655l7" Oct 30 09:16:20.630106 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:20.630075 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73629ad6-4888-4633-8dac-cf3aa331bca2-dshm\") pod \"e2e-conflict-a-3231e37d-kserve-65b67bbd57-655l7\" (UID: \"73629ad6-4888-4633-8dac-cf3aa331bca2\") " pod="e2e-models-e2e-worker-w3-176b6c/e2e-conflict-a-3231e37d-kserve-65b67bbd57-655l7" Oct 30 09:16:20.630236 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:20.630108 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73629ad6-4888-4633-8dac-cf3aa331bca2-tmp-dir\") pod \"e2e-conflict-a-3231e37d-kserve-65b67bbd57-655l7\" (UID: \"73629ad6-4888-4633-8dac-cf3aa331bca2\") " pod="e2e-models-e2e-worker-w3-176b6c/e2e-conflict-a-3231e37d-kserve-65b67bbd57-655l7" Oct 30 09:16:20.630884 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:20.630525 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73629ad6-4888-4633-8dac-cf3aa331bca2-tmp-dir\") pod \"e2e-conflict-a-3231e37d-kserve-65b67bbd57-655l7\" (UID: \"73629ad6-4888-4633-8dac-cf3aa331bca2\") " pod="e2e-models-e2e-worker-w3-176b6c/e2e-conflict-a-3231e37d-kserve-65b67bbd57-655l7" Oct 30 09:16:20.630884 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:20.630686 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73629ad6-4888-4633-8dac-cf3aa331bca2-home\") pod \"e2e-conflict-a-3231e37d-kserve-65b67bbd57-655l7\" (UID: \"73629ad6-4888-4633-8dac-cf3aa331bca2\") " pod="e2e-models-e2e-worker-w3-176b6c/e2e-conflict-a-3231e37d-kserve-65b67bbd57-655l7" Oct 30 09:16:20.631153 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:20.631089 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73629ad6-4888-4633-8dac-cf3aa331bca2-model-cache\") pod \"e2e-conflict-a-3231e37d-kserve-65b67bbd57-655l7\" (UID: \"73629ad6-4888-4633-8dac-cf3aa331bca2\") " pod="e2e-models-e2e-worker-w3-176b6c/e2e-conflict-a-3231e37d-kserve-65b67bbd57-655l7" Oct 30 09:16:20.633853 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:20.633824 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73629ad6-4888-4633-8dac-cf3aa331bca2-tls-certs\") pod \"e2e-conflict-a-3231e37d-kserve-65b67bbd57-655l7\" (UID: \"73629ad6-4888-4633-8dac-cf3aa331bca2\") " pod="e2e-models-e2e-worker-w3-176b6c/e2e-conflict-a-3231e37d-kserve-65b67bbd57-655l7" Oct 30 09:16:20.634209 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:20.634190 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73629ad6-4888-4633-8dac-cf3aa331bca2-dshm\") pod \"e2e-conflict-a-3231e37d-kserve-65b67bbd57-655l7\" (UID: \"73629ad6-4888-4633-8dac-cf3aa331bca2\") " pod="e2e-models-e2e-worker-w3-176b6c/e2e-conflict-a-3231e37d-kserve-65b67bbd57-655l7" Oct 30 09:16:20.649891 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:20.649867 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps92z\" (UniqueName: \"kubernetes.io/projected/73629ad6-4888-4633-8dac-cf3aa331bca2-kube-api-access-ps92z\") pod \"e2e-conflict-a-3231e37d-kserve-65b67bbd57-655l7\" (UID: \"73629ad6-4888-4633-8dac-cf3aa331bca2\") " pod="e2e-models-e2e-worker-w3-176b6c/e2e-conflict-a-3231e37d-kserve-65b67bbd57-655l7" Oct 30 09:16:20.723153 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:20.723121 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-176b6c/e2e-conflict-a-3231e37d-kserve-65b67bbd57-655l7" Oct 30 09:16:20.804510 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:16:20.804479 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eeb2baa_0243_4032_94c4_30be54e2cef5.slice/crio-9b1fa4ade541dffb2b29f4afa998214806ab5748f1869488dd2c9e3af4bfd0fd WatchSource:0}: Error finding container 9b1fa4ade541dffb2b29f4afa998214806ab5748f1869488dd2c9e3af4bfd0fd: Status 404 returned error can't find the container with id 9b1fa4ade541dffb2b29f4afa998214806ab5748f1869488dd2c9e3af4bfd0fd Oct 30 09:16:20.804626 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:20.804564 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd"] Oct 30 09:16:20.807151 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:20.807120 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:16:20.807243 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:20.807186 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:16:20.807243 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:20.807213 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:16:21.018887 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:16:21.018853 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod73629ad6_4888_4633_8dac_cf3aa331bca2.slice/crio-a20759c8419664a13d062542efb9f531c9b11283427ed4b734d4fe3bb05fe3bd WatchSource:0}: Error finding container a20759c8419664a13d062542efb9f531c9b11283427ed4b734d4fe3bb05fe3bd: Status 404 returned error can't find the container with id a20759c8419664a13d062542efb9f531c9b11283427ed4b734d4fe3bb05fe3bd Oct 30 09:16:21.019214 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:21.019190 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-176b6c/e2e-conflict-a-3231e37d-kserve-65b67bbd57-655l7"] Oct 30 09:16:21.203885 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:21.203841 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd" event={"ID":"1eeb2baa-0243-4032-94c4-30be54e2cef5","Type":"ContainerStarted","Data":"e3951ff3cc2e9faa615cbacb394823ca303e6c5ef94243bffb871db278878cdd"} Oct 30 09:16:21.204359 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:21.203907 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd" event={"ID":"1eeb2baa-0243-4032-94c4-30be54e2cef5","Type":"ContainerStarted","Data":"9b1fa4ade541dffb2b29f4afa998214806ab5748f1869488dd2c9e3af4bfd0fd"} Oct 30 09:16:21.205529 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:21.205505 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-176b6c/e2e-conflict-a-3231e37d-kserve-65b67bbd57-655l7" event={"ID":"73629ad6-4888-4633-8dac-cf3aa331bca2","Type":"ContainerStarted","Data":"c0a3c70018d61f4a96de207cbb9dc3288661981129cab21ca88ca6f04c2b199d"} Oct 30 09:16:21.205618 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:21.205538 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-176b6c/e2e-conflict-a-3231e37d-kserve-65b67bbd57-655l7" event={"ID":"73629ad6-4888-4633-8dac-cf3aa331bca2","Type":"ContainerStarted","Data":"a20759c8419664a13d062542efb9f531c9b11283427ed4b734d4fe3bb05fe3bd"} Oct 30 09:16:21.300819 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:21.300711 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd" podStartSLOduration=1.300696758 podStartE2EDuration="1.300696758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:16:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:16:21.253333614 +0000 UTC m=+1341.514894622" watchObservedRunningTime="2026-10-30 09:16:21.300696758 +0000 UTC m=+1341.562257745" Oct 30 09:16:21.301143 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:21.301121 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-176b6c/e2e-conflict-a-3231e37d-kserve-65b67bbd57-655l7" podStartSLOduration=1.3011163319999999 podStartE2EDuration="1.301116332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:16:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:16:21.298418575 +0000 UTC m=+1341.559979580" watchObservedRunningTime="2026-10-30 09:16:21.301116332 +0000 UTC m=+1341.562677320" Oct 30 09:16:21.495458 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:21.495424 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd" Oct 30 09:16:21.497027 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:21.497006 4562 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 09:16:21.497128 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:21.497043 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd" podUID="1eeb2baa-0243-4032-94c4-30be54e2cef5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 09:16:21.723994 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:21.723954 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-176b6c/e2e-conflict-a-3231e37d-kserve-65b67bbd57-655l7" Oct 30 09:16:22.494856 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:22.494818 4562 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 09:16:22.495344 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:22.494880 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd" podUID="1eeb2baa-0243-4032-94c4-30be54e2cef5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 09:16:23.494854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:23.494817 4562 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 09:16:23.495022 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:23.494874 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd" podUID="1eeb2baa-0243-4032-94c4-30be54e2cef5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 09:16:24.495731 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:24.495696 4562 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 09:16:24.496114 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:24.495751 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd" podUID="1eeb2baa-0243-4032-94c4-30be54e2cef5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 09:16:25.495720 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:25.495683 4562 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 09:16:25.495897 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:25.495748 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd" podUID="1eeb2baa-0243-4032-94c4-30be54e2cef5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 09:16:26.495731 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:26.495696 4562 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 09:16:26.495911 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:26.495748 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd" podUID="1eeb2baa-0243-4032-94c4-30be54e2cef5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 09:16:27.500479 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:27.500448 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd" Oct 30 09:16:27.500952 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:27.500814 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd" Oct 30 09:16:27.501612 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:27.501589 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd" Oct 30 09:16:28.596583 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:16:28.596546 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:16:30.724215 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:30.724181 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-176b6c/e2e-conflict-a-3231e37d-kserve-65b67bbd57-655l7" Oct 30 09:16:30.736640 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:30.736614 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-176b6c/e2e-conflict-a-3231e37d-kserve-65b67bbd57-655l7" Oct 30 09:16:31.267013 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:31.266986 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-176b6c/e2e-conflict-a-3231e37d-kserve-65b67bbd57-655l7" Oct 30 09:16:32.237040 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:32.236997 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7"] Oct 30 09:16:32.243322 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:32.243298 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7" Oct 30 09:16:32.261241 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:32.261181 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7"] Oct 30 09:16:32.270533 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:32.270510 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-c5447d-openshift-default-dockercfg-nx55s\"" Oct 30 09:16:32.281965 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:32.281940 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj8dp\" (UniqueName: \"kubernetes.io/projected/6c176252-e0bb-44a8-93d9-4902b0afcb06-kube-api-access-cj8dp\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7\" (UID: \"6c176252-e0bb-44a8-93d9-4902b0afcb06\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7" Oct 30 09:16:32.282066 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:32.281974 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c176252-e0bb-44a8-93d9-4902b0afcb06-credential-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7\" (UID: \"6c176252-e0bb-44a8-93d9-4902b0afcb06\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7" Oct 30 09:16:32.282066 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:32.281989 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c176252-e0bb-44a8-93d9-4902b0afcb06-istio-token\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7\" (UID: \"6c176252-e0bb-44a8-93d9-4902b0afcb06\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7" Oct 30 09:16:32.282160 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:32.282126 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c176252-e0bb-44a8-93d9-4902b0afcb06-istiod-ca-cert\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7\" (UID: \"6c176252-e0bb-44a8-93d9-4902b0afcb06\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7" Oct 30 09:16:32.282205 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:32.282163 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c176252-e0bb-44a8-93d9-4902b0afcb06-istio-envoy\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7\" (UID: \"6c176252-e0bb-44a8-93d9-4902b0afcb06\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7" Oct 30 09:16:32.282242 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:32.282222 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c176252-e0bb-44a8-93d9-4902b0afcb06-istio-podinfo\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7\" (UID: \"6c176252-e0bb-44a8-93d9-4902b0afcb06\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7" Oct 30 09:16:32.282339 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:32.282314 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c176252-e0bb-44a8-93d9-4902b0afcb06-workload-certs\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7\" (UID: \"6c176252-e0bb-44a8-93d9-4902b0afcb06\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7" Oct 30 09:16:32.282486 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:32.282366 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c176252-e0bb-44a8-93d9-4902b0afcb06-workload-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7\" (UID: \"6c176252-e0bb-44a8-93d9-4902b0afcb06\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7" Oct 30 09:16:32.282486 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:32.282455 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c176252-e0bb-44a8-93d9-4902b0afcb06-istio-data\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7\" (UID: \"6c176252-e0bb-44a8-93d9-4902b0afcb06\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7" Oct 30 09:16:32.383603 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:32.383575 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c176252-e0bb-44a8-93d9-4902b0afcb06-workload-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7\" (UID: \"6c176252-e0bb-44a8-93d9-4902b0afcb06\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7" Oct 30 09:16:32.383747 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:32.383641 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c176252-e0bb-44a8-93d9-4902b0afcb06-istio-data\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7\" (UID: \"6c176252-e0bb-44a8-93d9-4902b0afcb06\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7" Oct 30 09:16:32.383747 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:32.383673 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj8dp\" (UniqueName: \"kubernetes.io/projected/6c176252-e0bb-44a8-93d9-4902b0afcb06-kube-api-access-cj8dp\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7\" (UID: \"6c176252-e0bb-44a8-93d9-4902b0afcb06\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7" Oct 30 09:16:32.383747 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:32.383694 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c176252-e0bb-44a8-93d9-4902b0afcb06-credential-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7\" (UID: \"6c176252-e0bb-44a8-93d9-4902b0afcb06\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7" Oct 30 09:16:32.383747 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:32.383710 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c176252-e0bb-44a8-93d9-4902b0afcb06-istio-token\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7\" (UID: \"6c176252-e0bb-44a8-93d9-4902b0afcb06\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7" Oct 30 09:16:32.383925 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:32.383768 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c176252-e0bb-44a8-93d9-4902b0afcb06-istiod-ca-cert\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7\" (UID: \"6c176252-e0bb-44a8-93d9-4902b0afcb06\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7" Oct 30 09:16:32.383925 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:32.383805 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c176252-e0bb-44a8-93d9-4902b0afcb06-istio-envoy\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7\" (UID: \"6c176252-e0bb-44a8-93d9-4902b0afcb06\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7" Oct 30 09:16:32.383925 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:32.383844 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c176252-e0bb-44a8-93d9-4902b0afcb06-istio-podinfo\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7\" (UID: \"6c176252-e0bb-44a8-93d9-4902b0afcb06\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7" Oct 30 09:16:32.383925 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:32.383895 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c176252-e0bb-44a8-93d9-4902b0afcb06-workload-certs\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7\" (UID: \"6c176252-e0bb-44a8-93d9-4902b0afcb06\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7" Oct 30 09:16:32.384130 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:32.384033 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c176252-e0bb-44a8-93d9-4902b0afcb06-workload-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7\" (UID: \"6c176252-e0bb-44a8-93d9-4902b0afcb06\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7" Oct 30 09:16:32.384130 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:32.384085 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c176252-e0bb-44a8-93d9-4902b0afcb06-credential-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7\" (UID: \"6c176252-e0bb-44a8-93d9-4902b0afcb06\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7" Oct 30 09:16:32.384230 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:32.384125 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c176252-e0bb-44a8-93d9-4902b0afcb06-istio-data\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7\" (UID: \"6c176252-e0bb-44a8-93d9-4902b0afcb06\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7" Oct 30 09:16:32.384230 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:32.384205 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c176252-e0bb-44a8-93d9-4902b0afcb06-workload-certs\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7\" (UID: \"6c176252-e0bb-44a8-93d9-4902b0afcb06\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7" Oct 30 09:16:32.384721 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:32.384699 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c176252-e0bb-44a8-93d9-4902b0afcb06-istiod-ca-cert\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7\" (UID: \"6c176252-e0bb-44a8-93d9-4902b0afcb06\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7" Oct 30 09:16:32.385977 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:32.385955 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c176252-e0bb-44a8-93d9-4902b0afcb06-istio-envoy\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7\" (UID: \"6c176252-e0bb-44a8-93d9-4902b0afcb06\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7" Oct 30 09:16:32.386225 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:32.386208 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c176252-e0bb-44a8-93d9-4902b0afcb06-istio-podinfo\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7\" (UID: \"6c176252-e0bb-44a8-93d9-4902b0afcb06\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7" Oct 30 09:16:32.403245 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:32.403219 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c176252-e0bb-44a8-93d9-4902b0afcb06-istio-token\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7\" (UID: \"6c176252-e0bb-44a8-93d9-4902b0afcb06\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7" Oct 30 09:16:32.403450 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:32.403433 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj8dp\" (UniqueName: \"kubernetes.io/projected/6c176252-e0bb-44a8-93d9-4902b0afcb06-kube-api-access-cj8dp\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7\" (UID: \"6c176252-e0bb-44a8-93d9-4902b0afcb06\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7" Oct 30 09:16:32.562597 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:32.562517 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7" Oct 30 09:16:32.868538 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:16:32.868513 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c176252_e0bb_44a8_93d9_4902b0afcb06.slice/crio-25b6ab0be2f6f577ea3af3998035dbdfacdffe4514a4335f91a618c00b3c4289 WatchSource:0}: Error finding container 25b6ab0be2f6f577ea3af3998035dbdfacdffe4514a4335f91a618c00b3c4289: Status 404 returned error can't find the container with id 25b6ab0be2f6f577ea3af3998035dbdfacdffe4514a4335f91a618c00b3c4289 Oct 30 09:16:32.868778 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:32.868756 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7"] Oct 30 09:16:32.870905 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:32.870878 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:16:32.870996 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:32.870933 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:16:32.870996 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:32.870965 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:16:33.273441 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:33.273400 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7" event={"ID":"6c176252-e0bb-44a8-93d9-4902b0afcb06","Type":"ContainerStarted","Data":"9bd6b4f47ac172758390f3f1e0c060358816211371a863c9bdfe060980f85c34"} Oct 30 09:16:33.273801 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:33.273463 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7" event={"ID":"6c176252-e0bb-44a8-93d9-4902b0afcb06","Type":"ContainerStarted","Data":"25b6ab0be2f6f577ea3af3998035dbdfacdffe4514a4335f91a618c00b3c4289"} Oct 30 09:16:33.343260 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:33.343209 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7" podStartSLOduration=1.343195242 podStartE2EDuration="1.343195242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:16:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:16:33.328483094 +0000 UTC m=+1353.590044107" watchObservedRunningTime="2026-10-30 09:16:33.343195242 +0000 UTC m=+1353.604756235" Oct 30 09:16:33.562700 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:33.562622 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7" Oct 30 09:16:33.564141 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:33.564114 4562 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 09:16:33.564253 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:33.564169 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7" podUID="6c176252-e0bb-44a8-93d9-4902b0afcb06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 09:16:34.566433 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:34.564738 4562 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 09:16:34.566433 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:34.564790 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7" podUID="6c176252-e0bb-44a8-93d9-4902b0afcb06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 09:16:35.563813 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:35.563771 4562 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 09:16:35.563993 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:35.563838 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7" podUID="6c176252-e0bb-44a8-93d9-4902b0afcb06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 09:16:36.563858 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:36.563821 4562 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 09:16:36.564263 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:36.563875 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7" podUID="6c176252-e0bb-44a8-93d9-4902b0afcb06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 09:16:37.563853 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:37.563814 4562 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 09:16:37.564027 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:37.563878 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7" podUID="6c176252-e0bb-44a8-93d9-4902b0afcb06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 09:16:38.563826 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:38.563779 4562 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 09:16:38.564003 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:38.563845 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7" podUID="6c176252-e0bb-44a8-93d9-4902b0afcb06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 09:16:39.563800 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:39.563763 4562 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 09:16:39.564185 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:39.563820 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7" podUID="6c176252-e0bb-44a8-93d9-4902b0afcb06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 09:16:40.600360 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:40.600323 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7" Oct 30 09:16:40.600805 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:40.600720 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7" Oct 30 09:16:40.602220 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:40.602194 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7" Oct 30 09:16:56.788324 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:56.788282 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-176b6c/e2e-conflict-b-3231e37d-kserve-7ff8996b5f-c7656"] Oct 30 09:16:56.792372 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:56.792353 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-176b6c/e2e-conflict-b-3231e37d-kserve-7ff8996b5f-c7656" Oct 30 09:16:56.812131 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:56.812101 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-176b6c/e2e-conflict-b-3231e37d-kserve-7ff8996b5f-c7656"] Oct 30 09:16:56.820545 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:56.820515 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-176b6c\"/\"e2e-conflict-b-3231e37d-kserve-self-signed-certs\"" Oct 30 09:16:56.901822 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:56.901784 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnz8b\" (UniqueName: \"kubernetes.io/projected/bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc-kube-api-access-mnz8b\") pod \"e2e-conflict-b-3231e37d-kserve-7ff8996b5f-c7656\" (UID: \"bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc\") " pod="e2e-models-e2e-worker-w3-176b6c/e2e-conflict-b-3231e37d-kserve-7ff8996b5f-c7656" Oct 30 09:16:56.902031 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:56.901936 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc-tmp-dir\") pod \"e2e-conflict-b-3231e37d-kserve-7ff8996b5f-c7656\" (UID: \"bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc\") " pod="e2e-models-e2e-worker-w3-176b6c/e2e-conflict-b-3231e37d-kserve-7ff8996b5f-c7656" Oct 30 09:16:56.902031 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:56.901972 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc-tls-certs\") pod \"e2e-conflict-b-3231e37d-kserve-7ff8996b5f-c7656\" (UID: \"bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc\") " pod="e2e-models-e2e-worker-w3-176b6c/e2e-conflict-b-3231e37d-kserve-7ff8996b5f-c7656" Oct 30 09:16:56.902127 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:56.902061 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc-home\") pod \"e2e-conflict-b-3231e37d-kserve-7ff8996b5f-c7656\" (UID: \"bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc\") " pod="e2e-models-e2e-worker-w3-176b6c/e2e-conflict-b-3231e37d-kserve-7ff8996b5f-c7656" Oct 30 09:16:56.902127 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:56.902103 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc-model-cache\") pod \"e2e-conflict-b-3231e37d-kserve-7ff8996b5f-c7656\" (UID: \"bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc\") " pod="e2e-models-e2e-worker-w3-176b6c/e2e-conflict-b-3231e37d-kserve-7ff8996b5f-c7656" Oct 30 09:16:56.902203 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:56.902185 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc-dshm\") pod \"e2e-conflict-b-3231e37d-kserve-7ff8996b5f-c7656\" (UID: \"bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc\") " pod="e2e-models-e2e-worker-w3-176b6c/e2e-conflict-b-3231e37d-kserve-7ff8996b5f-c7656" Oct 30 09:16:57.003395 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:57.003343 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc-model-cache\") pod \"e2e-conflict-b-3231e37d-kserve-7ff8996b5f-c7656\" (UID: \"bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc\") " pod="e2e-models-e2e-worker-w3-176b6c/e2e-conflict-b-3231e37d-kserve-7ff8996b5f-c7656" Oct 30 09:16:57.003574 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:57.003421 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc-dshm\") pod \"e2e-conflict-b-3231e37d-kserve-7ff8996b5f-c7656\" (UID: \"bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc\") " pod="e2e-models-e2e-worker-w3-176b6c/e2e-conflict-b-3231e37d-kserve-7ff8996b5f-c7656" Oct 30 09:16:57.003574 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:57.003540 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnz8b\" (UniqueName: \"kubernetes.io/projected/bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc-kube-api-access-mnz8b\") pod \"e2e-conflict-b-3231e37d-kserve-7ff8996b5f-c7656\" (UID: \"bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc\") " pod="e2e-models-e2e-worker-w3-176b6c/e2e-conflict-b-3231e37d-kserve-7ff8996b5f-c7656" Oct 30 09:16:57.003682 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:57.003649 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc-tmp-dir\") pod \"e2e-conflict-b-3231e37d-kserve-7ff8996b5f-c7656\" (UID: \"bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc\") " pod="e2e-models-e2e-worker-w3-176b6c/e2e-conflict-b-3231e37d-kserve-7ff8996b5f-c7656" Oct 30 09:16:57.003734 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:57.003683 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc-tls-certs\") pod \"e2e-conflict-b-3231e37d-kserve-7ff8996b5f-c7656\" (UID: \"bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc\") " pod="e2e-models-e2e-worker-w3-176b6c/e2e-conflict-b-3231e37d-kserve-7ff8996b5f-c7656" Oct 30 09:16:57.003734 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:57.003710 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc-home\") pod \"e2e-conflict-b-3231e37d-kserve-7ff8996b5f-c7656\" (UID: \"bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc\") " pod="e2e-models-e2e-worker-w3-176b6c/e2e-conflict-b-3231e37d-kserve-7ff8996b5f-c7656" Oct 30 09:16:57.003906 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:57.003828 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc-model-cache\") pod \"e2e-conflict-b-3231e37d-kserve-7ff8996b5f-c7656\" (UID: \"bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc\") " pod="e2e-models-e2e-worker-w3-176b6c/e2e-conflict-b-3231e37d-kserve-7ff8996b5f-c7656" Oct 30 09:16:57.003994 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:57.003977 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc-home\") pod \"e2e-conflict-b-3231e37d-kserve-7ff8996b5f-c7656\" (UID: \"bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc\") " pod="e2e-models-e2e-worker-w3-176b6c/e2e-conflict-b-3231e37d-kserve-7ff8996b5f-c7656" Oct 30 09:16:57.004320 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:57.004294 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc-tmp-dir\") pod \"e2e-conflict-b-3231e37d-kserve-7ff8996b5f-c7656\" (UID: \"bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc\") " pod="e2e-models-e2e-worker-w3-176b6c/e2e-conflict-b-3231e37d-kserve-7ff8996b5f-c7656" Oct 30 09:16:57.005760 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:57.005738 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc-dshm\") pod \"e2e-conflict-b-3231e37d-kserve-7ff8996b5f-c7656\" (UID: \"bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc\") " pod="e2e-models-e2e-worker-w3-176b6c/e2e-conflict-b-3231e37d-kserve-7ff8996b5f-c7656" Oct 30 09:16:57.006241 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:57.006220 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc-tls-certs\") pod \"e2e-conflict-b-3231e37d-kserve-7ff8996b5f-c7656\" (UID: \"bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc\") " pod="e2e-models-e2e-worker-w3-176b6c/e2e-conflict-b-3231e37d-kserve-7ff8996b5f-c7656" Oct 30 09:16:57.042980 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:57.042889 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnz8b\" (UniqueName: \"kubernetes.io/projected/bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc-kube-api-access-mnz8b\") pod \"e2e-conflict-b-3231e37d-kserve-7ff8996b5f-c7656\" (UID: \"bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc\") " pod="e2e-models-e2e-worker-w3-176b6c/e2e-conflict-b-3231e37d-kserve-7ff8996b5f-c7656" Oct 30 09:16:57.112051 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:57.112019 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-176b6c/e2e-conflict-b-3231e37d-kserve-7ff8996b5f-c7656" Oct 30 09:16:57.408474 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:16:57.408443 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbbc3b5b6_06d9_4eca_9a0e_febe1748e5dc.slice/crio-8edf1099c3388acb41762ceb1de95f911a1b8fe11c48c9068112217c7fcde031 WatchSource:0}: Error finding container 8edf1099c3388acb41762ceb1de95f911a1b8fe11c48c9068112217c7fcde031: Status 404 returned error can't find the container with id 8edf1099c3388acb41762ceb1de95f911a1b8fe11c48c9068112217c7fcde031 Oct 30 09:16:57.408998 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:57.408968 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-176b6c/e2e-conflict-b-3231e37d-kserve-7ff8996b5f-c7656"] Oct 30 09:16:58.405784 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:58.405745 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-176b6c/e2e-conflict-b-3231e37d-kserve-7ff8996b5f-c7656" event={"ID":"bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc","Type":"ContainerStarted","Data":"025bcd802b0124e72bf2e7a88095185d57876c710f153abc4af2e0c838e680df"} Oct 30 09:16:58.406190 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:58.405797 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-176b6c/e2e-conflict-b-3231e37d-kserve-7ff8996b5f-c7656" event={"ID":"bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc","Type":"ContainerStarted","Data":"8edf1099c3388acb41762ceb1de95f911a1b8fe11c48c9068112217c7fcde031"} Oct 30 09:16:58.446289 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:58.446223 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-176b6c/e2e-conflict-b-3231e37d-kserve-7ff8996b5f-c7656" podStartSLOduration=2.446204471 podStartE2EDuration="2.446204471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:16:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:16:58.44498116 +0000 UTC m=+1378.706542164" watchObservedRunningTime="2026-10-30 09:16:58.446204471 +0000 UTC m=+1378.707765460" Oct 30 09:16:58.615282 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:16:58.615242 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:16:59.112495 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:16:59.112462 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-176b6c/e2e-conflict-b-3231e37d-kserve-7ff8996b5f-c7656" Oct 30 09:17:07.112447 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:07.112411 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-176b6c/e2e-conflict-b-3231e37d-kserve-7ff8996b5f-c7656" Oct 30 09:17:07.124846 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:07.124820 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-176b6c/e2e-conflict-b-3231e37d-kserve-7ff8996b5f-c7656" Oct 30 09:17:07.491262 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:07.491235 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-176b6c/e2e-conflict-b-3231e37d-kserve-7ff8996b5f-c7656" Oct 30 09:17:24.778290 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:24.778192 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-c5447d/distinct-2-w1-c5447d-kserve-676749d548-xlmsf"] Oct 30 09:17:24.783048 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:24.783022 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c5447d/distinct-2-w1-c5447d-kserve-676749d548-xlmsf" Oct 30 09:17:24.796814 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:24.796783 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-c5447d/distinct-2-w1-c5447d-kserve-676749d548-xlmsf"] Oct 30 09:17:24.807245 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:24.807207 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-c5447d\"/\"kube-root-ca.crt\"" Oct 30 09:17:24.807396 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:24.807254 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-c5447d\"/\"openshift-service-ca.crt\"" Oct 30 09:17:24.807396 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:24.807259 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-c5447d\"/\"distinct-2-w1-c5447d-kserve-self-signed-certs\"" Oct 30 09:17:24.811526 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:24.811509 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-c5447d\"/\"default-dockercfg-9qf68\"" Oct 30 09:17:24.932882 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:24.932837 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9ff897f-c520-4d0f-8820-e185b1d47dac-tmp-dir\") pod \"distinct-2-w1-c5447d-kserve-676749d548-xlmsf\" (UID: \"c9ff897f-c520-4d0f-8820-e185b1d47dac\") " pod="e2e-models-e2e-worker-w1-c5447d/distinct-2-w1-c5447d-kserve-676749d548-xlmsf" Oct 30 09:17:24.933058 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:24.932977 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9ff897f-c520-4d0f-8820-e185b1d47dac-model-cache\") pod \"distinct-2-w1-c5447d-kserve-676749d548-xlmsf\" (UID: \"c9ff897f-c520-4d0f-8820-e185b1d47dac\") " pod="e2e-models-e2e-worker-w1-c5447d/distinct-2-w1-c5447d-kserve-676749d548-xlmsf" Oct 30 09:17:24.933199 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:24.933177 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9ff897f-c520-4d0f-8820-e185b1d47dac-home\") pod \"distinct-2-w1-c5447d-kserve-676749d548-xlmsf\" (UID: \"c9ff897f-c520-4d0f-8820-e185b1d47dac\") " pod="e2e-models-e2e-worker-w1-c5447d/distinct-2-w1-c5447d-kserve-676749d548-xlmsf" Oct 30 09:17:24.933273 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:24.933214 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9ff897f-c520-4d0f-8820-e185b1d47dac-tls-certs\") pod \"distinct-2-w1-c5447d-kserve-676749d548-xlmsf\" (UID: \"c9ff897f-c520-4d0f-8820-e185b1d47dac\") " pod="e2e-models-e2e-worker-w1-c5447d/distinct-2-w1-c5447d-kserve-676749d548-xlmsf" Oct 30 09:17:24.933341 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:24.933323 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9ff897f-c520-4d0f-8820-e185b1d47dac-dshm\") pod \"distinct-2-w1-c5447d-kserve-676749d548-xlmsf\" (UID: \"c9ff897f-c520-4d0f-8820-e185b1d47dac\") " pod="e2e-models-e2e-worker-w1-c5447d/distinct-2-w1-c5447d-kserve-676749d548-xlmsf" Oct 30 09:17:24.933427 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:24.933357 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrn25\" (UniqueName: \"kubernetes.io/projected/c9ff897f-c520-4d0f-8820-e185b1d47dac-kube-api-access-vrn25\") pod \"distinct-2-w1-c5447d-kserve-676749d548-xlmsf\" (UID: \"c9ff897f-c520-4d0f-8820-e185b1d47dac\") " pod="e2e-models-e2e-worker-w1-c5447d/distinct-2-w1-c5447d-kserve-676749d548-xlmsf" Oct 30 09:17:25.034427 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:25.034311 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9ff897f-c520-4d0f-8820-e185b1d47dac-tmp-dir\") pod \"distinct-2-w1-c5447d-kserve-676749d548-xlmsf\" (UID: \"c9ff897f-c520-4d0f-8820-e185b1d47dac\") " pod="e2e-models-e2e-worker-w1-c5447d/distinct-2-w1-c5447d-kserve-676749d548-xlmsf" Oct 30 09:17:25.034427 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:25.034400 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9ff897f-c520-4d0f-8820-e185b1d47dac-model-cache\") pod \"distinct-2-w1-c5447d-kserve-676749d548-xlmsf\" (UID: \"c9ff897f-c520-4d0f-8820-e185b1d47dac\") " pod="e2e-models-e2e-worker-w1-c5447d/distinct-2-w1-c5447d-kserve-676749d548-xlmsf" Oct 30 09:17:25.034667 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:25.034541 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9ff897f-c520-4d0f-8820-e185b1d47dac-home\") pod \"distinct-2-w1-c5447d-kserve-676749d548-xlmsf\" (UID: \"c9ff897f-c520-4d0f-8820-e185b1d47dac\") " pod="e2e-models-e2e-worker-w1-c5447d/distinct-2-w1-c5447d-kserve-676749d548-xlmsf" Oct 30 09:17:25.034667 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:25.034565 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9ff897f-c520-4d0f-8820-e185b1d47dac-tls-certs\") pod \"distinct-2-w1-c5447d-kserve-676749d548-xlmsf\" (UID: \"c9ff897f-c520-4d0f-8820-e185b1d47dac\") " pod="e2e-models-e2e-worker-w1-c5447d/distinct-2-w1-c5447d-kserve-676749d548-xlmsf" Oct 30 09:17:25.034667 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:25.034614 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9ff897f-c520-4d0f-8820-e185b1d47dac-dshm\") pod \"distinct-2-w1-c5447d-kserve-676749d548-xlmsf\" (UID: \"c9ff897f-c520-4d0f-8820-e185b1d47dac\") " pod="e2e-models-e2e-worker-w1-c5447d/distinct-2-w1-c5447d-kserve-676749d548-xlmsf" Oct 30 09:17:25.034667 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:25.034641 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrn25\" (UniqueName: \"kubernetes.io/projected/c9ff897f-c520-4d0f-8820-e185b1d47dac-kube-api-access-vrn25\") pod \"distinct-2-w1-c5447d-kserve-676749d548-xlmsf\" (UID: \"c9ff897f-c520-4d0f-8820-e185b1d47dac\") " pod="e2e-models-e2e-worker-w1-c5447d/distinct-2-w1-c5447d-kserve-676749d548-xlmsf" Oct 30 09:17:25.034903 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:25.034831 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9ff897f-c520-4d0f-8820-e185b1d47dac-model-cache\") pod \"distinct-2-w1-c5447d-kserve-676749d548-xlmsf\" (UID: \"c9ff897f-c520-4d0f-8820-e185b1d47dac\") " pod="e2e-models-e2e-worker-w1-c5447d/distinct-2-w1-c5447d-kserve-676749d548-xlmsf" Oct 30 09:17:25.034971 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:25.034951 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9ff897f-c520-4d0f-8820-e185b1d47dac-home\") pod \"distinct-2-w1-c5447d-kserve-676749d548-xlmsf\" (UID: \"c9ff897f-c520-4d0f-8820-e185b1d47dac\") " pod="e2e-models-e2e-worker-w1-c5447d/distinct-2-w1-c5447d-kserve-676749d548-xlmsf" Oct 30 09:17:25.035269 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:25.035229 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9ff897f-c520-4d0f-8820-e185b1d47dac-tmp-dir\") pod \"distinct-2-w1-c5447d-kserve-676749d548-xlmsf\" (UID: \"c9ff897f-c520-4d0f-8820-e185b1d47dac\") " pod="e2e-models-e2e-worker-w1-c5447d/distinct-2-w1-c5447d-kserve-676749d548-xlmsf" Oct 30 09:17:25.037551 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:25.037501 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9ff897f-c520-4d0f-8820-e185b1d47dac-dshm\") pod \"distinct-2-w1-c5447d-kserve-676749d548-xlmsf\" (UID: \"c9ff897f-c520-4d0f-8820-e185b1d47dac\") " pod="e2e-models-e2e-worker-w1-c5447d/distinct-2-w1-c5447d-kserve-676749d548-xlmsf" Oct 30 09:17:25.037812 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:25.037769 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9ff897f-c520-4d0f-8820-e185b1d47dac-tls-certs\") pod \"distinct-2-w1-c5447d-kserve-676749d548-xlmsf\" (UID: \"c9ff897f-c520-4d0f-8820-e185b1d47dac\") " pod="e2e-models-e2e-worker-w1-c5447d/distinct-2-w1-c5447d-kserve-676749d548-xlmsf" Oct 30 09:17:25.054438 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:25.054415 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrn25\" (UniqueName: \"kubernetes.io/projected/c9ff897f-c520-4d0f-8820-e185b1d47dac-kube-api-access-vrn25\") pod \"distinct-2-w1-c5447d-kserve-676749d548-xlmsf\" (UID: \"c9ff897f-c520-4d0f-8820-e185b1d47dac\") " pod="e2e-models-e2e-worker-w1-c5447d/distinct-2-w1-c5447d-kserve-676749d548-xlmsf" Oct 30 09:17:25.097949 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:25.097924 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c5447d/distinct-2-w1-c5447d-kserve-676749d548-xlmsf" Oct 30 09:17:25.443712 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:25.443678 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-c5447d/distinct-2-w1-c5447d-kserve-676749d548-xlmsf"] Oct 30 09:17:25.585765 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:25.585714 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c5447d/distinct-2-w1-c5447d-kserve-676749d548-xlmsf" event={"ID":"c9ff897f-c520-4d0f-8820-e185b1d47dac","Type":"ContainerStarted","Data":"7e3c502cda9d264cb042b3daa3853f5bb421e93190b324b1a7429fba1287a936"} Oct 30 09:17:25.585958 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:25.585775 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c5447d/distinct-2-w1-c5447d-kserve-676749d548-xlmsf" event={"ID":"c9ff897f-c520-4d0f-8820-e185b1d47dac","Type":"ContainerStarted","Data":"3da253e25b21f02545ef82e53cc6b4634ac1783ad3e5f397e6af1a58ffebb4a2"} Oct 30 09:17:25.629354 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:25.629292 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-c5447d/distinct-2-w1-c5447d-kserve-676749d548-xlmsf" podStartSLOduration=1.629280481 podStartE2EDuration="1.629280481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:17:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:17:25.628454892 +0000 UTC m=+1405.890015893" watchObservedRunningTime="2026-10-30 09:17:25.629280481 +0000 UTC m=+1405.890841468" Oct 30 09:17:26.098301 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:26.098265 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c5447d/distinct-2-w1-c5447d-kserve-676749d548-xlmsf" Oct 30 09:17:26.905629 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:26.905595 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-176b6c/e2e-conflict-b-3231e37d-kserve-7ff8996b5f-c7656"] Oct 30 09:17:28.595659 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:17:28.595624 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:17:35.098349 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:35.098318 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-c5447d/distinct-2-w1-c5447d-kserve-676749d548-xlmsf" Oct 30 09:17:35.110825 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:35.110800 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-c5447d/distinct-2-w1-c5447d-kserve-676749d548-xlmsf" Oct 30 09:17:35.646973 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:35.646947 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c5447d/distinct-2-w1-c5447d-kserve-676749d548-xlmsf" Oct 30 09:17:36.620475 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:36.620438 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-176b6c/e2e-conflict-a-3231e37d-kserve-65b67bbd57-655l7"] Oct 30 09:17:41.926061 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:41.926005 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-176b6c/e2e-conflict-b-3231e37d-kserve-7ff8996b5f-c7656" podUID="bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc" containerName="main" containerID="cri-o://025bcd802b0124e72bf2e7a88095185d57876c710f153abc4af2e0c838e680df" gracePeriod=45 Oct 30 09:17:42.189466 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:42.189446 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-176b6c/e2e-conflict-b-3231e37d-kserve-7ff8996b5f-c7656" Oct 30 09:17:42.255697 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:42.255666 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc-tmp-dir\") pod \"bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc\" (UID: \"bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc\") " Oct 30 09:17:42.255841 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:42.255746 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc-kube-api-access-mnz8b\" (UniqueName: \"kubernetes.io/projected/bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc-kube-api-access-mnz8b\") pod \"bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc\" (UID: \"bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc\") " Oct 30 09:17:42.255841 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:42.255795 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc-dshm\") pod \"bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc\" (UID: \"bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc\") " Oct 30 09:17:42.255841 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:42.255819 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc-model-cache\") pod \"bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc\" (UID: \"bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc\") " Oct 30 09:17:42.256053 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:42.255940 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc-tmp-dir" pod "bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc" (UID: "bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:17:42.256053 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:42.255950 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc-home\" (UniqueName: \"kubernetes.io/empty-dir/bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc-home\") pod \"bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc\" (UID: \"bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc\") " Oct 30 09:17:42.256187 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:42.256104 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc-model-cache" pod "bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc" (UID: "bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:17:42.256255 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:42.256181 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc-home" pod "bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc" (UID: "bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:17:42.256508 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:42.256465 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc-tls-certs\" (UniqueName: \"kubernetes.io/secret/bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc-tls-certs\") pod \"bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc\" (UID: \"bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc\") " Oct 30 09:17:42.256963 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:42.256937 4562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc-model-cache\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:17:42.256963 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:42.256963 4562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc-home\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:17:42.257127 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:42.256975 4562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc-tmp-dir\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:17:42.258329 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:42.258297 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc-kube-api-access-mnz8b" pod "bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc" (UID: "bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc"). InnerVolumeSpecName "kube-api-access-mnz8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:17:42.258458 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:42.258404 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc-dshm" pod "bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc" (UID: "bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:17:42.258742 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:42.258719 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc-tls-certs" pod "bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc" (UID: "bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:17:42.358023 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:42.357995 4562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc-tls-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:17:42.358023 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:42.358021 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnz8b\" (UniqueName: \"kubernetes.io/projected/bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc-kube-api-access-mnz8b\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:17:42.358196 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:42.358032 4562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc-dshm\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:17:42.692906 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:42.692871 4562 generic.go:356] "Generic (PLEG): container finished" podID="bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc" containerID="025bcd802b0124e72bf2e7a88095185d57876c710f153abc4af2e0c838e680df" exitCode=0 Oct 30 09:17:42.693077 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:42.692913 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-176b6c/e2e-conflict-b-3231e37d-kserve-7ff8996b5f-c7656" Oct 30 09:17:42.693077 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:42.692964 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-176b6c/e2e-conflict-b-3231e37d-kserve-7ff8996b5f-c7656" event={"ID":"bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc","Type":"ContainerDied","Data":"025bcd802b0124e72bf2e7a88095185d57876c710f153abc4af2e0c838e680df"} Oct 30 09:17:42.693077 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:42.693004 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-176b6c/e2e-conflict-b-3231e37d-kserve-7ff8996b5f-c7656" event={"ID":"bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc","Type":"ContainerDied","Data":"8edf1099c3388acb41762ceb1de95f911a1b8fe11c48c9068112217c7fcde031"} Oct 30 09:17:42.693077 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:42.693024 4562 scope.go:122] "RemoveContainer" containerID="025bcd802b0124e72bf2e7a88095185d57876c710f153abc4af2e0c838e680df" Oct 30 09:17:42.703201 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:42.703179 4562 scope.go:122] "RemoveContainer" containerID="025bcd802b0124e72bf2e7a88095185d57876c710f153abc4af2e0c838e680df" Oct 30 09:17:42.703553 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:17:42.703530 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"025bcd802b0124e72bf2e7a88095185d57876c710f153abc4af2e0c838e680df\": container with ID starting with 025bcd802b0124e72bf2e7a88095185d57876c710f153abc4af2e0c838e680df not found: ID does not exist" containerID="025bcd802b0124e72bf2e7a88095185d57876c710f153abc4af2e0c838e680df" Oct 30 09:17:42.703657 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:42.703560 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"025bcd802b0124e72bf2e7a88095185d57876c710f153abc4af2e0c838e680df"} err="failed to get container status \"025bcd802b0124e72bf2e7a88095185d57876c710f153abc4af2e0c838e680df\": rpc error: code = NotFound desc = could not find container \"025bcd802b0124e72bf2e7a88095185d57876c710f153abc4af2e0c838e680df\": container with ID starting with 025bcd802b0124e72bf2e7a88095185d57876c710f153abc4af2e0c838e680df not found: ID does not exist" Oct 30 09:17:42.754175 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:42.754148 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-176b6c/e2e-conflict-b-3231e37d-kserve-7ff8996b5f-c7656"] Oct 30 09:17:42.758113 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:42.758094 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-176b6c/e2e-conflict-b-3231e37d-kserve-7ff8996b5f-c7656"] Oct 30 09:17:43.873978 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:43.873943 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc" path="/var/lib/kubelet/pods/bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc/volumes" Oct 30 09:17:51.641582 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:51.641511 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-176b6c/e2e-conflict-a-3231e37d-kserve-65b67bbd57-655l7" podUID="73629ad6-4888-4633-8dac-cf3aa331bca2" containerName="main" containerID="cri-o://c0a3c70018d61f4a96de207cbb9dc3288661981129cab21ca88ca6f04c2b199d" gracePeriod=45 Oct 30 09:17:52.027305 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:52.027280 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-176b6c/e2e-conflict-a-3231e37d-kserve-65b67bbd57-655l7" Oct 30 09:17:52.077171 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:52.077137 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73629ad6-4888-4633-8dac-cf3aa331bca2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/73629ad6-4888-4633-8dac-cf3aa331bca2-dshm\") pod \"73629ad6-4888-4633-8dac-cf3aa331bca2\" (UID: \"73629ad6-4888-4633-8dac-cf3aa331bca2\") " Oct 30 09:17:52.077452 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:52.077429 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73629ad6-4888-4633-8dac-cf3aa331bca2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73629ad6-4888-4633-8dac-cf3aa331bca2-tmp-dir\") pod \"73629ad6-4888-4633-8dac-cf3aa331bca2\" (UID: \"73629ad6-4888-4633-8dac-cf3aa331bca2\") " Oct 30 09:17:52.077608 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:52.077592 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73629ad6-4888-4633-8dac-cf3aa331bca2-kube-api-access-ps92z\" (UniqueName: \"kubernetes.io/projected/73629ad6-4888-4633-8dac-cf3aa331bca2-kube-api-access-ps92z\") pod \"73629ad6-4888-4633-8dac-cf3aa331bca2\" (UID: \"73629ad6-4888-4633-8dac-cf3aa331bca2\") " Oct 30 09:17:52.077864 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:52.077845 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73629ad6-4888-4633-8dac-cf3aa331bca2-home\" (UniqueName: \"kubernetes.io/empty-dir/73629ad6-4888-4633-8dac-cf3aa331bca2-home\") pod \"73629ad6-4888-4633-8dac-cf3aa331bca2\" (UID: \"73629ad6-4888-4633-8dac-cf3aa331bca2\") " Oct 30 09:17:52.078051 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:52.078033 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73629ad6-4888-4633-8dac-cf3aa331bca2-tls-certs\" (UniqueName: \"kubernetes.io/secret/73629ad6-4888-4633-8dac-cf3aa331bca2-tls-certs\") pod \"73629ad6-4888-4633-8dac-cf3aa331bca2\" (UID: \"73629ad6-4888-4633-8dac-cf3aa331bca2\") " Oct 30 09:17:52.078181 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:52.078165 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73629ad6-4888-4633-8dac-cf3aa331bca2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73629ad6-4888-4633-8dac-cf3aa331bca2-model-cache\") pod \"73629ad6-4888-4633-8dac-cf3aa331bca2\" (UID: \"73629ad6-4888-4633-8dac-cf3aa331bca2\") " Oct 30 09:17:52.078894 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:52.078871 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73629ad6-4888-4633-8dac-cf3aa331bca2-model-cache" pod "73629ad6-4888-4633-8dac-cf3aa331bca2" (UID: "73629ad6-4888-4633-8dac-cf3aa331bca2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:17:52.079200 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:52.079181 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73629ad6-4888-4633-8dac-cf3aa331bca2-tmp-dir" pod "73629ad6-4888-4633-8dac-cf3aa331bca2" (UID: "73629ad6-4888-4633-8dac-cf3aa331bca2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:17:52.081109 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:52.081082 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73629ad6-4888-4633-8dac-cf3aa331bca2-home" pod "73629ad6-4888-4633-8dac-cf3aa331bca2" (UID: "73629ad6-4888-4633-8dac-cf3aa331bca2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:17:52.095602 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:52.095580 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73629ad6-4888-4633-8dac-cf3aa331bca2-kube-api-access-ps92z" pod "73629ad6-4888-4633-8dac-cf3aa331bca2" (UID: "73629ad6-4888-4633-8dac-cf3aa331bca2"). InnerVolumeSpecName "kube-api-access-ps92z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:17:52.104508 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:52.104462 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73629ad6-4888-4633-8dac-cf3aa331bca2-dshm" pod "73629ad6-4888-4633-8dac-cf3aa331bca2" (UID: "73629ad6-4888-4633-8dac-cf3aa331bca2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:17:52.105035 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:52.104999 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73629ad6-4888-4633-8dac-cf3aa331bca2-tls-certs" pod "73629ad6-4888-4633-8dac-cf3aa331bca2" (UID: "73629ad6-4888-4633-8dac-cf3aa331bca2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:17:52.180020 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:52.179989 4562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73629ad6-4888-4633-8dac-cf3aa331bca2-home\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:17:52.180020 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:52.180014 4562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73629ad6-4888-4633-8dac-cf3aa331bca2-tls-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:17:52.180020 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:52.180025 4562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73629ad6-4888-4633-8dac-cf3aa331bca2-model-cache\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:17:52.180232 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:52.180034 4562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73629ad6-4888-4633-8dac-cf3aa331bca2-dshm\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:17:52.180232 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:52.180042 4562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73629ad6-4888-4633-8dac-cf3aa331bca2-tmp-dir\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:17:52.180232 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:52.180054 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ps92z\" (UniqueName: \"kubernetes.io/projected/73629ad6-4888-4633-8dac-cf3aa331bca2-kube-api-access-ps92z\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:17:52.785580 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:52.785544 4562 generic.go:356] "Generic (PLEG): container finished" podID="73629ad6-4888-4633-8dac-cf3aa331bca2" containerID="c0a3c70018d61f4a96de207cbb9dc3288661981129cab21ca88ca6f04c2b199d" exitCode=0 Oct 30 09:17:52.786022 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:52.785623 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-176b6c/e2e-conflict-a-3231e37d-kserve-65b67bbd57-655l7" event={"ID":"73629ad6-4888-4633-8dac-cf3aa331bca2","Type":"ContainerDied","Data":"c0a3c70018d61f4a96de207cbb9dc3288661981129cab21ca88ca6f04c2b199d"} Oct 30 09:17:52.786022 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:52.785660 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-176b6c/e2e-conflict-a-3231e37d-kserve-65b67bbd57-655l7" event={"ID":"73629ad6-4888-4633-8dac-cf3aa331bca2","Type":"ContainerDied","Data":"a20759c8419664a13d062542efb9f531c9b11283427ed4b734d4fe3bb05fe3bd"} Oct 30 09:17:52.786022 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:52.785684 4562 scope.go:122] "RemoveContainer" containerID="c0a3c70018d61f4a96de207cbb9dc3288661981129cab21ca88ca6f04c2b199d" Oct 30 09:17:52.786022 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:52.785839 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-176b6c/e2e-conflict-a-3231e37d-kserve-65b67bbd57-655l7" Oct 30 09:17:52.797750 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:52.797731 4562 scope.go:122] "RemoveContainer" containerID="c0a3c70018d61f4a96de207cbb9dc3288661981129cab21ca88ca6f04c2b199d" Oct 30 09:17:52.798057 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:17:52.798038 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0a3c70018d61f4a96de207cbb9dc3288661981129cab21ca88ca6f04c2b199d\": container with ID starting with c0a3c70018d61f4a96de207cbb9dc3288661981129cab21ca88ca6f04c2b199d not found: ID does not exist" containerID="c0a3c70018d61f4a96de207cbb9dc3288661981129cab21ca88ca6f04c2b199d" Oct 30 09:17:52.798157 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:52.798060 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0a3c70018d61f4a96de207cbb9dc3288661981129cab21ca88ca6f04c2b199d"} err="failed to get container status \"c0a3c70018d61f4a96de207cbb9dc3288661981129cab21ca88ca6f04c2b199d\": rpc error: code = NotFound desc = could not find container \"c0a3c70018d61f4a96de207cbb9dc3288661981129cab21ca88ca6f04c2b199d\": container with ID starting with c0a3c70018d61f4a96de207cbb9dc3288661981129cab21ca88ca6f04c2b199d not found: ID does not exist" Oct 30 09:17:52.840483 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:52.840423 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-176b6c/e2e-conflict-a-3231e37d-kserve-65b67bbd57-655l7"] Oct 30 09:17:52.854220 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:52.854183 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-176b6c/e2e-conflict-a-3231e37d-kserve-65b67bbd57-655l7"] Oct 30 09:17:53.150349 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:53.150273 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-176b6c/distinct-w3-176b6c-kserve-f57b7579c-k4pc5"] Oct 30 09:17:53.150814 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:53.150800 4562 cpu_manager.go:414] "containerMap: removing container" podUID="73629ad6-4888-4633-8dac-cf3aa331bca2" containerName="main" Oct 30 09:17:53.150867 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:53.150815 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73629ad6-4888-4633-8dac-cf3aa331bca2" containerName="main" Oct 30 09:17:53.150867 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:53.150834 4562 cpu_manager.go:414] "containerMap: removing container" podUID="bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc" containerName="main" Oct 30 09:17:53.150867 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:53.150838 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc" containerName="main" Oct 30 09:17:53.151104 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:53.150915 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="73629ad6-4888-4633-8dac-cf3aa331bca2" containerName="main" Oct 30 09:17:53.151104 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:53.150929 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbc3b5b6-06d9-4eca-9a0e-febe1748e5dc" containerName="main" Oct 30 09:17:53.155946 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:53.155922 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-176b6c/distinct-w3-176b6c-kserve-f57b7579c-k4pc5" Oct 30 09:17:53.175861 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:53.175832 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-176b6c/distinct-w3-176b6c-kserve-f57b7579c-k4pc5"] Oct 30 09:17:53.185025 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:53.184980 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-176b6c\"/\"default-dockercfg-z2c5q\"" Oct 30 09:17:53.185341 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:53.184999 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-176b6c\"/\"kube-root-ca.crt\"" Oct 30 09:17:53.185341 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:53.185276 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-176b6c\"/\"openshift-service-ca.crt\"" Oct 30 09:17:53.185519 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:53.185325 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-176b6c\"/\"distinct-w3-176b6c-kserve-self-signed-certs\"" Oct 30 09:17:53.189832 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:53.189810 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28fd4072-399c-4be5-a0d8-1be6966da4af-home\") pod \"distinct-w3-176b6c-kserve-f57b7579c-k4pc5\" (UID: \"28fd4072-399c-4be5-a0d8-1be6966da4af\") " pod="e2e-models-e2e-worker-w3-176b6c/distinct-w3-176b6c-kserve-f57b7579c-k4pc5" Oct 30 09:17:53.189930 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:53.189859 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28fd4072-399c-4be5-a0d8-1be6966da4af-tmp-dir\") pod \"distinct-w3-176b6c-kserve-f57b7579c-k4pc5\" (UID: \"28fd4072-399c-4be5-a0d8-1be6966da4af\") " pod="e2e-models-e2e-worker-w3-176b6c/distinct-w3-176b6c-kserve-f57b7579c-k4pc5" Oct 30 09:17:53.190019 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:53.190002 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28fd4072-399c-4be5-a0d8-1be6966da4af-dshm\") pod \"distinct-w3-176b6c-kserve-f57b7579c-k4pc5\" (UID: \"28fd4072-399c-4be5-a0d8-1be6966da4af\") " pod="e2e-models-e2e-worker-w3-176b6c/distinct-w3-176b6c-kserve-f57b7579c-k4pc5" Oct 30 09:17:53.190066 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:53.190049 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28fd4072-399c-4be5-a0d8-1be6966da4af-tls-certs\") pod \"distinct-w3-176b6c-kserve-f57b7579c-k4pc5\" (UID: \"28fd4072-399c-4be5-a0d8-1be6966da4af\") " pod="e2e-models-e2e-worker-w3-176b6c/distinct-w3-176b6c-kserve-f57b7579c-k4pc5" Oct 30 09:17:53.190186 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:53.190170 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjfdp\" (UniqueName: \"kubernetes.io/projected/28fd4072-399c-4be5-a0d8-1be6966da4af-kube-api-access-vjfdp\") pod \"distinct-w3-176b6c-kserve-f57b7579c-k4pc5\" (UID: \"28fd4072-399c-4be5-a0d8-1be6966da4af\") " pod="e2e-models-e2e-worker-w3-176b6c/distinct-w3-176b6c-kserve-f57b7579c-k4pc5" Oct 30 09:17:53.190227 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:53.190207 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28fd4072-399c-4be5-a0d8-1be6966da4af-model-cache\") pod \"distinct-w3-176b6c-kserve-f57b7579c-k4pc5\" (UID: \"28fd4072-399c-4be5-a0d8-1be6966da4af\") " pod="e2e-models-e2e-worker-w3-176b6c/distinct-w3-176b6c-kserve-f57b7579c-k4pc5" Oct 30 09:17:53.291625 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:53.291590 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28fd4072-399c-4be5-a0d8-1be6966da4af-tls-certs\") pod \"distinct-w3-176b6c-kserve-f57b7579c-k4pc5\" (UID: \"28fd4072-399c-4be5-a0d8-1be6966da4af\") " pod="e2e-models-e2e-worker-w3-176b6c/distinct-w3-176b6c-kserve-f57b7579c-k4pc5" Oct 30 09:17:53.291813 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:53.291667 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjfdp\" (UniqueName: \"kubernetes.io/projected/28fd4072-399c-4be5-a0d8-1be6966da4af-kube-api-access-vjfdp\") pod \"distinct-w3-176b6c-kserve-f57b7579c-k4pc5\" (UID: \"28fd4072-399c-4be5-a0d8-1be6966da4af\") " pod="e2e-models-e2e-worker-w3-176b6c/distinct-w3-176b6c-kserve-f57b7579c-k4pc5" Oct 30 09:17:53.291813 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:53.291702 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28fd4072-399c-4be5-a0d8-1be6966da4af-model-cache\") pod \"distinct-w3-176b6c-kserve-f57b7579c-k4pc5\" (UID: \"28fd4072-399c-4be5-a0d8-1be6966da4af\") " pod="e2e-models-e2e-worker-w3-176b6c/distinct-w3-176b6c-kserve-f57b7579c-k4pc5" Oct 30 09:17:53.291813 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:53.291742 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28fd4072-399c-4be5-a0d8-1be6966da4af-home\") pod \"distinct-w3-176b6c-kserve-f57b7579c-k4pc5\" (UID: \"28fd4072-399c-4be5-a0d8-1be6966da4af\") " pod="e2e-models-e2e-worker-w3-176b6c/distinct-w3-176b6c-kserve-f57b7579c-k4pc5" Oct 30 09:17:53.291987 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:53.291882 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28fd4072-399c-4be5-a0d8-1be6966da4af-tmp-dir\") pod \"distinct-w3-176b6c-kserve-f57b7579c-k4pc5\" (UID: \"28fd4072-399c-4be5-a0d8-1be6966da4af\") " pod="e2e-models-e2e-worker-w3-176b6c/distinct-w3-176b6c-kserve-f57b7579c-k4pc5" Oct 30 09:17:53.291987 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:53.291962 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28fd4072-399c-4be5-a0d8-1be6966da4af-dshm\") pod \"distinct-w3-176b6c-kserve-f57b7579c-k4pc5\" (UID: \"28fd4072-399c-4be5-a0d8-1be6966da4af\") " pod="e2e-models-e2e-worker-w3-176b6c/distinct-w3-176b6c-kserve-f57b7579c-k4pc5" Oct 30 09:17:53.292235 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:53.292202 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28fd4072-399c-4be5-a0d8-1be6966da4af-home\") pod \"distinct-w3-176b6c-kserve-f57b7579c-k4pc5\" (UID: \"28fd4072-399c-4be5-a0d8-1be6966da4af\") " pod="e2e-models-e2e-worker-w3-176b6c/distinct-w3-176b6c-kserve-f57b7579c-k4pc5" Oct 30 09:17:53.292235 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:53.292228 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28fd4072-399c-4be5-a0d8-1be6966da4af-model-cache\") pod \"distinct-w3-176b6c-kserve-f57b7579c-k4pc5\" (UID: \"28fd4072-399c-4be5-a0d8-1be6966da4af\") " pod="e2e-models-e2e-worker-w3-176b6c/distinct-w3-176b6c-kserve-f57b7579c-k4pc5" Oct 30 09:17:53.292406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:53.292270 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28fd4072-399c-4be5-a0d8-1be6966da4af-tmp-dir\") pod \"distinct-w3-176b6c-kserve-f57b7579c-k4pc5\" (UID: \"28fd4072-399c-4be5-a0d8-1be6966da4af\") " pod="e2e-models-e2e-worker-w3-176b6c/distinct-w3-176b6c-kserve-f57b7579c-k4pc5" Oct 30 09:17:53.293978 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:53.293957 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28fd4072-399c-4be5-a0d8-1be6966da4af-dshm\") pod \"distinct-w3-176b6c-kserve-f57b7579c-k4pc5\" (UID: \"28fd4072-399c-4be5-a0d8-1be6966da4af\") " pod="e2e-models-e2e-worker-w3-176b6c/distinct-w3-176b6c-kserve-f57b7579c-k4pc5" Oct 30 09:17:53.294394 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:53.294361 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28fd4072-399c-4be5-a0d8-1be6966da4af-tls-certs\") pod \"distinct-w3-176b6c-kserve-f57b7579c-k4pc5\" (UID: \"28fd4072-399c-4be5-a0d8-1be6966da4af\") " pod="e2e-models-e2e-worker-w3-176b6c/distinct-w3-176b6c-kserve-f57b7579c-k4pc5" Oct 30 09:17:53.310785 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:53.310762 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjfdp\" (UniqueName: \"kubernetes.io/projected/28fd4072-399c-4be5-a0d8-1be6966da4af-kube-api-access-vjfdp\") pod \"distinct-w3-176b6c-kserve-f57b7579c-k4pc5\" (UID: \"28fd4072-399c-4be5-a0d8-1be6966da4af\") " pod="e2e-models-e2e-worker-w3-176b6c/distinct-w3-176b6c-kserve-f57b7579c-k4pc5" Oct 30 09:17:53.468582 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:53.468550 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-176b6c/distinct-w3-176b6c-kserve-f57b7579c-k4pc5" Oct 30 09:17:53.760924 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:17:53.760883 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod28fd4072_399c_4be5_a0d8_1be6966da4af.slice/crio-f13dfee49c3cfd70cd8f918bf692c1d364589bd30d60b61eebf432420c19bfd5 WatchSource:0}: Error finding container f13dfee49c3cfd70cd8f918bf692c1d364589bd30d60b61eebf432420c19bfd5: Status 404 returned error can't find the container with id f13dfee49c3cfd70cd8f918bf692c1d364589bd30d60b61eebf432420c19bfd5 Oct 30 09:17:53.761614 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:53.761589 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-176b6c/distinct-w3-176b6c-kserve-f57b7579c-k4pc5"] Oct 30 09:17:53.792436 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:53.792409 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-176b6c/distinct-w3-176b6c-kserve-f57b7579c-k4pc5" event={"ID":"28fd4072-399c-4be5-a0d8-1be6966da4af","Type":"ContainerStarted","Data":"f13dfee49c3cfd70cd8f918bf692c1d364589bd30d60b61eebf432420c19bfd5"} Oct 30 09:17:53.874186 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:53.874157 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73629ad6-4888-4633-8dac-cf3aa331bca2" path="/var/lib/kubelet/pods/73629ad6-4888-4633-8dac-cf3aa331bca2/volumes" Oct 30 09:17:54.800034 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:54.799998 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-176b6c/distinct-w3-176b6c-kserve-f57b7579c-k4pc5" event={"ID":"28fd4072-399c-4be5-a0d8-1be6966da4af","Type":"ContainerStarted","Data":"b376c572007906a2386d0e29cc622df795f0467d733dcda1cf97611cd380e422"} Oct 30 09:17:54.840976 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:54.840912 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-176b6c/distinct-w3-176b6c-kserve-f57b7579c-k4pc5" podStartSLOduration=1.840896509 podStartE2EDuration="1.840896509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:17:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:17:54.840464275 +0000 UTC m=+1435.102025275" watchObservedRunningTime="2026-10-30 09:17:54.840896509 +0000 UTC m=+1435.102457500" Oct 30 09:17:55.469355 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:55.469326 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-176b6c/distinct-w3-176b6c-kserve-f57b7579c-k4pc5" Oct 30 09:17:56.111419 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:56.111368 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-2f69c8/distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb"] Oct 30 09:17:56.116527 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:56.116498 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb" Oct 30 09:17:56.132780 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:56.132754 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2f69c8/distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb"] Oct 30 09:17:56.142855 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:56.142824 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-2f69c8\"/\"openshift-service-ca.crt\"" Oct 30 09:17:56.142855 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:56.142831 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-2f69c8\"/\"kube-root-ca.crt\"" Oct 30 09:17:56.143016 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:56.142880 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2f69c8\"/\"default-dockercfg-lh8pz\"" Oct 30 09:17:56.146977 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:56.146957 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2f69c8\"/\"distinct-2-w0-2f69c8-kserve-self-signed-certs\"" Oct 30 09:17:56.227250 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:56.227218 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg69t\" (UniqueName: \"kubernetes.io/projected/a540c86f-c856-42a8-9f73-b5b9b690f42b-kube-api-access-tg69t\") pod \"distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb\" (UID: \"a540c86f-c856-42a8-9f73-b5b9b690f42b\") " pod="e2e-models-e2e-worker-w0-2f69c8/distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb" Oct 30 09:17:56.227250 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:56.227252 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a540c86f-c856-42a8-9f73-b5b9b690f42b-home\") pod \"distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb\" (UID: \"a540c86f-c856-42a8-9f73-b5b9b690f42b\") " pod="e2e-models-e2e-worker-w0-2f69c8/distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb" Oct 30 09:17:56.227492 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:56.227280 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a540c86f-c856-42a8-9f73-b5b9b690f42b-tmp-dir\") pod \"distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb\" (UID: \"a540c86f-c856-42a8-9f73-b5b9b690f42b\") " pod="e2e-models-e2e-worker-w0-2f69c8/distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb" Oct 30 09:17:56.227492 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:56.227419 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a540c86f-c856-42a8-9f73-b5b9b690f42b-dshm\") pod \"distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb\" (UID: \"a540c86f-c856-42a8-9f73-b5b9b690f42b\") " pod="e2e-models-e2e-worker-w0-2f69c8/distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb" Oct 30 09:17:56.227492 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:56.227462 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a540c86f-c856-42a8-9f73-b5b9b690f42b-tls-certs\") pod \"distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb\" (UID: \"a540c86f-c856-42a8-9f73-b5b9b690f42b\") " pod="e2e-models-e2e-worker-w0-2f69c8/distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb" Oct 30 09:17:56.227594 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:56.227567 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a540c86f-c856-42a8-9f73-b5b9b690f42b-model-cache\") pod \"distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb\" (UID: \"a540c86f-c856-42a8-9f73-b5b9b690f42b\") " pod="e2e-models-e2e-worker-w0-2f69c8/distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb" Oct 30 09:17:56.329280 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:56.329240 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg69t\" (UniqueName: \"kubernetes.io/projected/a540c86f-c856-42a8-9f73-b5b9b690f42b-kube-api-access-tg69t\") pod \"distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb\" (UID: \"a540c86f-c856-42a8-9f73-b5b9b690f42b\") " pod="e2e-models-e2e-worker-w0-2f69c8/distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb" Oct 30 09:17:56.329961 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:56.329929 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a540c86f-c856-42a8-9f73-b5b9b690f42b-home\") pod \"distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb\" (UID: \"a540c86f-c856-42a8-9f73-b5b9b690f42b\") " pod="e2e-models-e2e-worker-w0-2f69c8/distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb" Oct 30 09:17:56.330636 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:56.330616 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a540c86f-c856-42a8-9f73-b5b9b690f42b-tmp-dir\") pod \"distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb\" (UID: \"a540c86f-c856-42a8-9f73-b5b9b690f42b\") " pod="e2e-models-e2e-worker-w0-2f69c8/distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb" Oct 30 09:17:56.331149 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:56.331131 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a540c86f-c856-42a8-9f73-b5b9b690f42b-dshm\") pod \"distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb\" (UID: \"a540c86f-c856-42a8-9f73-b5b9b690f42b\") " pod="e2e-models-e2e-worker-w0-2f69c8/distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb" Oct 30 09:17:56.331667 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:56.331641 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a540c86f-c856-42a8-9f73-b5b9b690f42b-tls-certs\") pod \"distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb\" (UID: \"a540c86f-c856-42a8-9f73-b5b9b690f42b\") " pod="e2e-models-e2e-worker-w0-2f69c8/distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb" Oct 30 09:17:56.332324 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:56.332304 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a540c86f-c856-42a8-9f73-b5b9b690f42b-model-cache\") pod \"distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb\" (UID: \"a540c86f-c856-42a8-9f73-b5b9b690f42b\") " pod="e2e-models-e2e-worker-w0-2f69c8/distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb" Oct 30 09:17:56.349407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:56.331028 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a540c86f-c856-42a8-9f73-b5b9b690f42b-tmp-dir\") pod \"distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb\" (UID: \"a540c86f-c856-42a8-9f73-b5b9b690f42b\") " pod="e2e-models-e2e-worker-w0-2f69c8/distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb" Oct 30 09:17:56.350302 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:56.332758 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a540c86f-c856-42a8-9f73-b5b9b690f42b-model-cache\") pod \"distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb\" (UID: \"a540c86f-c856-42a8-9f73-b5b9b690f42b\") " pod="e2e-models-e2e-worker-w0-2f69c8/distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb" Oct 30 09:17:56.350302 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:56.330537 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a540c86f-c856-42a8-9f73-b5b9b690f42b-home\") pod \"distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb\" (UID: \"a540c86f-c856-42a8-9f73-b5b9b690f42b\") " pod="e2e-models-e2e-worker-w0-2f69c8/distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb" Oct 30 09:17:56.351852 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:56.351821 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a540c86f-c856-42a8-9f73-b5b9b690f42b-dshm\") pod \"distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb\" (UID: \"a540c86f-c856-42a8-9f73-b5b9b690f42b\") " pod="e2e-models-e2e-worker-w0-2f69c8/distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb" Oct 30 09:17:56.360469 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:56.359251 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a540c86f-c856-42a8-9f73-b5b9b690f42b-tls-certs\") pod \"distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb\" (UID: \"a540c86f-c856-42a8-9f73-b5b9b690f42b\") " pod="e2e-models-e2e-worker-w0-2f69c8/distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb" Oct 30 09:17:56.360469 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:56.359275 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg69t\" (UniqueName: \"kubernetes.io/projected/a540c86f-c856-42a8-9f73-b5b9b690f42b-kube-api-access-tg69t\") pod \"distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb\" (UID: \"a540c86f-c856-42a8-9f73-b5b9b690f42b\") " pod="e2e-models-e2e-worker-w0-2f69c8/distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb" Oct 30 09:17:56.436147 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:56.436071 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb" Oct 30 09:17:56.729101 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:17:56.729076 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda540c86f_c856_42a8_9f73_b5b9b690f42b.slice/crio-a0751bfaaaaa91b93af34962cbcebb50d7dcf847be062dfc665b211bef2818fb WatchSource:0}: Error finding container a0751bfaaaaa91b93af34962cbcebb50d7dcf847be062dfc665b211bef2818fb: Status 404 returned error can't find the container with id a0751bfaaaaa91b93af34962cbcebb50d7dcf847be062dfc665b211bef2818fb Oct 30 09:17:56.733783 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:56.733762 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2f69c8/distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb"] Oct 30 09:17:56.813128 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:56.813093 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb" event={"ID":"a540c86f-c856-42a8-9f73-b5b9b690f42b","Type":"ContainerStarted","Data":"fb0a8053bc154633938bee2dec78a3cf6dd626c8791aa8210b83a8dca615fde8"} Oct 30 09:17:56.813230 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:56.813146 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb" event={"ID":"a540c86f-c856-42a8-9f73-b5b9b690f42b","Type":"ContainerStarted","Data":"a0751bfaaaaa91b93af34962cbcebb50d7dcf847be062dfc665b211bef2818fb"} Oct 30 09:17:56.856464 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:56.856415 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb" podStartSLOduration=0.856400161 podStartE2EDuration="856.400161ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:17:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:17:56.856158613 +0000 UTC m=+1437.117719600" watchObservedRunningTime="2026-10-30 09:17:56.856400161 +0000 UTC m=+1437.117961146" Oct 30 09:17:57.436603 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:17:57.436569 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb" Oct 30 09:17:58.595744 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:17:58.595710 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:18:02.815887 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:02.815855 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-e3d2ff/distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55"] Oct 30 09:18:02.821929 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:02.821903 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55" Oct 30 09:18:02.837725 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:02.837692 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e3d2ff/distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55"] Oct 30 09:18:02.847058 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:02.847033 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-e3d2ff\"/\"openshift-service-ca.crt\"" Oct 30 09:18:02.847232 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:02.847090 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-e3d2ff\"/\"kube-root-ca.crt\"" Oct 30 09:18:02.850858 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:02.850840 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e3d2ff\"/\"default-dockercfg-jsvq8\"" Oct 30 09:18:02.850925 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:02.850868 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e3d2ff\"/\"distinct-2-w2-e3d2ff-kserve-self-signed-certs\"" Oct 30 09:18:02.907976 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:02.907946 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88be34c3-a9e9-4489-a449-206a99ef1663-dshm\") pod \"distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55\" (UID: \"88be34c3-a9e9-4489-a449-206a99ef1663\") " pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55" Oct 30 09:18:02.908103 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:02.907982 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88be34c3-a9e9-4489-a449-206a99ef1663-home\") pod \"distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55\" (UID: \"88be34c3-a9e9-4489-a449-206a99ef1663\") " pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55" Oct 30 09:18:02.908147 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:02.908105 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88be34c3-a9e9-4489-a449-206a99ef1663-model-cache\") pod \"distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55\" (UID: \"88be34c3-a9e9-4489-a449-206a99ef1663\") " pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55" Oct 30 09:18:02.908243 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:02.908229 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24zx9\" (UniqueName: \"kubernetes.io/projected/88be34c3-a9e9-4489-a449-206a99ef1663-kube-api-access-24zx9\") pod \"distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55\" (UID: \"88be34c3-a9e9-4489-a449-206a99ef1663\") " pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55" Oct 30 09:18:02.908284 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:02.908267 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88be34c3-a9e9-4489-a449-206a99ef1663-tls-certs\") pod \"distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55\" (UID: \"88be34c3-a9e9-4489-a449-206a99ef1663\") " pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55" Oct 30 09:18:02.908318 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:02.908308 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88be34c3-a9e9-4489-a449-206a99ef1663-tmp-dir\") pod \"distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55\" (UID: \"88be34c3-a9e9-4489-a449-206a99ef1663\") " pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55" Oct 30 09:18:03.009405 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:03.009144 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24zx9\" (UniqueName: \"kubernetes.io/projected/88be34c3-a9e9-4489-a449-206a99ef1663-kube-api-access-24zx9\") pod \"distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55\" (UID: \"88be34c3-a9e9-4489-a449-206a99ef1663\") " pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55" Oct 30 09:18:03.009405 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:03.009229 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88be34c3-a9e9-4489-a449-206a99ef1663-tls-certs\") pod \"distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55\" (UID: \"88be34c3-a9e9-4489-a449-206a99ef1663\") " pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55" Oct 30 09:18:03.009405 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:03.009265 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88be34c3-a9e9-4489-a449-206a99ef1663-tmp-dir\") pod \"distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55\" (UID: \"88be34c3-a9e9-4489-a449-206a99ef1663\") " pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55" Oct 30 09:18:03.009405 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:03.009320 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88be34c3-a9e9-4489-a449-206a99ef1663-dshm\") pod \"distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55\" (UID: \"88be34c3-a9e9-4489-a449-206a99ef1663\") " pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55" Oct 30 09:18:03.009405 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:03.009352 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88be34c3-a9e9-4489-a449-206a99ef1663-home\") pod \"distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55\" (UID: \"88be34c3-a9e9-4489-a449-206a99ef1663\") " pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55" Oct 30 09:18:03.009800 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:03.009449 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88be34c3-a9e9-4489-a449-206a99ef1663-model-cache\") pod \"distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55\" (UID: \"88be34c3-a9e9-4489-a449-206a99ef1663\") " pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55" Oct 30 09:18:03.009960 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:03.009839 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88be34c3-a9e9-4489-a449-206a99ef1663-tmp-dir\") pod \"distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55\" (UID: \"88be34c3-a9e9-4489-a449-206a99ef1663\") " pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55" Oct 30 09:18:03.010126 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:03.010094 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88be34c3-a9e9-4489-a449-206a99ef1663-home\") pod \"distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55\" (UID: \"88be34c3-a9e9-4489-a449-206a99ef1663\") " pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55" Oct 30 09:18:03.010282 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:03.010217 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88be34c3-a9e9-4489-a449-206a99ef1663-model-cache\") pod \"distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55\" (UID: \"88be34c3-a9e9-4489-a449-206a99ef1663\") " pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55" Oct 30 09:18:03.013111 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:03.013088 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88be34c3-a9e9-4489-a449-206a99ef1663-tls-certs\") pod \"distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55\" (UID: \"88be34c3-a9e9-4489-a449-206a99ef1663\") " pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55" Oct 30 09:18:03.018749 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:03.018719 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88be34c3-a9e9-4489-a449-206a99ef1663-dshm\") pod \"distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55\" (UID: \"88be34c3-a9e9-4489-a449-206a99ef1663\") " pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55" Oct 30 09:18:03.042868 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:03.042836 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24zx9\" (UniqueName: \"kubernetes.io/projected/88be34c3-a9e9-4489-a449-206a99ef1663-kube-api-access-24zx9\") pod \"distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55\" (UID: \"88be34c3-a9e9-4489-a449-206a99ef1663\") " pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55" Oct 30 09:18:03.135813 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:03.135742 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55" Oct 30 09:18:03.430080 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:18:03.430051 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88be34c3_a9e9_4489_a449_206a99ef1663.slice/crio-a99dbbef07136fd285aee2b8b3eb53e3db6426957f08cedd9e672a1be889cd11 WatchSource:0}: Error finding container a99dbbef07136fd285aee2b8b3eb53e3db6426957f08cedd9e672a1be889cd11: Status 404 returned error can't find the container with id a99dbbef07136fd285aee2b8b3eb53e3db6426957f08cedd9e672a1be889cd11 Oct 30 09:18:03.430737 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:03.430705 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e3d2ff/distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55"] Oct 30 09:18:03.469294 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:03.469270 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-176b6c/distinct-w3-176b6c-kserve-f57b7579c-k4pc5" Oct 30 09:18:03.483724 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:03.483701 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-176b6c/distinct-w3-176b6c-kserve-f57b7579c-k4pc5" Oct 30 09:18:03.852724 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:03.852687 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55" event={"ID":"88be34c3-a9e9-4489-a449-206a99ef1663","Type":"ContainerStarted","Data":"2e3a51433bccf125f099e070d60bba655bbb755e38aae223d00d08ee795595ee"} Oct 30 09:18:03.853194 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:03.852733 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55" event={"ID":"88be34c3-a9e9-4489-a449-206a99ef1663","Type":"ContainerStarted","Data":"a99dbbef07136fd285aee2b8b3eb53e3db6426957f08cedd9e672a1be889cd11"} Oct 30 09:18:03.863810 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:03.863785 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-176b6c/distinct-w3-176b6c-kserve-f57b7579c-k4pc5" Oct 30 09:18:03.894704 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:03.894659 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55" podStartSLOduration=1.894646061 podStartE2EDuration="1.894646061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:18:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:18:03.892419602 +0000 UTC m=+1444.153980587" watchObservedRunningTime="2026-10-30 09:18:03.894646061 +0000 UTC m=+1444.156207049" Oct 30 09:18:04.136855 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:04.136771 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55" Oct 30 09:18:06.436427 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:06.436374 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb" Oct 30 09:18:06.449264 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:06.449237 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb" Oct 30 09:18:06.879560 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:06.879529 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb" Oct 30 09:18:09.615406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:09.614758 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb"] Oct 30 09:18:09.619794 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:09.619769 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb" Oct 30 09:18:09.731618 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:09.731583 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb"] Oct 30 09:18:09.761732 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:09.761031 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28431c22-78da-4145-9842-e7d92b360558-istio-envoy\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb\" (UID: \"28431c22-78da-4145-9842-e7d92b360558\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb" Oct 30 09:18:09.761732 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:09.761087 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28431c22-78da-4145-9842-e7d92b360558-istio-data\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb\" (UID: \"28431c22-78da-4145-9842-e7d92b360558\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb" Oct 30 09:18:09.761732 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:09.761140 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h77tv\" (UniqueName: \"kubernetes.io/projected/28431c22-78da-4145-9842-e7d92b360558-kube-api-access-h77tv\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb\" (UID: \"28431c22-78da-4145-9842-e7d92b360558\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb" Oct 30 09:18:09.761732 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:09.761174 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28431c22-78da-4145-9842-e7d92b360558-istiod-ca-cert\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb\" (UID: \"28431c22-78da-4145-9842-e7d92b360558\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb" Oct 30 09:18:09.761732 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:09.761207 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28431c22-78da-4145-9842-e7d92b360558-credential-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb\" (UID: \"28431c22-78da-4145-9842-e7d92b360558\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb" Oct 30 09:18:09.761732 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:09.761288 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28431c22-78da-4145-9842-e7d92b360558-istio-token\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb\" (UID: \"28431c22-78da-4145-9842-e7d92b360558\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb" Oct 30 09:18:09.761732 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:09.761447 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28431c22-78da-4145-9842-e7d92b360558-istio-podinfo\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb\" (UID: \"28431c22-78da-4145-9842-e7d92b360558\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb" Oct 30 09:18:09.761732 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:09.761509 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28431c22-78da-4145-9842-e7d92b360558-workload-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb\" (UID: \"28431c22-78da-4145-9842-e7d92b360558\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb" Oct 30 09:18:09.761732 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:09.761539 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28431c22-78da-4145-9842-e7d92b360558-workload-certs\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb\" (UID: \"28431c22-78da-4145-9842-e7d92b360558\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb" Oct 30 09:18:09.890460 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:09.867043 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28431c22-78da-4145-9842-e7d92b360558-istio-token\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb\" (UID: \"28431c22-78da-4145-9842-e7d92b360558\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb" Oct 30 09:18:09.890864 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:09.890830 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28431c22-78da-4145-9842-e7d92b360558-istio-podinfo\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb\" (UID: \"28431c22-78da-4145-9842-e7d92b360558\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb" Oct 30 09:18:09.891045 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:09.891027 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28431c22-78da-4145-9842-e7d92b360558-workload-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb\" (UID: \"28431c22-78da-4145-9842-e7d92b360558\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb" Oct 30 09:18:09.891184 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:09.891167 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28431c22-78da-4145-9842-e7d92b360558-workload-certs\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb\" (UID: \"28431c22-78da-4145-9842-e7d92b360558\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb" Oct 30 09:18:09.891314 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:09.891298 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28431c22-78da-4145-9842-e7d92b360558-istio-envoy\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb\" (UID: \"28431c22-78da-4145-9842-e7d92b360558\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb" Oct 30 09:18:09.891456 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:09.891439 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28431c22-78da-4145-9842-e7d92b360558-istio-data\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb\" (UID: \"28431c22-78da-4145-9842-e7d92b360558\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb" Oct 30 09:18:09.891595 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:09.891577 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h77tv\" (UniqueName: \"kubernetes.io/projected/28431c22-78da-4145-9842-e7d92b360558-kube-api-access-h77tv\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb\" (UID: \"28431c22-78da-4145-9842-e7d92b360558\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb" Oct 30 09:18:09.891706 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:09.891691 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28431c22-78da-4145-9842-e7d92b360558-istiod-ca-cert\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb\" (UID: \"28431c22-78da-4145-9842-e7d92b360558\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb" Oct 30 09:18:09.891833 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:09.891818 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28431c22-78da-4145-9842-e7d92b360558-credential-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb\" (UID: \"28431c22-78da-4145-9842-e7d92b360558\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb" Oct 30 09:18:09.900978 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:09.900951 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28431c22-78da-4145-9842-e7d92b360558-istio-podinfo\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb\" (UID: \"28431c22-78da-4145-9842-e7d92b360558\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb" Oct 30 09:18:09.911401 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:09.901451 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28431c22-78da-4145-9842-e7d92b360558-workload-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb\" (UID: \"28431c22-78da-4145-9842-e7d92b360558\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb" Oct 30 09:18:09.911814 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:09.911794 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28431c22-78da-4145-9842-e7d92b360558-workload-certs\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb\" (UID: \"28431c22-78da-4145-9842-e7d92b360558\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb" Oct 30 09:18:09.919649 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:09.919625 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28431c22-78da-4145-9842-e7d92b360558-istio-envoy\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb\" (UID: \"28431c22-78da-4145-9842-e7d92b360558\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb" Oct 30 09:18:09.934846 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:09.934823 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28431c22-78da-4145-9842-e7d92b360558-istio-data\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb\" (UID: \"28431c22-78da-4145-9842-e7d92b360558\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb" Oct 30 09:18:09.935865 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:09.935843 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28431c22-78da-4145-9842-e7d92b360558-istiod-ca-cert\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb\" (UID: \"28431c22-78da-4145-9842-e7d92b360558\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb" Oct 30 09:18:09.948401 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:09.936184 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28431c22-78da-4145-9842-e7d92b360558-credential-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb\" (UID: \"28431c22-78da-4145-9842-e7d92b360558\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb" Oct 30 09:18:09.959421 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:09.951944 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28431c22-78da-4145-9842-e7d92b360558-istio-token\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb\" (UID: \"28431c22-78da-4145-9842-e7d92b360558\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb" Oct 30 09:18:10.027487 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:10.027450 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h77tv\" (UniqueName: \"kubernetes.io/projected/28431c22-78da-4145-9842-e7d92b360558-kube-api-access-h77tv\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb\" (UID: \"28431c22-78da-4145-9842-e7d92b360558\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb" Oct 30 09:18:10.233640 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:10.233600 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb" Oct 30 09:18:10.689635 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:10.689602 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb"] Oct 30 09:18:10.709492 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:18:10.708460 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28431c22_78da_4145_9842_e7d92b360558.slice/crio-52545ea1acdef5b53f73ef2f9646595ffdd6a4450fe38f0256b7f7bd3cfb1d9c WatchSource:0}: Error finding container 52545ea1acdef5b53f73ef2f9646595ffdd6a4450fe38f0256b7f7bd3cfb1d9c: Status 404 returned error can't find the container with id 52545ea1acdef5b53f73ef2f9646595ffdd6a4450fe38f0256b7f7bd3cfb1d9c Oct 30 09:18:10.710851 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:10.710817 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:18:10.710953 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:10.710902 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:18:10.710953 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:10.710933 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:18:10.963476 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:10.963365 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb" event={"ID":"28431c22-78da-4145-9842-e7d92b360558","Type":"ContainerStarted","Data":"a245dbbb93cdaea99d02ce7029d9269ce0cdaddc9f2e9de5aeddce8277d4d83d"} Oct 30 09:18:10.963476 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:10.963433 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb" event={"ID":"28431c22-78da-4145-9842-e7d92b360558","Type":"ContainerStarted","Data":"52545ea1acdef5b53f73ef2f9646595ffdd6a4450fe38f0256b7f7bd3cfb1d9c"} Oct 30 09:18:11.012087 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:11.012026 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb" podStartSLOduration=2.012007293 podStartE2EDuration="2.012007293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:18:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:18:11.009031054 +0000 UTC m=+1451.270592043" watchObservedRunningTime="2026-10-30 09:18:11.012007293 +0000 UTC m=+1451.273568281" Oct 30 09:18:11.235098 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:11.235015 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb" Oct 30 09:18:11.236478 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:11.236453 4562 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 09:18:11.236590 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:11.236498 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb" podUID="28431c22-78da-4145-9842-e7d92b360558" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 09:18:12.235303 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:12.235263 4562 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 09:18:12.235722 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:12.235322 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb" podUID="28431c22-78da-4145-9842-e7d92b360558" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 09:18:13.136540 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:13.136506 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55" Oct 30 09:18:13.156023 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:13.155982 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55" Oct 30 09:18:13.234659 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:13.234628 4562 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 09:18:13.234837 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:13.234678 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb" podUID="28431c22-78da-4145-9842-e7d92b360558" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 09:18:13.991649 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:13.991615 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55" Oct 30 09:18:14.235209 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:14.235178 4562 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 09:18:14.235404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:14.235240 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb" podUID="28431c22-78da-4145-9842-e7d92b360558" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 09:18:15.235247 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:15.235205 4562 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 09:18:15.235658 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:15.235275 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb" podUID="28431c22-78da-4145-9842-e7d92b360558" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 09:18:16.234671 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:16.234636 4562 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 09:18:16.234859 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:16.234692 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb" podUID="28431c22-78da-4145-9842-e7d92b360558" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 09:18:17.235673 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:17.235630 4562 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 09:18:17.236161 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:17.235721 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb" podUID="28431c22-78da-4145-9842-e7d92b360558" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 09:18:18.274653 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:18.274608 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb" Oct 30 09:18:18.275521 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:18.275461 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb" Oct 30 09:18:18.276757 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:18.276735 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb" Oct 30 09:18:24.742740 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:24.742697 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9"] Oct 30 09:18:24.747403 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:24.747352 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9" Oct 30 09:18:24.783900 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:24.783862 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9"] Oct 30 09:18:24.843912 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:24.843870 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4c52d5e-0a44-479d-bfce-072508f5527a-istio-envoy\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9\" (UID: \"f4c52d5e-0a44-479d-bfce-072508f5527a\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9" Oct 30 09:18:24.844133 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:24.844039 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4c52d5e-0a44-479d-bfce-072508f5527a-workload-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9\" (UID: \"f4c52d5e-0a44-479d-bfce-072508f5527a\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9" Oct 30 09:18:24.844133 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:24.844095 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4c52d5e-0a44-479d-bfce-072508f5527a-credential-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9\" (UID: \"f4c52d5e-0a44-479d-bfce-072508f5527a\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9" Oct 30 09:18:24.844239 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:24.844164 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4c52d5e-0a44-479d-bfce-072508f5527a-istio-token\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9\" (UID: \"f4c52d5e-0a44-479d-bfce-072508f5527a\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9" Oct 30 09:18:24.844239 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:24.844187 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnqsq\" (UniqueName: \"kubernetes.io/projected/f4c52d5e-0a44-479d-bfce-072508f5527a-kube-api-access-pnqsq\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9\" (UID: \"f4c52d5e-0a44-479d-bfce-072508f5527a\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9" Oct 30 09:18:24.844239 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:24.844227 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4c52d5e-0a44-479d-bfce-072508f5527a-istiod-ca-cert\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9\" (UID: \"f4c52d5e-0a44-479d-bfce-072508f5527a\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9" Oct 30 09:18:24.844463 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:24.844314 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4c52d5e-0a44-479d-bfce-072508f5527a-istio-podinfo\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9\" (UID: \"f4c52d5e-0a44-479d-bfce-072508f5527a\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9" Oct 30 09:18:24.844523 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:24.844453 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4c52d5e-0a44-479d-bfce-072508f5527a-istio-data\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9\" (UID: \"f4c52d5e-0a44-479d-bfce-072508f5527a\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9" Oct 30 09:18:24.844578 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:24.844562 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4c52d5e-0a44-479d-bfce-072508f5527a-workload-certs\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9\" (UID: \"f4c52d5e-0a44-479d-bfce-072508f5527a\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9" Oct 30 09:18:24.945796 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:24.945738 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4c52d5e-0a44-479d-bfce-072508f5527a-workload-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9\" (UID: \"f4c52d5e-0a44-479d-bfce-072508f5527a\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9" Oct 30 09:18:24.945796 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:24.945790 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4c52d5e-0a44-479d-bfce-072508f5527a-credential-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9\" (UID: \"f4c52d5e-0a44-479d-bfce-072508f5527a\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9" Oct 30 09:18:24.946064 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:24.945835 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4c52d5e-0a44-479d-bfce-072508f5527a-istio-token\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9\" (UID: \"f4c52d5e-0a44-479d-bfce-072508f5527a\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9" Oct 30 09:18:24.946064 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:24.945864 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnqsq\" (UniqueName: \"kubernetes.io/projected/f4c52d5e-0a44-479d-bfce-072508f5527a-kube-api-access-pnqsq\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9\" (UID: \"f4c52d5e-0a44-479d-bfce-072508f5527a\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9" Oct 30 09:18:24.946064 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:24.945915 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4c52d5e-0a44-479d-bfce-072508f5527a-istiod-ca-cert\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9\" (UID: \"f4c52d5e-0a44-479d-bfce-072508f5527a\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9" Oct 30 09:18:24.946064 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:24.945961 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4c52d5e-0a44-479d-bfce-072508f5527a-istio-podinfo\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9\" (UID: \"f4c52d5e-0a44-479d-bfce-072508f5527a\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9" Oct 30 09:18:24.946064 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:24.946005 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4c52d5e-0a44-479d-bfce-072508f5527a-istio-data\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9\" (UID: \"f4c52d5e-0a44-479d-bfce-072508f5527a\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9" Oct 30 09:18:24.946328 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:24.946083 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4c52d5e-0a44-479d-bfce-072508f5527a-workload-certs\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9\" (UID: \"f4c52d5e-0a44-479d-bfce-072508f5527a\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9" Oct 30 09:18:24.946328 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:24.946160 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4c52d5e-0a44-479d-bfce-072508f5527a-istio-envoy\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9\" (UID: \"f4c52d5e-0a44-479d-bfce-072508f5527a\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9" Oct 30 09:18:24.947192 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:24.947097 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4c52d5e-0a44-479d-bfce-072508f5527a-credential-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9\" (UID: \"f4c52d5e-0a44-479d-bfce-072508f5527a\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9" Oct 30 09:18:24.947456 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:24.947367 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4c52d5e-0a44-479d-bfce-072508f5527a-workload-certs\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9\" (UID: \"f4c52d5e-0a44-479d-bfce-072508f5527a\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9" Oct 30 09:18:24.947456 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:24.947405 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4c52d5e-0a44-479d-bfce-072508f5527a-istio-data\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9\" (UID: \"f4c52d5e-0a44-479d-bfce-072508f5527a\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9" Oct 30 09:18:24.947456 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:24.947432 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4c52d5e-0a44-479d-bfce-072508f5527a-workload-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9\" (UID: \"f4c52d5e-0a44-479d-bfce-072508f5527a\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9" Oct 30 09:18:24.947845 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:24.947820 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4c52d5e-0a44-479d-bfce-072508f5527a-istiod-ca-cert\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9\" (UID: \"f4c52d5e-0a44-479d-bfce-072508f5527a\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9" Oct 30 09:18:24.949154 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:24.949131 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4c52d5e-0a44-479d-bfce-072508f5527a-istio-envoy\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9\" (UID: \"f4c52d5e-0a44-479d-bfce-072508f5527a\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9" Oct 30 09:18:24.949689 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:24.949665 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4c52d5e-0a44-479d-bfce-072508f5527a-istio-podinfo\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9\" (UID: \"f4c52d5e-0a44-479d-bfce-072508f5527a\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9" Oct 30 09:18:24.977078 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:24.977048 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnqsq\" (UniqueName: \"kubernetes.io/projected/f4c52d5e-0a44-479d-bfce-072508f5527a-kube-api-access-pnqsq\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9\" (UID: \"f4c52d5e-0a44-479d-bfce-072508f5527a\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9" Oct 30 09:18:24.977458 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:24.977440 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4c52d5e-0a44-479d-bfce-072508f5527a-istio-token\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9\" (UID: \"f4c52d5e-0a44-479d-bfce-072508f5527a\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9" Oct 30 09:18:25.065441 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:25.062454 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9" Oct 30 09:18:25.403575 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:18:25.403539 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4c52d5e_0a44_479d_bfce_072508f5527a.slice/crio-302ddc7252d3415e00f841874973bd44a309559a07ce4d6b5913e5747574a61c WatchSource:0}: Error finding container 302ddc7252d3415e00f841874973bd44a309559a07ce4d6b5913e5747574a61c: Status 404 returned error can't find the container with id 302ddc7252d3415e00f841874973bd44a309559a07ce4d6b5913e5747574a61c Oct 30 09:18:25.404578 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:25.404546 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9"] Oct 30 09:18:25.406000 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:25.405972 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:18:25.406078 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:25.406032 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:18:25.406078 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:25.406062 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:18:26.083505 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:26.082498 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9" event={"ID":"f4c52d5e-0a44-479d-bfce-072508f5527a","Type":"ContainerStarted","Data":"f085a92738e9188c7ebb3623a0491b2758556fa495f56eed5f600e252cea6cde"} Oct 30 09:18:26.083505 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:26.082564 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9" event={"ID":"f4c52d5e-0a44-479d-bfce-072508f5527a","Type":"ContainerStarted","Data":"302ddc7252d3415e00f841874973bd44a309559a07ce4d6b5913e5747574a61c"} Oct 30 09:18:27.079406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:27.071825 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9" Oct 30 09:18:27.079406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:27.075466 4562 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 09:18:27.079406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:27.075531 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9" podUID="f4c52d5e-0a44-479d-bfce-072508f5527a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 09:18:28.065931 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:28.065895 4562 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 09:18:28.066615 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:28.066567 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9" podUID="f4c52d5e-0a44-479d-bfce-072508f5527a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 09:18:28.595133 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:18:28.595092 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:18:29.069987 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:29.064718 4562 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 09:18:29.069987 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:29.064777 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9" podUID="f4c52d5e-0a44-479d-bfce-072508f5527a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 09:18:30.063698 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:30.063659 4562 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 09:18:30.064000 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:30.063971 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9" podUID="f4c52d5e-0a44-479d-bfce-072508f5527a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 09:18:31.068523 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:31.068489 4562 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 09:18:31.069116 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:31.069069 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9" podUID="f4c52d5e-0a44-479d-bfce-072508f5527a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 09:18:32.062746 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:32.062713 4562 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 09:18:32.062910 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:32.062765 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9" podUID="f4c52d5e-0a44-479d-bfce-072508f5527a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 09:18:32.613729 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:32.613672 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9" podStartSLOduration=8.613657899 podStartE2EDuration="8.613657899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:18:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:18:26.183634721 +0000 UTC m=+1466.445195712" watchObservedRunningTime="2026-10-30 09:18:32.613657899 +0000 UTC m=+1472.875218886" Oct 30 09:18:32.616196 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:32.616169 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh"] Oct 30 09:18:32.621286 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:32.621264 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh" Oct 30 09:18:32.657839 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:32.657807 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh"] Oct 30 09:18:32.725350 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:32.725316 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9491d089-4927-4270-82f2-fecd0f108311-istio-envoy\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh\" (UID: \"9491d089-4927-4270-82f2-fecd0f108311\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh" Oct 30 09:18:32.725526 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:32.725394 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twwrq\" (UniqueName: \"kubernetes.io/projected/9491d089-4927-4270-82f2-fecd0f108311-kube-api-access-twwrq\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh\" (UID: \"9491d089-4927-4270-82f2-fecd0f108311\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh" Oct 30 09:18:32.725526 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:32.725423 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9491d089-4927-4270-82f2-fecd0f108311-workload-certs\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh\" (UID: \"9491d089-4927-4270-82f2-fecd0f108311\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh" Oct 30 09:18:32.725620 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:32.725529 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9491d089-4927-4270-82f2-fecd0f108311-workload-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh\" (UID: \"9491d089-4927-4270-82f2-fecd0f108311\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh" Oct 30 09:18:32.725620 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:32.725594 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9491d089-4927-4270-82f2-fecd0f108311-istio-podinfo\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh\" (UID: \"9491d089-4927-4270-82f2-fecd0f108311\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh" Oct 30 09:18:32.725763 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:32.725737 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9491d089-4927-4270-82f2-fecd0f108311-credential-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh\" (UID: \"9491d089-4927-4270-82f2-fecd0f108311\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh" Oct 30 09:18:32.725871 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:32.725852 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9491d089-4927-4270-82f2-fecd0f108311-istio-token\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh\" (UID: \"9491d089-4927-4270-82f2-fecd0f108311\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh" Oct 30 09:18:32.725928 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:32.725913 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9491d089-4927-4270-82f2-fecd0f108311-istiod-ca-cert\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh\" (UID: \"9491d089-4927-4270-82f2-fecd0f108311\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh" Oct 30 09:18:32.725997 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:32.725985 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9491d089-4927-4270-82f2-fecd0f108311-istio-data\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh\" (UID: \"9491d089-4927-4270-82f2-fecd0f108311\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh" Oct 30 09:18:32.827350 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:32.827314 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9491d089-4927-4270-82f2-fecd0f108311-istio-token\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh\" (UID: \"9491d089-4927-4270-82f2-fecd0f108311\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh" Oct 30 09:18:32.827557 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:32.827421 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9491d089-4927-4270-82f2-fecd0f108311-istiod-ca-cert\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh\" (UID: \"9491d089-4927-4270-82f2-fecd0f108311\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh" Oct 30 09:18:32.827557 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:32.827453 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9491d089-4927-4270-82f2-fecd0f108311-istio-data\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh\" (UID: \"9491d089-4927-4270-82f2-fecd0f108311\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh" Oct 30 09:18:32.827557 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:32.827526 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9491d089-4927-4270-82f2-fecd0f108311-istio-envoy\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh\" (UID: \"9491d089-4927-4270-82f2-fecd0f108311\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh" Oct 30 09:18:32.827733 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:32.827643 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twwrq\" (UniqueName: \"kubernetes.io/projected/9491d089-4927-4270-82f2-fecd0f108311-kube-api-access-twwrq\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh\" (UID: \"9491d089-4927-4270-82f2-fecd0f108311\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh" Oct 30 09:18:32.827733 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:32.827707 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9491d089-4927-4270-82f2-fecd0f108311-workload-certs\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh\" (UID: \"9491d089-4927-4270-82f2-fecd0f108311\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh" Oct 30 09:18:32.827839 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:32.827765 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9491d089-4927-4270-82f2-fecd0f108311-workload-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh\" (UID: \"9491d089-4927-4270-82f2-fecd0f108311\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh" Oct 30 09:18:32.827839 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:32.827823 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9491d089-4927-4270-82f2-fecd0f108311-istio-podinfo\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh\" (UID: \"9491d089-4927-4270-82f2-fecd0f108311\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh" Oct 30 09:18:32.827943 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:32.827906 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9491d089-4927-4270-82f2-fecd0f108311-credential-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh\" (UID: \"9491d089-4927-4270-82f2-fecd0f108311\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh" Oct 30 09:18:32.828277 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:32.828253 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9491d089-4927-4270-82f2-fecd0f108311-istiod-ca-cert\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh\" (UID: \"9491d089-4927-4270-82f2-fecd0f108311\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh" Oct 30 09:18:32.828466 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:32.828331 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9491d089-4927-4270-82f2-fecd0f108311-credential-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh\" (UID: \"9491d089-4927-4270-82f2-fecd0f108311\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh" Oct 30 09:18:32.828580 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:32.828443 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9491d089-4927-4270-82f2-fecd0f108311-istio-data\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh\" (UID: \"9491d089-4927-4270-82f2-fecd0f108311\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh" Oct 30 09:18:32.828580 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:32.828500 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9491d089-4927-4270-82f2-fecd0f108311-workload-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh\" (UID: \"9491d089-4927-4270-82f2-fecd0f108311\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh" Oct 30 09:18:32.828796 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:32.828774 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9491d089-4927-4270-82f2-fecd0f108311-workload-certs\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh\" (UID: \"9491d089-4927-4270-82f2-fecd0f108311\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh" Oct 30 09:18:32.830767 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:32.830742 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9491d089-4927-4270-82f2-fecd0f108311-istio-envoy\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh\" (UID: \"9491d089-4927-4270-82f2-fecd0f108311\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh" Oct 30 09:18:32.831295 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:32.831262 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9491d089-4927-4270-82f2-fecd0f108311-istio-podinfo\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh\" (UID: \"9491d089-4927-4270-82f2-fecd0f108311\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh" Oct 30 09:18:32.849351 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:32.849295 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9491d089-4927-4270-82f2-fecd0f108311-istio-token\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh\" (UID: \"9491d089-4927-4270-82f2-fecd0f108311\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh" Oct 30 09:18:32.849864 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:32.849821 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twwrq\" (UniqueName: \"kubernetes.io/projected/9491d089-4927-4270-82f2-fecd0f108311-kube-api-access-twwrq\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh\" (UID: \"9491d089-4927-4270-82f2-fecd0f108311\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh" Oct 30 09:18:32.938938 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:32.938913 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh" Oct 30 09:18:33.279921 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:33.279822 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9" Oct 30 09:18:33.281156 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:33.281132 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9" Oct 30 09:18:33.282753 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:33.282732 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9" Oct 30 09:18:33.290238 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:33.290214 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh"] Oct 30 09:18:33.291278 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:18:33.291248 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9491d089_4927_4270_82f2_fecd0f108311.slice/crio-6f6ea8eb248692526d9a787f5dd1c35c4bf96a6a7bf631b76ad8552cc682e27f WatchSource:0}: Error finding container 6f6ea8eb248692526d9a787f5dd1c35c4bf96a6a7bf631b76ad8552cc682e27f: Status 404 returned error can't find the container with id 6f6ea8eb248692526d9a787f5dd1c35c4bf96a6a7bf631b76ad8552cc682e27f Oct 30 09:18:33.293531 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:33.293503 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:18:33.293587 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:33.293568 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:18:33.293632 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:33.293598 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:18:34.206376 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:34.206337 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh" event={"ID":"9491d089-4927-4270-82f2-fecd0f108311","Type":"ContainerStarted","Data":"5ad2bc72a04755052f3f98d927116b4d65e50d9a8d9f3d6f37017c84c6f7d18e"} Oct 30 09:18:34.206798 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:34.206415 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh" event={"ID":"9491d089-4927-4270-82f2-fecd0f108311","Type":"ContainerStarted","Data":"6f6ea8eb248692526d9a787f5dd1c35c4bf96a6a7bf631b76ad8552cc682e27f"} Oct 30 09:18:34.955406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:34.942699 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh" Oct 30 09:18:34.963898 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:34.963531 4562 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 09:18:34.963898 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:34.963605 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh" podUID="9491d089-4927-4270-82f2-fecd0f108311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 09:18:35.603701 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:35.603650 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh" podStartSLOduration=3.603635284 podStartE2EDuration="3.603635284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:18:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:18:34.26440945 +0000 UTC m=+1474.525970450" watchObservedRunningTime="2026-10-30 09:18:35.603635284 +0000 UTC m=+1475.865196272" Oct 30 09:18:35.604089 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:35.603966 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd"] Oct 30 09:18:35.608078 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:35.608059 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd" Oct 30 09:18:35.636009 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:35.634974 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd"] Oct 30 09:18:35.706873 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:35.706843 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/802df8bf-9c24-428b-9d87-96f58054ad68-istio-data\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd\" (UID: \"802df8bf-9c24-428b-9d87-96f58054ad68\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd" Oct 30 09:18:35.706873 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:35.706875 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/802df8bf-9c24-428b-9d87-96f58054ad68-istio-token\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd\" (UID: \"802df8bf-9c24-428b-9d87-96f58054ad68\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd" Oct 30 09:18:35.707134 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:35.706899 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/802df8bf-9c24-428b-9d87-96f58054ad68-workload-certs\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd\" (UID: \"802df8bf-9c24-428b-9d87-96f58054ad68\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd" Oct 30 09:18:35.707134 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:35.707019 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htb2w\" (UniqueName: \"kubernetes.io/projected/802df8bf-9c24-428b-9d87-96f58054ad68-kube-api-access-htb2w\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd\" (UID: \"802df8bf-9c24-428b-9d87-96f58054ad68\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd" Oct 30 09:18:35.707134 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:35.707080 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/802df8bf-9c24-428b-9d87-96f58054ad68-istio-podinfo\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd\" (UID: \"802df8bf-9c24-428b-9d87-96f58054ad68\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd" Oct 30 09:18:35.707278 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:35.707190 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/802df8bf-9c24-428b-9d87-96f58054ad68-istio-envoy\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd\" (UID: \"802df8bf-9c24-428b-9d87-96f58054ad68\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd" Oct 30 09:18:35.707278 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:35.707254 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/802df8bf-9c24-428b-9d87-96f58054ad68-credential-socket\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd\" (UID: \"802df8bf-9c24-428b-9d87-96f58054ad68\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd" Oct 30 09:18:35.707374 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:35.707314 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/802df8bf-9c24-428b-9d87-96f58054ad68-istiod-ca-cert\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd\" (UID: \"802df8bf-9c24-428b-9d87-96f58054ad68\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd" Oct 30 09:18:35.707471 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:35.707374 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/802df8bf-9c24-428b-9d87-96f58054ad68-workload-socket\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd\" (UID: \"802df8bf-9c24-428b-9d87-96f58054ad68\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd" Oct 30 09:18:35.808354 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:35.808319 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/802df8bf-9c24-428b-9d87-96f58054ad68-istio-envoy\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd\" (UID: \"802df8bf-9c24-428b-9d87-96f58054ad68\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd" Oct 30 09:18:35.808535 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:35.808361 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/802df8bf-9c24-428b-9d87-96f58054ad68-credential-socket\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd\" (UID: \"802df8bf-9c24-428b-9d87-96f58054ad68\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd" Oct 30 09:18:35.808535 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:35.808462 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/802df8bf-9c24-428b-9d87-96f58054ad68-istiod-ca-cert\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd\" (UID: \"802df8bf-9c24-428b-9d87-96f58054ad68\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd" Oct 30 09:18:35.808535 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:35.808493 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/802df8bf-9c24-428b-9d87-96f58054ad68-workload-socket\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd\" (UID: \"802df8bf-9c24-428b-9d87-96f58054ad68\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd" Oct 30 09:18:35.808688 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:35.808571 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/802df8bf-9c24-428b-9d87-96f58054ad68-istio-data\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd\" (UID: \"802df8bf-9c24-428b-9d87-96f58054ad68\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd" Oct 30 09:18:35.808688 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:35.808595 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/802df8bf-9c24-428b-9d87-96f58054ad68-istio-token\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd\" (UID: \"802df8bf-9c24-428b-9d87-96f58054ad68\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd" Oct 30 09:18:35.808688 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:35.808632 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/802df8bf-9c24-428b-9d87-96f58054ad68-workload-certs\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd\" (UID: \"802df8bf-9c24-428b-9d87-96f58054ad68\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd" Oct 30 09:18:35.808688 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:35.808682 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htb2w\" (UniqueName: \"kubernetes.io/projected/802df8bf-9c24-428b-9d87-96f58054ad68-kube-api-access-htb2w\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd\" (UID: \"802df8bf-9c24-428b-9d87-96f58054ad68\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd" Oct 30 09:18:35.808881 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:35.808733 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/802df8bf-9c24-428b-9d87-96f58054ad68-istio-podinfo\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd\" (UID: \"802df8bf-9c24-428b-9d87-96f58054ad68\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd" Oct 30 09:18:35.808940 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:35.808885 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/802df8bf-9c24-428b-9d87-96f58054ad68-credential-socket\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd\" (UID: \"802df8bf-9c24-428b-9d87-96f58054ad68\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd" Oct 30 09:18:35.809002 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:35.808983 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/802df8bf-9c24-428b-9d87-96f58054ad68-workload-socket\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd\" (UID: \"802df8bf-9c24-428b-9d87-96f58054ad68\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd" Oct 30 09:18:35.809078 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:35.809050 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/802df8bf-9c24-428b-9d87-96f58054ad68-workload-certs\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd\" (UID: \"802df8bf-9c24-428b-9d87-96f58054ad68\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd" Oct 30 09:18:35.809247 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:35.809160 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/802df8bf-9c24-428b-9d87-96f58054ad68-istio-data\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd\" (UID: \"802df8bf-9c24-428b-9d87-96f58054ad68\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd" Oct 30 09:18:35.809325 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:35.809282 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/802df8bf-9c24-428b-9d87-96f58054ad68-istiod-ca-cert\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd\" (UID: \"802df8bf-9c24-428b-9d87-96f58054ad68\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd" Oct 30 09:18:35.810935 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:35.810904 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/802df8bf-9c24-428b-9d87-96f58054ad68-istio-envoy\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd\" (UID: \"802df8bf-9c24-428b-9d87-96f58054ad68\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd" Oct 30 09:18:35.811113 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:35.811096 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/802df8bf-9c24-428b-9d87-96f58054ad68-istio-podinfo\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd\" (UID: \"802df8bf-9c24-428b-9d87-96f58054ad68\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd" Oct 30 09:18:35.833131 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:35.833103 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/802df8bf-9c24-428b-9d87-96f58054ad68-istio-token\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd\" (UID: \"802df8bf-9c24-428b-9d87-96f58054ad68\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd" Oct 30 09:18:35.833228 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:35.833138 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htb2w\" (UniqueName: \"kubernetes.io/projected/802df8bf-9c24-428b-9d87-96f58054ad68-kube-api-access-htb2w\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd\" (UID: \"802df8bf-9c24-428b-9d87-96f58054ad68\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd" Oct 30 09:18:35.926006 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:35.925931 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd" Oct 30 09:18:35.959634 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:35.957550 4562 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 09:18:35.959634 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:35.957612 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh" podUID="9491d089-4927-4270-82f2-fecd0f108311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 09:18:36.267368 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:36.267341 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd"] Oct 30 09:18:36.269828 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:18:36.269803 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod802df8bf_9c24_428b_9d87_96f58054ad68.slice/crio-5f12043fee692da94e520a04b806d89d9791667669f61f47bd5c6742a6ade7c4 WatchSource:0}: Error finding container 5f12043fee692da94e520a04b806d89d9791667669f61f47bd5c6742a6ade7c4: Status 404 returned error can't find the container with id 5f12043fee692da94e520a04b806d89d9791667669f61f47bd5c6742a6ade7c4 Oct 30 09:18:36.272124 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:36.272096 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:18:36.272192 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:36.272161 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:18:36.272229 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:36.272190 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:18:36.948406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:36.942841 4562 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 09:18:36.948406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:36.942913 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh" podUID="9491d089-4927-4270-82f2-fecd0f108311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 09:18:37.234044 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:37.233950 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd" event={"ID":"802df8bf-9c24-428b-9d87-96f58054ad68","Type":"ContainerStarted","Data":"599ca44ae43f5ca43d5317ff007cafd1348723aedf14a528035d83a8badbd353"} Oct 30 09:18:37.234044 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:37.234015 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd" event={"ID":"802df8bf-9c24-428b-9d87-96f58054ad68","Type":"ContainerStarted","Data":"5f12043fee692da94e520a04b806d89d9791667669f61f47bd5c6742a6ade7c4"} Oct 30 09:18:37.321405 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:37.320543 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd" podStartSLOduration=2.32052524 podStartE2EDuration="2.32052524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:18:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:18:37.302622742 +0000 UTC m=+1477.564183730" watchObservedRunningTime="2026-10-30 09:18:37.32052524 +0000 UTC m=+1477.582086229" Oct 30 09:18:37.926716 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:37.926675 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd" Oct 30 09:18:37.941804 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:37.941771 4562 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 09:18:37.943255 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:37.943225 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd" podUID="802df8bf-9c24-428b-9d87-96f58054ad68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 09:18:37.943646 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:37.943170 4562 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 09:18:37.943777 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:37.943758 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh" podUID="9491d089-4927-4270-82f2-fecd0f108311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 09:18:38.645408 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:38.643470 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/distinct-2-w1-c5447d-kserve-676749d548-xlmsf" podUID="c9ff897f-c520-4d0f-8820-e185b1d47dac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:38.878409 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:38.877470 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb" podUID="a540c86f-c856-42a8-9f73-b5b9b690f42b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:38.927919 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:38.927820 4562 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 09:18:38.928178 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:38.928146 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd" podUID="802df8bf-9c24-428b-9d87-96f58054ad68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 09:18:38.939829 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:38.939791 4562 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 09:18:38.940102 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:38.940070 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh" podUID="9491d089-4927-4270-82f2-fecd0f108311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 09:18:39.646528 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:39.646478 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/distinct-2-w1-c5447d-kserve-676749d548-xlmsf" podUID="c9ff897f-c520-4d0f-8820-e185b1d47dac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:39.647629 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:39.647601 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c5447d/distinct-2-w1-c5447d-kserve-676749d548-xlmsf" Oct 30 09:18:39.940463 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:39.928477 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-176b6c/distinct-w3-176b6c-kserve-f57b7579c-k4pc5" podUID="28fd4072-399c-4be5-a0d8-1be6966da4af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:39.956661 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:39.956625 4562 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 09:18:39.960527 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:39.956932 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh" podUID="9491d089-4927-4270-82f2-fecd0f108311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 09:18:40.249653 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:40.017873 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb" podUID="a540c86f-c856-42a8-9f73-b5b9b690f42b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:40.253540 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:40.041210 4562 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 09:18:40.307585 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:40.307538 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd" podUID="802df8bf-9c24-428b-9d87-96f58054ad68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 09:18:40.442009 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:40.441968 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb" Oct 30 09:18:40.442269 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:40.442253 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb"] Oct 30 09:18:40.590454 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:40.586881 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55" podUID="88be34c3-a9e9-4489-a449-206a99ef1663" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:40.590454 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:40.589697 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb"] Oct 30 09:18:40.590454 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:40.589858 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb" Oct 30 09:18:40.665593 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:40.665535 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/distinct-2-w1-c5447d-kserve-676749d548-xlmsf" podUID="c9ff897f-c520-4d0f-8820-e185b1d47dac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:40.713820 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:40.713772 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbe6d301-322a-4e4b-880b-36d5704928e4-istio-token\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb\" (UID: \"cbe6d301-322a-4e4b-880b-36d5704928e4\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb" Oct 30 09:18:40.722613 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:40.722579 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbe6d301-322a-4e4b-880b-36d5704928e4-istiod-ca-cert\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb\" (UID: \"cbe6d301-322a-4e4b-880b-36d5704928e4\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb" Oct 30 09:18:40.730525 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:40.726518 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbe6d301-322a-4e4b-880b-36d5704928e4-istio-envoy\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb\" (UID: \"cbe6d301-322a-4e4b-880b-36d5704928e4\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb" Oct 30 09:18:40.742547 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:40.738677 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr9c4\" (UniqueName: \"kubernetes.io/projected/cbe6d301-322a-4e4b-880b-36d5704928e4-kube-api-access-dr9c4\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb\" (UID: \"cbe6d301-322a-4e4b-880b-36d5704928e4\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb" Oct 30 09:18:40.811738 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:40.811698 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe6d301-322a-4e4b-880b-36d5704928e4-workload-socket\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb\" (UID: \"cbe6d301-322a-4e4b-880b-36d5704928e4\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb" Oct 30 09:18:40.857806 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:40.857712 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbe6d301-322a-4e4b-880b-36d5704928e4-istio-podinfo\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb\" (UID: \"cbe6d301-322a-4e4b-880b-36d5704928e4\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb" Oct 30 09:18:40.858290 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:40.858268 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe6d301-322a-4e4b-880b-36d5704928e4-credential-socket\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb\" (UID: \"cbe6d301-322a-4e4b-880b-36d5704928e4\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb" Oct 30 09:18:40.858592 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:40.858482 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbe6d301-322a-4e4b-880b-36d5704928e4-istio-data\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb\" (UID: \"cbe6d301-322a-4e4b-880b-36d5704928e4\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb" Oct 30 09:18:40.858907 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:40.858886 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbe6d301-322a-4e4b-880b-36d5704928e4-workload-certs\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb\" (UID: \"cbe6d301-322a-4e4b-880b-36d5704928e4\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb" Oct 30 09:18:40.938917 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:40.938873 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-176b6c/distinct-w3-176b6c-kserve-f57b7579c-k4pc5" podUID="28fd4072-399c-4be5-a0d8-1be6966da4af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:40.954826 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:40.954777 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-176b6c/distinct-w3-176b6c-kserve-f57b7579c-k4pc5" Oct 30 09:18:40.973145 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:40.973112 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbe6d301-322a-4e4b-880b-36d5704928e4-istio-data\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb\" (UID: \"cbe6d301-322a-4e4b-880b-36d5704928e4\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb" Oct 30 09:18:41.019008 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:41.018973 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbe6d301-322a-4e4b-880b-36d5704928e4-workload-certs\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb\" (UID: \"cbe6d301-322a-4e4b-880b-36d5704928e4\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb" Oct 30 09:18:41.019334 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:41.019310 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbe6d301-322a-4e4b-880b-36d5704928e4-istio-token\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb\" (UID: \"cbe6d301-322a-4e4b-880b-36d5704928e4\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb" Oct 30 09:18:41.019586 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:41.019567 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbe6d301-322a-4e4b-880b-36d5704928e4-istiod-ca-cert\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb\" (UID: \"cbe6d301-322a-4e4b-880b-36d5704928e4\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb" Oct 30 09:18:41.019789 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:41.019773 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbe6d301-322a-4e4b-880b-36d5704928e4-istio-envoy\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb\" (UID: \"cbe6d301-322a-4e4b-880b-36d5704928e4\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb" Oct 30 09:18:41.019933 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:41.019915 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr9c4\" (UniqueName: \"kubernetes.io/projected/cbe6d301-322a-4e4b-880b-36d5704928e4-kube-api-access-dr9c4\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb\" (UID: \"cbe6d301-322a-4e4b-880b-36d5704928e4\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb" Oct 30 09:18:41.020165 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:41.020145 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe6d301-322a-4e4b-880b-36d5704928e4-workload-socket\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb\" (UID: \"cbe6d301-322a-4e4b-880b-36d5704928e4\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb" Oct 30 09:18:41.020350 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:41.020320 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbe6d301-322a-4e4b-880b-36d5704928e4-istio-podinfo\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb\" (UID: \"cbe6d301-322a-4e4b-880b-36d5704928e4\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb" Oct 30 09:18:41.020564 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:41.020546 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe6d301-322a-4e4b-880b-36d5704928e4-credential-socket\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb\" (UID: \"cbe6d301-322a-4e4b-880b-36d5704928e4\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb" Oct 30 09:18:41.028436 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:40.974935 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbe6d301-322a-4e4b-880b-36d5704928e4-istio-data\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb\" (UID: \"cbe6d301-322a-4e4b-880b-36d5704928e4\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb" Oct 30 09:18:41.066542 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:41.029024 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbe6d301-322a-4e4b-880b-36d5704928e4-workload-certs\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb\" (UID: \"cbe6d301-322a-4e4b-880b-36d5704928e4\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb" Oct 30 09:18:41.095541 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:41.007259 4562 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 09:18:41.095817 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:41.095783 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh" podUID="9491d089-4927-4270-82f2-fecd0f108311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 09:18:41.095939 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:41.043729 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe6d301-322a-4e4b-880b-36d5704928e4-workload-socket\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb\" (UID: \"cbe6d301-322a-4e4b-880b-36d5704928e4\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb" Oct 30 09:18:41.096041 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:41.043487 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbe6d301-322a-4e4b-880b-36d5704928e4-istiod-ca-cert\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb\" (UID: \"cbe6d301-322a-4e4b-880b-36d5704928e4\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb" Oct 30 09:18:41.096136 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:41.007305 4562 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 09:18:41.096246 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:41.096228 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd" podUID="802df8bf-9c24-428b-9d87-96f58054ad68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 09:18:41.173456 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:41.170766 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe6d301-322a-4e4b-880b-36d5704928e4-credential-socket\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb\" (UID: \"cbe6d301-322a-4e4b-880b-36d5704928e4\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb" Oct 30 09:18:41.269407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:41.265511 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb" podUID="a540c86f-c856-42a8-9f73-b5b9b690f42b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:41.304418 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:41.298360 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbe6d301-322a-4e4b-880b-36d5704928e4-istio-podinfo\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb\" (UID: \"cbe6d301-322a-4e4b-880b-36d5704928e4\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb" Oct 30 09:18:41.340407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:41.333684 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbe6d301-322a-4e4b-880b-36d5704928e4-istio-envoy\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb\" (UID: \"cbe6d301-322a-4e4b-880b-36d5704928e4\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb" Oct 30 09:18:41.340407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:41.334347 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr9c4\" (UniqueName: \"kubernetes.io/projected/cbe6d301-322a-4e4b-880b-36d5704928e4-kube-api-access-dr9c4\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb\" (UID: \"cbe6d301-322a-4e4b-880b-36d5704928e4\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb" Oct 30 09:18:41.374600 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:41.367826 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbe6d301-322a-4e4b-880b-36d5704928e4-istio-token\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb\" (UID: \"cbe6d301-322a-4e4b-880b-36d5704928e4\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb" Oct 30 09:18:41.474407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:41.469547 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb" Oct 30 09:18:41.597407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:41.596469 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55" podUID="88be34c3-a9e9-4489-a449-206a99ef1663" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:41.597407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:41.597078 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55" Oct 30 09:18:41.735454 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:41.732951 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-176b6c/distinct-w3-176b6c-kserve-f57b7579c-k4pc5" Oct 30 09:18:41.735454 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:41.735115 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c5447d/distinct-2-w1-c5447d-kserve-676749d548-xlmsf" Oct 30 09:18:41.735454 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:41.735189 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55" Oct 30 09:18:41.746228 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:41.746201 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb" Oct 30 09:18:41.926919 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:41.926889 4562 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 09:18:41.927120 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:41.927089 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd" podUID="802df8bf-9c24-428b-9d87-96f58054ad68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 09:18:41.954861 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:41.954754 4562 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 09:18:41.954861 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:41.954806 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh" podUID="9491d089-4927-4270-82f2-fecd0f108311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 09:18:42.279156 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:42.279107 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:18:42.279286 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:42.279207 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:18:42.282580 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:42.282523 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:18:42.289374 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:42.289347 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb"] Oct 30 09:18:42.786779 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:42.786726 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb" event={"ID":"cbe6d301-322a-4e4b-880b-36d5704928e4","Type":"ContainerStarted","Data":"4e8810ec5af8d99242d5b02d5aca72833b7a7ec6e75474e2a4bae2cefb2b3f35"} Oct 30 09:18:42.787306 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:42.786807 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb" event={"ID":"cbe6d301-322a-4e4b-880b-36d5704928e4","Type":"ContainerStarted","Data":"e7b6f538ba33a36c058042e740b31e170626c3ca030c9c89577c40352ed09000"} Oct 30 09:18:42.876403 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:42.874348 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb" podStartSLOduration=3.874329915 podStartE2EDuration="3.874329915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:18:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:18:42.859978574 +0000 UTC m=+1483.121539564" watchObservedRunningTime="2026-10-30 09:18:42.874329915 +0000 UTC m=+1483.135890904" Oct 30 09:18:42.931407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:42.926617 4562 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 09:18:42.931407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:42.926681 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd" podUID="802df8bf-9c24-428b-9d87-96f58054ad68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 09:18:43.488049 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:43.487441 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb" Oct 30 09:18:43.488049 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:43.487571 4562 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 09:18:43.488049 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:43.487614 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb" podUID="cbe6d301-322a-4e4b-880b-36d5704928e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 09:18:43.935502 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:43.935414 4562 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 09:18:43.936069 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:43.936030 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd" podUID="802df8bf-9c24-428b-9d87-96f58054ad68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 09:18:43.985527 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:43.985490 4562 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 09:18:43.985786 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:43.985757 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh" podUID="9491d089-4927-4270-82f2-fecd0f108311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:44.485748 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:44.485712 4562 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 09:18:44.486155 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:44.486107 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb" podUID="cbe6d301-322a-4e4b-880b-36d5704928e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 09:18:44.927133 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:44.927045 4562 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 09:18:44.927133 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:44.927107 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd" podUID="802df8bf-9c24-428b-9d87-96f58054ad68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 09:18:45.011689 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:45.011627 4562 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 09:18:45.012269 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:45.012233 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh" podUID="9491d089-4927-4270-82f2-fecd0f108311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:45.213770 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:45.213734 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh" Oct 30 09:18:45.214174 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:45.214148 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh" Oct 30 09:18:45.215156 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:45.215135 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh" Oct 30 09:18:45.483471 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:45.483333 4562 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 09:18:45.483471 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:45.483414 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb" podUID="cbe6d301-322a-4e4b-880b-36d5704928e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 09:18:46.469406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:46.467331 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd" Oct 30 09:18:46.469406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:46.468140 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd" Oct 30 09:18:46.469406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:46.469140 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd" Oct 30 09:18:46.483100 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:46.483066 4562 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 09:18:46.483282 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:46.483116 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb" podUID="cbe6d301-322a-4e4b-880b-36d5704928e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 09:18:47.483322 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:47.482876 4562 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 09:18:47.483322 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:47.482937 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb" podUID="cbe6d301-322a-4e4b-880b-36d5704928e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 09:18:47.706743 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:47.706694 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl"] Oct 30 09:18:47.712911 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:47.712887 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl" Oct 30 09:18:47.732012 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:47.731945 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl"] Oct 30 09:18:47.870863 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:47.870757 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21455e45-a059-4b1a-81db-c4bba40c8442-istiod-ca-cert\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl\" (UID: \"21455e45-a059-4b1a-81db-c4bba40c8442\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl" Oct 30 09:18:47.870863 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:47.870825 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21455e45-a059-4b1a-81db-c4bba40c8442-credential-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl\" (UID: \"21455e45-a059-4b1a-81db-c4bba40c8442\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl" Oct 30 09:18:47.871110 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:47.870876 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21455e45-a059-4b1a-81db-c4bba40c8442-workload-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl\" (UID: \"21455e45-a059-4b1a-81db-c4bba40c8442\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl" Oct 30 09:18:47.871110 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:47.870949 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21455e45-a059-4b1a-81db-c4bba40c8442-istio-token\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl\" (UID: \"21455e45-a059-4b1a-81db-c4bba40c8442\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl" Oct 30 09:18:47.871110 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:47.870987 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21455e45-a059-4b1a-81db-c4bba40c8442-istio-data\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl\" (UID: \"21455e45-a059-4b1a-81db-c4bba40c8442\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl" Oct 30 09:18:47.871110 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:47.871031 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21455e45-a059-4b1a-81db-c4bba40c8442-workload-certs\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl\" (UID: \"21455e45-a059-4b1a-81db-c4bba40c8442\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl" Oct 30 09:18:47.871110 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:47.871104 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21455e45-a059-4b1a-81db-c4bba40c8442-istio-envoy\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl\" (UID: \"21455e45-a059-4b1a-81db-c4bba40c8442\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl" Oct 30 09:18:47.871422 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:47.871143 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21455e45-a059-4b1a-81db-c4bba40c8442-istio-podinfo\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl\" (UID: \"21455e45-a059-4b1a-81db-c4bba40c8442\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl" Oct 30 09:18:47.871422 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:47.871216 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h85mp\" (UniqueName: \"kubernetes.io/projected/21455e45-a059-4b1a-81db-c4bba40c8442-kube-api-access-h85mp\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl\" (UID: \"21455e45-a059-4b1a-81db-c4bba40c8442\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl" Oct 30 09:18:47.972148 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:47.972109 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21455e45-a059-4b1a-81db-c4bba40c8442-workload-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl\" (UID: \"21455e45-a059-4b1a-81db-c4bba40c8442\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl" Oct 30 09:18:47.972339 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:47.972180 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21455e45-a059-4b1a-81db-c4bba40c8442-istio-token\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl\" (UID: \"21455e45-a059-4b1a-81db-c4bba40c8442\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl" Oct 30 09:18:47.972339 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:47.972215 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21455e45-a059-4b1a-81db-c4bba40c8442-istio-data\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl\" (UID: \"21455e45-a059-4b1a-81db-c4bba40c8442\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl" Oct 30 09:18:47.972339 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:47.972243 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21455e45-a059-4b1a-81db-c4bba40c8442-workload-certs\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl\" (UID: \"21455e45-a059-4b1a-81db-c4bba40c8442\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl" Oct 30 09:18:47.972339 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:47.972315 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21455e45-a059-4b1a-81db-c4bba40c8442-istio-envoy\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl\" (UID: \"21455e45-a059-4b1a-81db-c4bba40c8442\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl" Oct 30 09:18:47.972603 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:47.972343 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21455e45-a059-4b1a-81db-c4bba40c8442-istio-podinfo\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl\" (UID: \"21455e45-a059-4b1a-81db-c4bba40c8442\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl" Oct 30 09:18:47.972603 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:47.972455 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h85mp\" (UniqueName: \"kubernetes.io/projected/21455e45-a059-4b1a-81db-c4bba40c8442-kube-api-access-h85mp\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl\" (UID: \"21455e45-a059-4b1a-81db-c4bba40c8442\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl" Oct 30 09:18:47.972603 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:47.972511 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21455e45-a059-4b1a-81db-c4bba40c8442-istiod-ca-cert\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl\" (UID: \"21455e45-a059-4b1a-81db-c4bba40c8442\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl" Oct 30 09:18:47.972603 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:47.972551 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21455e45-a059-4b1a-81db-c4bba40c8442-credential-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl\" (UID: \"21455e45-a059-4b1a-81db-c4bba40c8442\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl" Oct 30 09:18:47.972816 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:47.972645 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21455e45-a059-4b1a-81db-c4bba40c8442-workload-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl\" (UID: \"21455e45-a059-4b1a-81db-c4bba40c8442\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl" Oct 30 09:18:47.972816 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:47.972660 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21455e45-a059-4b1a-81db-c4bba40c8442-workload-certs\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl\" (UID: \"21455e45-a059-4b1a-81db-c4bba40c8442\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl" Oct 30 09:18:47.972816 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:47.972732 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21455e45-a059-4b1a-81db-c4bba40c8442-istio-data\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl\" (UID: \"21455e45-a059-4b1a-81db-c4bba40c8442\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl" Oct 30 09:18:47.973057 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:47.973035 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21455e45-a059-4b1a-81db-c4bba40c8442-credential-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl\" (UID: \"21455e45-a059-4b1a-81db-c4bba40c8442\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl" Oct 30 09:18:47.973613 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:47.973586 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21455e45-a059-4b1a-81db-c4bba40c8442-istiod-ca-cert\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl\" (UID: \"21455e45-a059-4b1a-81db-c4bba40c8442\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl" Oct 30 09:18:47.975072 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:47.975042 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21455e45-a059-4b1a-81db-c4bba40c8442-istio-envoy\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl\" (UID: \"21455e45-a059-4b1a-81db-c4bba40c8442\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl" Oct 30 09:18:47.975193 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:47.975088 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21455e45-a059-4b1a-81db-c4bba40c8442-istio-podinfo\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl\" (UID: \"21455e45-a059-4b1a-81db-c4bba40c8442\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl" Oct 30 09:18:48.003070 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:48.003044 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21455e45-a059-4b1a-81db-c4bba40c8442-istio-token\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl\" (UID: \"21455e45-a059-4b1a-81db-c4bba40c8442\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl" Oct 30 09:18:48.003230 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:48.003208 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h85mp\" (UniqueName: \"kubernetes.io/projected/21455e45-a059-4b1a-81db-c4bba40c8442-kube-api-access-h85mp\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl\" (UID: \"21455e45-a059-4b1a-81db-c4bba40c8442\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl" Oct 30 09:18:48.056001 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:48.055964 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl" Oct 30 09:18:48.483190 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:48.483155 4562 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 09:18:48.483412 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:48.483215 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb" podUID="cbe6d301-322a-4e4b-880b-36d5704928e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 09:18:48.497563 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:18:48.497532 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21455e45_a059_4b1a_81db_c4bba40c8442.slice/crio-2e1d4b80b56b633a63951d5c86fc5d85fb41cbfd5a62d6fe034b46e65862d1e3 WatchSource:0}: Error finding container 2e1d4b80b56b633a63951d5c86fc5d85fb41cbfd5a62d6fe034b46e65862d1e3: Status 404 returned error can't find the container with id 2e1d4b80b56b633a63951d5c86fc5d85fb41cbfd5a62d6fe034b46e65862d1e3 Oct 30 09:18:48.500335 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:48.500307 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl"] Oct 30 09:18:48.500547 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:48.500512 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:18:48.500615 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:48.500597 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:18:48.500655 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:48.500644 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:18:48.844262 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:48.844153 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl" event={"ID":"21455e45-a059-4b1a-81db-c4bba40c8442","Type":"ContainerStarted","Data":"1dc51e04524037602c25b3a193f6613599961ec7d4b01a772fb093a62f172415"} Oct 30 09:18:48.844262 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:48.844223 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl" event={"ID":"21455e45-a059-4b1a-81db-c4bba40c8442","Type":"ContainerStarted","Data":"2e1d4b80b56b633a63951d5c86fc5d85fb41cbfd5a62d6fe034b46e65862d1e3"} Oct 30 09:18:48.920524 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:48.920471 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl" podStartSLOduration=1.920457318 podStartE2EDuration="1.920457318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:18:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:18:48.91640768 +0000 UTC m=+1489.177968695" watchObservedRunningTime="2026-10-30 09:18:48.920457318 +0000 UTC m=+1489.182018373" Oct 30 09:18:49.057607 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:49.057575 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl" Oct 30 09:18:49.059250 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:49.059223 4562 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 09:18:49.059409 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:49.059277 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl" podUID="21455e45-a059-4b1a-81db-c4bba40c8442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 09:18:49.913974 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:49.913937 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb" Oct 30 09:18:49.915050 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:49.915025 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb" Oct 30 09:18:49.915997 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:49.915968 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb" Oct 30 09:18:50.059518 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:50.058712 4562 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 09:18:50.059518 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:50.058773 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl" podUID="21455e45-a059-4b1a-81db-c4bba40c8442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 09:18:51.069428 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:51.064695 4562 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 09:18:51.069428 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:51.064752 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl" podUID="21455e45-a059-4b1a-81db-c4bba40c8442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 09:18:52.069574 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:52.069532 4562 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 09:18:52.069978 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:52.069604 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl" podUID="21455e45-a059-4b1a-81db-c4bba40c8442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 09:18:53.058735 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:53.058690 4562 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 09:18:53.058993 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:53.058964 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl" podUID="21455e45-a059-4b1a-81db-c4bba40c8442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 09:18:54.060349 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:54.060316 4562 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 09:18:54.060947 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:54.060918 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl" podUID="21455e45-a059-4b1a-81db-c4bba40c8442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 09:18:55.058365 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:55.058322 4562 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 09:18:55.058547 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:55.058419 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl" podUID="21455e45-a059-4b1a-81db-c4bba40c8442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 09:18:57.084503 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:57.084463 4562 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 09:18:57.085080 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:57.085046 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl" podUID="21455e45-a059-4b1a-81db-c4bba40c8442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:57.232337 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:57.230686 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl" Oct 30 09:18:57.232337 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:57.232179 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl" Oct 30 09:18:57.253341 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:57.253311 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl" Oct 30 09:18:57.754407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:57.744161 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55" podUID="88be34c3-a9e9-4489-a449-206a99ef1663" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 09:18:58.772525 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:58.772468 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/distinct-2-w1-c5447d-kserve-676749d548-xlmsf" podUID="c9ff897f-c520-4d0f-8820-e185b1d47dac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:58.805749 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:58.791248 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb" podUID="a540c86f-c856-42a8-9f73-b5b9b690f42b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Oct 30 09:18:58.863331 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:58.828434 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55" podUID="88be34c3-a9e9-4489-a449-206a99ef1663" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 09:18:58.863331 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:18:58.841605 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:18:58.863331 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:58.843902 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55" Oct 30 09:18:58.863331 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:58.845212 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-176b6c/distinct-w3-176b6c-kserve-f57b7579c-k4pc5" podUID="28fd4072-399c-4be5-a0d8-1be6966da4af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:18:58.930816 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:18:58.930774 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55" Oct 30 09:19:00.027517 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:00.027484 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6q7n_1f51db5a-5720-4dea-ad02-f336b35d43b1/ovn-acl-logging/0.log" Oct 30 09:19:00.044973 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:00.044949 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6q7n_1f51db5a-5720-4dea-ad02-f336b35d43b1/ovn-acl-logging/0.log" Oct 30 09:19:04.284071 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:04.284032 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw"] Oct 30 09:19:04.291758 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:04.291728 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw" Oct 30 09:19:04.312732 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:04.312706 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw"] Oct 30 09:19:04.339456 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:04.339429 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc12bf1-47c5-4d5e-b295-079f027d8019-workload-socket\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw\" (UID: \"ecc12bf1-47c5-4d5e-b295-079f027d8019\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw" Oct 30 09:19:04.339583 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:04.339468 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb6rz\" (UniqueName: \"kubernetes.io/projected/ecc12bf1-47c5-4d5e-b295-079f027d8019-kube-api-access-bb6rz\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw\" (UID: \"ecc12bf1-47c5-4d5e-b295-079f027d8019\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw" Oct 30 09:19:04.339583 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:04.339492 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc12bf1-47c5-4d5e-b295-079f027d8019-credential-socket\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw\" (UID: \"ecc12bf1-47c5-4d5e-b295-079f027d8019\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw" Oct 30 09:19:04.339678 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:04.339647 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecc12bf1-47c5-4d5e-b295-079f027d8019-istio-podinfo\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw\" (UID: \"ecc12bf1-47c5-4d5e-b295-079f027d8019\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw" Oct 30 09:19:04.339714 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:04.339699 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecc12bf1-47c5-4d5e-b295-079f027d8019-workload-certs\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw\" (UID: \"ecc12bf1-47c5-4d5e-b295-079f027d8019\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw" Oct 30 09:19:04.339764 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:04.339728 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecc12bf1-47c5-4d5e-b295-079f027d8019-istio-envoy\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw\" (UID: \"ecc12bf1-47c5-4d5e-b295-079f027d8019\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw" Oct 30 09:19:04.339833 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:04.339821 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecc12bf1-47c5-4d5e-b295-079f027d8019-istiod-ca-cert\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw\" (UID: \"ecc12bf1-47c5-4d5e-b295-079f027d8019\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw" Oct 30 09:19:04.339871 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:04.339847 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecc12bf1-47c5-4d5e-b295-079f027d8019-istio-data\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw\" (UID: \"ecc12bf1-47c5-4d5e-b295-079f027d8019\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw" Oct 30 09:19:04.339921 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:04.339881 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecc12bf1-47c5-4d5e-b295-079f027d8019-istio-token\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw\" (UID: \"ecc12bf1-47c5-4d5e-b295-079f027d8019\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw" Oct 30 09:19:04.440919 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:04.440882 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecc12bf1-47c5-4d5e-b295-079f027d8019-istio-token\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw\" (UID: \"ecc12bf1-47c5-4d5e-b295-079f027d8019\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw" Oct 30 09:19:04.441100 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:04.441071 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc12bf1-47c5-4d5e-b295-079f027d8019-workload-socket\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw\" (UID: \"ecc12bf1-47c5-4d5e-b295-079f027d8019\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw" Oct 30 09:19:04.441166 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:04.441118 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb6rz\" (UniqueName: \"kubernetes.io/projected/ecc12bf1-47c5-4d5e-b295-079f027d8019-kube-api-access-bb6rz\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw\" (UID: \"ecc12bf1-47c5-4d5e-b295-079f027d8019\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw" Oct 30 09:19:04.441166 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:04.441146 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc12bf1-47c5-4d5e-b295-079f027d8019-credential-socket\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw\" (UID: \"ecc12bf1-47c5-4d5e-b295-079f027d8019\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw" Oct 30 09:19:04.441269 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:04.441211 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecc12bf1-47c5-4d5e-b295-079f027d8019-istio-podinfo\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw\" (UID: \"ecc12bf1-47c5-4d5e-b295-079f027d8019\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw" Oct 30 09:19:04.441269 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:04.441251 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecc12bf1-47c5-4d5e-b295-079f027d8019-workload-certs\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw\" (UID: \"ecc12bf1-47c5-4d5e-b295-079f027d8019\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw" Oct 30 09:19:04.441393 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:04.441292 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecc12bf1-47c5-4d5e-b295-079f027d8019-istio-envoy\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw\" (UID: \"ecc12bf1-47c5-4d5e-b295-079f027d8019\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw" Oct 30 09:19:04.441393 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:04.441360 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecc12bf1-47c5-4d5e-b295-079f027d8019-istiod-ca-cert\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw\" (UID: \"ecc12bf1-47c5-4d5e-b295-079f027d8019\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw" Oct 30 09:19:04.441518 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:04.441443 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecc12bf1-47c5-4d5e-b295-079f027d8019-istio-data\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw\" (UID: \"ecc12bf1-47c5-4d5e-b295-079f027d8019\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw" Oct 30 09:19:04.441608 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:04.441584 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc12bf1-47c5-4d5e-b295-079f027d8019-workload-socket\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw\" (UID: \"ecc12bf1-47c5-4d5e-b295-079f027d8019\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw" Oct 30 09:19:04.441663 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:04.441607 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc12bf1-47c5-4d5e-b295-079f027d8019-credential-socket\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw\" (UID: \"ecc12bf1-47c5-4d5e-b295-079f027d8019\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw" Oct 30 09:19:04.441797 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:04.441769 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecc12bf1-47c5-4d5e-b295-079f027d8019-istio-data\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw\" (UID: \"ecc12bf1-47c5-4d5e-b295-079f027d8019\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw" Oct 30 09:19:04.441925 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:04.441853 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecc12bf1-47c5-4d5e-b295-079f027d8019-workload-certs\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw\" (UID: \"ecc12bf1-47c5-4d5e-b295-079f027d8019\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw" Oct 30 09:19:04.442246 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:04.442223 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecc12bf1-47c5-4d5e-b295-079f027d8019-istiod-ca-cert\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw\" (UID: \"ecc12bf1-47c5-4d5e-b295-079f027d8019\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw" Oct 30 09:19:04.444049 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:04.444030 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecc12bf1-47c5-4d5e-b295-079f027d8019-istio-envoy\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw\" (UID: \"ecc12bf1-47c5-4d5e-b295-079f027d8019\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw" Oct 30 09:19:04.444368 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:04.444349 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecc12bf1-47c5-4d5e-b295-079f027d8019-istio-podinfo\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw\" (UID: \"ecc12bf1-47c5-4d5e-b295-079f027d8019\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw" Oct 30 09:19:04.466796 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:04.466776 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecc12bf1-47c5-4d5e-b295-079f027d8019-istio-token\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw\" (UID: \"ecc12bf1-47c5-4d5e-b295-079f027d8019\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw" Oct 30 09:19:04.468572 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:04.468555 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb6rz\" (UniqueName: \"kubernetes.io/projected/ecc12bf1-47c5-4d5e-b295-079f027d8019-kube-api-access-bb6rz\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw\" (UID: \"ecc12bf1-47c5-4d5e-b295-079f027d8019\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw" Oct 30 09:19:04.608262 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:04.608172 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw" Oct 30 09:19:04.933507 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:19:04.933476 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecc12bf1_47c5_4d5e_b295_079f027d8019.slice/crio-4c3cfd6be84834cd152104098304003f668de00a7f760021ced3b68746176af3 WatchSource:0}: Error finding container 4c3cfd6be84834cd152104098304003f668de00a7f760021ced3b68746176af3: Status 404 returned error can't find the container with id 4c3cfd6be84834cd152104098304003f668de00a7f760021ced3b68746176af3 Oct 30 09:19:04.934299 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:04.934273 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw"] Oct 30 09:19:04.936057 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:04.936029 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:19:04.936129 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:04.936089 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:19:04.936129 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:04.936120 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:19:05.208958 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:05.208923 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw" event={"ID":"ecc12bf1-47c5-4d5e-b295-079f027d8019","Type":"ContainerStarted","Data":"c2b900f0f688c57b8463d6b13a51e3f17e8855adb654dd1deda65057d159acb6"} Oct 30 09:19:05.209129 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:05.208984 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw" event={"ID":"ecc12bf1-47c5-4d5e-b295-079f027d8019","Type":"ContainerStarted","Data":"4c3cfd6be84834cd152104098304003f668de00a7f760021ced3b68746176af3"} Oct 30 09:19:05.608588 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:05.608505 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw" Oct 30 09:19:05.610839 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:05.610810 4562 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 09:19:05.610972 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:05.610859 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw" podUID="ecc12bf1-47c5-4d5e-b295-079f027d8019" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 09:19:06.608927 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:06.608890 4562 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 09:19:06.609332 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:06.608939 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw" podUID="ecc12bf1-47c5-4d5e-b295-079f027d8019" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 09:19:07.609173 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:07.609137 4562 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 09:19:07.609692 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:07.609196 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw" podUID="ecc12bf1-47c5-4d5e-b295-079f027d8019" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 09:19:08.609146 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:08.609112 4562 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 09:19:08.609303 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:08.609163 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw" podUID="ecc12bf1-47c5-4d5e-b295-079f027d8019" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 09:19:09.609551 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:09.609516 4562 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 09:19:09.609966 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:09.609579 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw" podUID="ecc12bf1-47c5-4d5e-b295-079f027d8019" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 09:19:10.609433 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:10.609397 4562 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 09:19:10.609593 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:10.609459 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw" podUID="ecc12bf1-47c5-4d5e-b295-079f027d8019" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 09:19:11.614406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:11.613945 4562 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 09:19:11.614406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:11.614072 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw" podUID="ecc12bf1-47c5-4d5e-b295-079f027d8019" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 09:19:12.614372 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:12.614342 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw" Oct 30 09:19:12.614876 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:12.614857 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw" Oct 30 09:19:12.615625 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:12.615606 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw" Oct 30 09:19:12.682250 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:12.682205 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw" podStartSLOduration=8.682193992 podStartE2EDuration="8.682193992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:19:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:19:05.275651247 +0000 UTC m=+1505.537212248" watchObservedRunningTime="2026-10-30 09:19:12.682193992 +0000 UTC m=+1512.943754979" Oct 30 09:19:17.536204 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:17.536164 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8eada0cf-7c5b9fd4ff-9jzlk"] Oct 30 09:19:17.541943 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:17.541921 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8eada0cf-7c5b9fd4ff-9jzlk" Oct 30 09:19:17.574190 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:17.574162 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-8eada0cf-dockercfg-k59xw\"" Oct 30 09:19:17.574348 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:17.574205 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-8eada0cf\"" Oct 30 09:19:17.577544 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:17.577522 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8eada0cf-7c5b9fd4ff-9jzlk"] Oct 30 09:19:17.685672 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:17.685637 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q8j5\" (UniqueName: \"kubernetes.io/projected/58bd2646-2220-4f53-9133-c27c4f8b0c15-kube-api-access-4q8j5\") pod \"payload-pre-processing-e2e-mt-8eada0cf-7c5b9fd4ff-9jzlk\" (UID: \"58bd2646-2220-4f53-9133-c27c4f8b0c15\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8eada0cf-7c5b9fd4ff-9jzlk" Oct 30 09:19:17.685853 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:17.685804 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58bd2646-2220-4f53-9133-c27c4f8b0c15-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8eada0cf-7c5b9fd4ff-9jzlk\" (UID: \"58bd2646-2220-4f53-9133-c27c4f8b0c15\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8eada0cf-7c5b9fd4ff-9jzlk" Oct 30 09:19:17.691108 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:17.691085 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8eada0cf-5884b8bb8-lp9q6"] Oct 30 09:19:17.695215 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:17.695195 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8eada0cf-5884b8bb8-lp9q6" Oct 30 09:19:17.751954 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:17.750451 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8eada0cf-5884b8bb8-lp9q6"] Oct 30 09:19:17.916992 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:17.916877 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q8j5\" (UniqueName: \"kubernetes.io/projected/58bd2646-2220-4f53-9133-c27c4f8b0c15-kube-api-access-4q8j5\") pod \"payload-pre-processing-e2e-mt-8eada0cf-7c5b9fd4ff-9jzlk\" (UID: \"58bd2646-2220-4f53-9133-c27c4f8b0c15\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8eada0cf-7c5b9fd4ff-9jzlk" Oct 30 09:19:17.917302 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:17.917265 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58bd2646-2220-4f53-9133-c27c4f8b0c15-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8eada0cf-7c5b9fd4ff-9jzlk\" (UID: \"58bd2646-2220-4f53-9133-c27c4f8b0c15\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8eada0cf-7c5b9fd4ff-9jzlk" Oct 30 09:19:17.917602 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:17.917567 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79e02fb4-8a91-48ee-8fce-7fb16274008d-plugins-config-volume\") pod \"payload-processing-e2e-mt-8eada0cf-5884b8bb8-lp9q6\" (UID: \"79e02fb4-8a91-48ee-8fce-7fb16274008d\") " pod="openshift-ingress/payload-processing-e2e-mt-8eada0cf-5884b8bb8-lp9q6" Oct 30 09:19:17.917756 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:17.917739 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pjtn\" (UniqueName: \"kubernetes.io/projected/79e02fb4-8a91-48ee-8fce-7fb16274008d-kube-api-access-6pjtn\") pod \"payload-processing-e2e-mt-8eada0cf-5884b8bb8-lp9q6\" (UID: \"79e02fb4-8a91-48ee-8fce-7fb16274008d\") " pod="openshift-ingress/payload-processing-e2e-mt-8eada0cf-5884b8bb8-lp9q6" Oct 30 09:19:17.918564 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:17.918526 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58bd2646-2220-4f53-9133-c27c4f8b0c15-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8eada0cf-7c5b9fd4ff-9jzlk\" (UID: \"58bd2646-2220-4f53-9133-c27c4f8b0c15\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8eada0cf-7c5b9fd4ff-9jzlk" Oct 30 09:19:17.950172 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:17.950134 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q8j5\" (UniqueName: \"kubernetes.io/projected/58bd2646-2220-4f53-9133-c27c4f8b0c15-kube-api-access-4q8j5\") pod \"payload-pre-processing-e2e-mt-8eada0cf-7c5b9fd4ff-9jzlk\" (UID: \"58bd2646-2220-4f53-9133-c27c4f8b0c15\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8eada0cf-7c5b9fd4ff-9jzlk" Oct 30 09:19:18.018678 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:18.018645 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79e02fb4-8a91-48ee-8fce-7fb16274008d-plugins-config-volume\") pod \"payload-processing-e2e-mt-8eada0cf-5884b8bb8-lp9q6\" (UID: \"79e02fb4-8a91-48ee-8fce-7fb16274008d\") " pod="openshift-ingress/payload-processing-e2e-mt-8eada0cf-5884b8bb8-lp9q6" Oct 30 09:19:18.018678 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:18.018682 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pjtn\" (UniqueName: \"kubernetes.io/projected/79e02fb4-8a91-48ee-8fce-7fb16274008d-kube-api-access-6pjtn\") pod \"payload-processing-e2e-mt-8eada0cf-5884b8bb8-lp9q6\" (UID: \"79e02fb4-8a91-48ee-8fce-7fb16274008d\") " pod="openshift-ingress/payload-processing-e2e-mt-8eada0cf-5884b8bb8-lp9q6" Oct 30 09:19:18.019418 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:18.019358 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79e02fb4-8a91-48ee-8fce-7fb16274008d-plugins-config-volume\") pod \"payload-processing-e2e-mt-8eada0cf-5884b8bb8-lp9q6\" (UID: \"79e02fb4-8a91-48ee-8fce-7fb16274008d\") " pod="openshift-ingress/payload-processing-e2e-mt-8eada0cf-5884b8bb8-lp9q6" Oct 30 09:19:18.049039 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:18.049013 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pjtn\" (UniqueName: \"kubernetes.io/projected/79e02fb4-8a91-48ee-8fce-7fb16274008d-kube-api-access-6pjtn\") pod \"payload-processing-e2e-mt-8eada0cf-5884b8bb8-lp9q6\" (UID: \"79e02fb4-8a91-48ee-8fce-7fb16274008d\") " pod="openshift-ingress/payload-processing-e2e-mt-8eada0cf-5884b8bb8-lp9q6" Oct 30 09:19:18.157803 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:18.157770 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8eada0cf-7c5b9fd4ff-9jzlk" Oct 30 09:19:18.309075 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:18.309046 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8eada0cf-5884b8bb8-lp9q6" Oct 30 09:19:18.482785 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:19:18.482749 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58bd2646_2220_4f53_9133_c27c4f8b0c15.slice/crio-3531443a80f805583d0e7c52b21fe5567973a6cebabeecc435f8f363c0927bf9 WatchSource:0}: Error finding container 3531443a80f805583d0e7c52b21fe5567973a6cebabeecc435f8f363c0927bf9: Status 404 returned error can't find the container with id 3531443a80f805583d0e7c52b21fe5567973a6cebabeecc435f8f363c0927bf9 Oct 30 09:19:18.483290 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:18.483262 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8eada0cf-7c5b9fd4ff-9jzlk"] Oct 30 09:19:18.737484 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:18.729083 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8eada0cf-5884b8bb8-lp9q6"] Oct 30 09:19:18.752447 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:19:18.749717 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79e02fb4_8a91_48ee_8fce_7fb16274008d.slice/crio-2698839250a51127eac65e63f50c14c3bcd5f385cf1236dec0c70823a7247ee2 WatchSource:0}: Error finding container 2698839250a51127eac65e63f50c14c3bcd5f385cf1236dec0c70823a7247ee2: Status 404 returned error can't find the container with id 2698839250a51127eac65e63f50c14c3bcd5f385cf1236dec0c70823a7247ee2 Oct 30 09:19:19.342800 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:19.342747 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8eada0cf-7c5b9fd4ff-9jzlk" event={"ID":"58bd2646-2220-4f53-9133-c27c4f8b0c15","Type":"ContainerStarted","Data":"bd03c52899d66ab056f94c1cd59c14d1a06f38c7fcf5f68d9a508d0ccb5611d3"} Oct 30 09:19:19.342995 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:19.342881 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8eada0cf-7c5b9fd4ff-9jzlk" event={"ID":"58bd2646-2220-4f53-9133-c27c4f8b0c15","Type":"ContainerStarted","Data":"3531443a80f805583d0e7c52b21fe5567973a6cebabeecc435f8f363c0927bf9"} Oct 30 09:19:19.342995 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:19.342920 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-8eada0cf-7c5b9fd4ff-9jzlk" Oct 30 09:19:19.344880 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:19.344854 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8eada0cf-5884b8bb8-lp9q6" event={"ID":"79e02fb4-8a91-48ee-8fce-7fb16274008d","Type":"ContainerStarted","Data":"7fc14de99612c11ff44a5329298c52a2cd6a0c00530c64ea0ca84993dd95275b"} Oct 30 09:19:19.345021 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:19.344895 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8eada0cf-5884b8bb8-lp9q6" event={"ID":"79e02fb4-8a91-48ee-8fce-7fb16274008d","Type":"ContainerStarted","Data":"2698839250a51127eac65e63f50c14c3bcd5f385cf1236dec0c70823a7247ee2"} Oct 30 09:19:19.345021 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:19.344948 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-8eada0cf-5884b8bb8-lp9q6" Oct 30 09:19:19.394302 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:19.394202 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-8eada0cf-7c5b9fd4ff-9jzlk" podStartSLOduration=2.39418813 podStartE2EDuration="2.39418813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:19:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:19:19.388662862 +0000 UTC m=+1519.650223862" watchObservedRunningTime="2026-10-30 09:19:19.39418813 +0000 UTC m=+1519.655749118" Oct 30 09:19:19.434202 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:19.434146 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-8eada0cf-5884b8bb8-lp9q6" podStartSLOduration=2.434134793 podStartE2EDuration="2.434134793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:19:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:19:19.433662959 +0000 UTC m=+1519.695223948" watchObservedRunningTime="2026-10-30 09:19:19.434134793 +0000 UTC m=+1519.695695780" Oct 30 09:19:20.750408 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:20.743944 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-176b6c/distinct-w3-176b6c-kserve-f57b7579c-k4pc5" podUID="28fd4072-399c-4be5-a0d8-1be6966da4af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:20.758406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:20.757472 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb" podUID="a540c86f-c856-42a8-9f73-b5b9b690f42b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:20.865408 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:20.856150 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd"] Oct 30 09:19:20.886807 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:20.872747 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd" Oct 30 09:19:20.929922 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:20.929645 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd"] Oct 30 09:19:21.081643 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:21.007655 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ed52c04-442a-4a04-8a72-c40062a5e3ba-istio-podinfo\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd\" (UID: \"9ed52c04-442a-4a04-8a72-c40062a5e3ba\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd" Oct 30 09:19:21.329790 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:21.329751 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gstff\" (UniqueName: \"kubernetes.io/projected/9ed52c04-442a-4a04-8a72-c40062a5e3ba-kube-api-access-gstff\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd\" (UID: \"9ed52c04-442a-4a04-8a72-c40062a5e3ba\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd" Oct 30 09:19:21.330334 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:21.330308 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ed52c04-442a-4a04-8a72-c40062a5e3ba-workload-socket\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd\" (UID: \"9ed52c04-442a-4a04-8a72-c40062a5e3ba\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd" Oct 30 09:19:21.334613 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:21.334546 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ed52c04-442a-4a04-8a72-c40062a5e3ba-istiod-ca-cert\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd\" (UID: \"9ed52c04-442a-4a04-8a72-c40062a5e3ba\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd" Oct 30 09:19:21.342412 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:21.334850 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ed52c04-442a-4a04-8a72-c40062a5e3ba-istio-envoy\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd\" (UID: \"9ed52c04-442a-4a04-8a72-c40062a5e3ba\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd" Oct 30 09:19:21.342848 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:21.342821 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ed52c04-442a-4a04-8a72-c40062a5e3ba-istio-token\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd\" (UID: \"9ed52c04-442a-4a04-8a72-c40062a5e3ba\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd" Oct 30 09:19:21.368858 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:21.368820 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ed52c04-442a-4a04-8a72-c40062a5e3ba-credential-socket\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd\" (UID: \"9ed52c04-442a-4a04-8a72-c40062a5e3ba\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd" Oct 30 09:19:21.381589 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:21.381555 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ed52c04-442a-4a04-8a72-c40062a5e3ba-workload-certs\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd\" (UID: \"9ed52c04-442a-4a04-8a72-c40062a5e3ba\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd" Oct 30 09:19:21.381943 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:21.381917 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ed52c04-442a-4a04-8a72-c40062a5e3ba-istio-data\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd\" (UID: \"9ed52c04-442a-4a04-8a72-c40062a5e3ba\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd" Oct 30 09:19:21.394407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:21.032478 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55" podUID="88be34c3-a9e9-4489-a449-206a99ef1663" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:21.506415 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:21.505519 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ed52c04-442a-4a04-8a72-c40062a5e3ba-istio-podinfo\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd\" (UID: \"9ed52c04-442a-4a04-8a72-c40062a5e3ba\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd" Oct 30 09:19:21.506415 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:21.505576 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gstff\" (UniqueName: \"kubernetes.io/projected/9ed52c04-442a-4a04-8a72-c40062a5e3ba-kube-api-access-gstff\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd\" (UID: \"9ed52c04-442a-4a04-8a72-c40062a5e3ba\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd" Oct 30 09:19:21.506415 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:21.505623 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ed52c04-442a-4a04-8a72-c40062a5e3ba-workload-socket\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd\" (UID: \"9ed52c04-442a-4a04-8a72-c40062a5e3ba\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd" Oct 30 09:19:21.506415 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:21.505672 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ed52c04-442a-4a04-8a72-c40062a5e3ba-istiod-ca-cert\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd\" (UID: \"9ed52c04-442a-4a04-8a72-c40062a5e3ba\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd" Oct 30 09:19:21.506415 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:21.505715 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ed52c04-442a-4a04-8a72-c40062a5e3ba-istio-envoy\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd\" (UID: \"9ed52c04-442a-4a04-8a72-c40062a5e3ba\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd" Oct 30 09:19:21.506415 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:21.505742 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ed52c04-442a-4a04-8a72-c40062a5e3ba-istio-token\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd\" (UID: \"9ed52c04-442a-4a04-8a72-c40062a5e3ba\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd" Oct 30 09:19:21.506415 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:21.505862 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ed52c04-442a-4a04-8a72-c40062a5e3ba-credential-socket\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd\" (UID: \"9ed52c04-442a-4a04-8a72-c40062a5e3ba\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd" Oct 30 09:19:21.506415 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:21.505887 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ed52c04-442a-4a04-8a72-c40062a5e3ba-workload-certs\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd\" (UID: \"9ed52c04-442a-4a04-8a72-c40062a5e3ba\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd" Oct 30 09:19:21.506415 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:21.505943 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ed52c04-442a-4a04-8a72-c40062a5e3ba-istio-data\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd\" (UID: \"9ed52c04-442a-4a04-8a72-c40062a5e3ba\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd" Oct 30 09:19:21.507065 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:21.506473 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ed52c04-442a-4a04-8a72-c40062a5e3ba-istio-data\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd\" (UID: \"9ed52c04-442a-4a04-8a72-c40062a5e3ba\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd" Oct 30 09:19:21.591552 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:21.552827 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ed52c04-442a-4a04-8a72-c40062a5e3ba-workload-socket\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd\" (UID: \"9ed52c04-442a-4a04-8a72-c40062a5e3ba\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd" Oct 30 09:19:21.714714 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:21.564759 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ed52c04-442a-4a04-8a72-c40062a5e3ba-credential-socket\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd\" (UID: \"9ed52c04-442a-4a04-8a72-c40062a5e3ba\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd" Oct 30 09:19:21.714937 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:21.583777 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ed52c04-442a-4a04-8a72-c40062a5e3ba-istiod-ca-cert\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd\" (UID: \"9ed52c04-442a-4a04-8a72-c40062a5e3ba\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd" Oct 30 09:19:21.715046 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:21.645882 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ed52c04-442a-4a04-8a72-c40062a5e3ba-workload-certs\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd\" (UID: \"9ed52c04-442a-4a04-8a72-c40062a5e3ba\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd" Oct 30 09:19:21.715145 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:21.678059 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ed52c04-442a-4a04-8a72-c40062a5e3ba-istio-envoy\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd\" (UID: \"9ed52c04-442a-4a04-8a72-c40062a5e3ba\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd" Oct 30 09:19:21.715246 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:21.691002 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ed52c04-442a-4a04-8a72-c40062a5e3ba-istio-podinfo\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd\" (UID: \"9ed52c04-442a-4a04-8a72-c40062a5e3ba\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd" Oct 30 09:19:21.715339 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:21.696213 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gstff\" (UniqueName: \"kubernetes.io/projected/9ed52c04-442a-4a04-8a72-c40062a5e3ba-kube-api-access-gstff\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd\" (UID: \"9ed52c04-442a-4a04-8a72-c40062a5e3ba\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd" Oct 30 09:19:21.786822 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:21.775627 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ed52c04-442a-4a04-8a72-c40062a5e3ba-istio-token\") pod \"e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd\" (UID: \"9ed52c04-442a-4a04-8a72-c40062a5e3ba\") " pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd" Oct 30 09:19:22.037433 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:22.031311 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd" Oct 30 09:19:22.410872 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:22.410830 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55" podUID="88be34c3-a9e9-4489-a449-206a99ef1663" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:22.413025 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:22.413001 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55" Oct 30 09:19:22.794536 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:22.794491 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb" podUID="a540c86f-c856-42a8-9f73-b5b9b690f42b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Oct 30 09:19:22.862406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:22.795467 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/distinct-2-w1-c5447d-kserve-676749d548-xlmsf" podUID="c9ff897f-c520-4d0f-8820-e185b1d47dac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Oct 30 09:19:22.862406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:22.809316 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-176b6c/distinct-w3-176b6c-kserve-f57b7579c-k4pc5" podUID="28fd4072-399c-4be5-a0d8-1be6966da4af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Oct 30 09:19:23.417649 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:23.417594 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55" podUID="88be34c3-a9e9-4489-a449-206a99ef1663" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:23.879467 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:23.869661 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb" podUID="a540c86f-c856-42a8-9f73-b5b9b690f42b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Oct 30 09:19:23.879467 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:23.870610 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb" Oct 30 09:19:24.071407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:24.071196 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/distinct-2-w1-c5447d-kserve-676749d548-xlmsf" podUID="c9ff897f-c520-4d0f-8820-e185b1d47dac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:24.104931 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:24.073865 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-176b6c/distinct-w3-176b6c-kserve-f57b7579c-k4pc5" podUID="28fd4072-399c-4be5-a0d8-1be6966da4af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Oct 30 09:19:24.482407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:24.480511 4562 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-176b6c/distinct-w3-176b6c-kserve-f57b7579c-k4pc5" podUID="28fd4072-399c-4be5-a0d8-1be6966da4af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:24.528671 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:24.526808 4562 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55" podUID="88be34c3-a9e9-4489-a449-206a99ef1663" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:24.550464 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:24.550434 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c5447d/distinct-2-w1-c5447d-kserve-676749d548-xlmsf" Oct 30 09:19:24.550994 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:24.550973 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-176b6c/distinct-w3-176b6c-kserve-f57b7579c-k4pc5" Oct 30 09:19:24.551155 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:24.551142 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55" Oct 30 09:19:24.551280 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:24.551268 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c5447d/distinct-2-w1-c5447d-kserve-676749d548-xlmsf" Oct 30 09:19:24.553963 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:24.553946 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb" Oct 30 09:19:24.554089 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:24.554077 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-176b6c/distinct-w3-176b6c-kserve-f57b7579c-k4pc5" Oct 30 09:19:24.598272 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:24.577001 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd"] Oct 30 09:19:24.598272 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:19:24.584334 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ed52c04_442a_4a04_8a72_c40062a5e3ba.slice/crio-ec0b04ad109f07ebccb9a1553dbb326b91e1e45749abf658bb5bfa104b0584f5 WatchSource:0}: Error finding container ec0b04ad109f07ebccb9a1553dbb326b91e1e45749abf658bb5bfa104b0584f5: Status 404 returned error can't find the container with id ec0b04ad109f07ebccb9a1553dbb326b91e1e45749abf658bb5bfa104b0584f5 Oct 30 09:19:24.598272 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:24.589154 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:19:24.598272 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:24.589240 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:19:24.598272 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:24.589287 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:19:25.256166 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:25.256126 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-648892-f58f78dd7-xf8xh"] Oct 30 09:19:25.261971 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:25.261954 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-648892-f58f78dd7-xf8xh" Oct 30 09:19:25.276900 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:25.276874 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-648892-f58f78dd7-xf8xh"] Oct 30 09:19:25.277581 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:25.277557 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-648892-dockercfg-x6qpq\"" Oct 30 09:19:25.285406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:25.285362 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-648892\"" Oct 30 09:19:25.351614 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:25.351576 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ps9w\" (UniqueName: \"kubernetes.io/projected/174415ce-8818-44e6-b26e-f95c8c4b3521-kube-api-access-5ps9w\") pod \"payload-pre-processing-e2e-api-a-648892-f58f78dd7-xf8xh\" (UID: \"174415ce-8818-44e6-b26e-f95c8c4b3521\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-648892-f58f78dd7-xf8xh" Oct 30 09:19:25.351773 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:25.351648 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/174415ce-8818-44e6-b26e-f95c8c4b3521-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-648892-f58f78dd7-xf8xh\" (UID: \"174415ce-8818-44e6-b26e-f95c8c4b3521\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-648892-f58f78dd7-xf8xh" Oct 30 09:19:25.359855 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:25.359826 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-648892-578444d976-xxlck"] Oct 30 09:19:25.379405 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:25.366642 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-648892-578444d976-xxlck" Oct 30 09:19:25.422420 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:25.410925 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-648892-578444d976-xxlck"] Oct 30 09:19:25.458402 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:25.456160 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56d7083b-d7cf-46d7-9469-6bcf9ef1d02a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-648892-578444d976-xxlck\" (UID: \"56d7083b-d7cf-46d7-9469-6bcf9ef1d02a\") " pod="openshift-ingress/payload-processing-e2e-api-a-648892-578444d976-xxlck" Oct 30 09:19:25.458402 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:25.456351 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ps9w\" (UniqueName: \"kubernetes.io/projected/174415ce-8818-44e6-b26e-f95c8c4b3521-kube-api-access-5ps9w\") pod \"payload-pre-processing-e2e-api-a-648892-f58f78dd7-xf8xh\" (UID: \"174415ce-8818-44e6-b26e-f95c8c4b3521\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-648892-f58f78dd7-xf8xh" Oct 30 09:19:25.458402 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:25.456412 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g744h\" (UniqueName: \"kubernetes.io/projected/56d7083b-d7cf-46d7-9469-6bcf9ef1d02a-kube-api-access-g744h\") pod \"payload-processing-e2e-api-a-648892-578444d976-xxlck\" (UID: \"56d7083b-d7cf-46d7-9469-6bcf9ef1d02a\") " pod="openshift-ingress/payload-processing-e2e-api-a-648892-578444d976-xxlck" Oct 30 09:19:25.458402 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:25.456468 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/174415ce-8818-44e6-b26e-f95c8c4b3521-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-648892-f58f78dd7-xf8xh\" (UID: \"174415ce-8818-44e6-b26e-f95c8c4b3521\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-648892-f58f78dd7-xf8xh" Oct 30 09:19:25.458402 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:25.457529 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/174415ce-8818-44e6-b26e-f95c8c4b3521-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-648892-f58f78dd7-xf8xh\" (UID: \"174415ce-8818-44e6-b26e-f95c8c4b3521\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-648892-f58f78dd7-xf8xh" Oct 30 09:19:25.583855 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:25.578522 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g744h\" (UniqueName: \"kubernetes.io/projected/56d7083b-d7cf-46d7-9469-6bcf9ef1d02a-kube-api-access-g744h\") pod \"payload-processing-e2e-api-a-648892-578444d976-xxlck\" (UID: \"56d7083b-d7cf-46d7-9469-6bcf9ef1d02a\") " pod="openshift-ingress/payload-processing-e2e-api-a-648892-578444d976-xxlck" Oct 30 09:19:25.583855 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:25.578823 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56d7083b-d7cf-46d7-9469-6bcf9ef1d02a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-648892-578444d976-xxlck\" (UID: \"56d7083b-d7cf-46d7-9469-6bcf9ef1d02a\") " pod="openshift-ingress/payload-processing-e2e-api-a-648892-578444d976-xxlck" Oct 30 09:19:25.592423 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:25.591773 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56d7083b-d7cf-46d7-9469-6bcf9ef1d02a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-648892-578444d976-xxlck\" (UID: \"56d7083b-d7cf-46d7-9469-6bcf9ef1d02a\") " pod="openshift-ingress/payload-processing-e2e-api-a-648892-578444d976-xxlck" Oct 30 09:19:25.596882 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:25.596852 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ps9w\" (UniqueName: \"kubernetes.io/projected/174415ce-8818-44e6-b26e-f95c8c4b3521-kube-api-access-5ps9w\") pod \"payload-pre-processing-e2e-api-a-648892-f58f78dd7-xf8xh\" (UID: \"174415ce-8818-44e6-b26e-f95c8c4b3521\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-648892-f58f78dd7-xf8xh" Oct 30 09:19:25.700773 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:25.700730 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd" event={"ID":"9ed52c04-442a-4a04-8a72-c40062a5e3ba","Type":"ContainerStarted","Data":"6866fbc01b29285c1c8232c865f27766be9819dda948a40292c79c33fd74df4c"} Oct 30 09:19:25.726503 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:25.726460 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd" event={"ID":"9ed52c04-442a-4a04-8a72-c40062a5e3ba","Type":"ContainerStarted","Data":"ec0b04ad109f07ebccb9a1553dbb326b91e1e45749abf658bb5bfa104b0584f5"} Oct 30 09:19:25.772164 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:25.772094 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd" podStartSLOduration=5.772070418 podStartE2EDuration="5.772070418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:19:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:19:25.756803764 +0000 UTC m=+1526.018364756" watchObservedRunningTime="2026-10-30 09:19:25.772070418 +0000 UTC m=+1526.033631407" Oct 30 09:19:25.797459 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:25.787729 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g744h\" (UniqueName: \"kubernetes.io/projected/56d7083b-d7cf-46d7-9469-6bcf9ef1d02a-kube-api-access-g744h\") pod \"payload-processing-e2e-api-a-648892-578444d976-xxlck\" (UID: \"56d7083b-d7cf-46d7-9469-6bcf9ef1d02a\") " pod="openshift-ingress/payload-processing-e2e-api-a-648892-578444d976-xxlck" Oct 30 09:19:25.877134 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:25.877044 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-648892-f58f78dd7-xf8xh" Oct 30 09:19:26.033503 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:26.033466 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-648892-578444d976-xxlck" Oct 30 09:19:26.036633 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:26.036599 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd" Oct 30 09:19:26.038234 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:26.038204 4562 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 09:19:26.038340 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:26.038280 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd" podUID="9ed52c04-442a-4a04-8a72-c40062a5e3ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 09:19:26.184199 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:19:26.184169 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod174415ce_8818_44e6_b26e_f95c8c4b3521.slice/crio-da8778cfa990e3de51ae2990f744d8f88eb162cae626fd9794f21a102452ff07 WatchSource:0}: Error finding container da8778cfa990e3de51ae2990f744d8f88eb162cae626fd9794f21a102452ff07: Status 404 returned error can't find the container with id da8778cfa990e3de51ae2990f744d8f88eb162cae626fd9794f21a102452ff07 Oct 30 09:19:26.184350 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:26.184333 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-648892-f58f78dd7-xf8xh"] Oct 30 09:19:26.370585 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:26.370555 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-648892-578444d976-xxlck"] Oct 30 09:19:26.371755 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:19:26.371725 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56d7083b_d7cf_46d7_9469_6bcf9ef1d02a.slice/crio-fd3bd20f21f8242df2d023614deaf1bd769936938ae1634e61065c12848b2c22 WatchSource:0}: Error finding container fd3bd20f21f8242df2d023614deaf1bd769936938ae1634e61065c12848b2c22: Status 404 returned error can't find the container with id fd3bd20f21f8242df2d023614deaf1bd769936938ae1634e61065c12848b2c22 Oct 30 09:19:26.695247 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:26.695178 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-648892-578444d976-xxlck" event={"ID":"56d7083b-d7cf-46d7-9469-6bcf9ef1d02a","Type":"ContainerStarted","Data":"efbc0162f569d63bd4c10db2c9a92844ffceccb8e45d9ff8d23599fdfcd03107"} Oct 30 09:19:26.695247 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:26.695246 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-648892-578444d976-xxlck" event={"ID":"56d7083b-d7cf-46d7-9469-6bcf9ef1d02a","Type":"ContainerStarted","Data":"fd3bd20f21f8242df2d023614deaf1bd769936938ae1634e61065c12848b2c22"} Oct 30 09:19:26.695533 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:26.695278 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-648892-578444d976-xxlck" Oct 30 09:19:26.696990 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:26.696961 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-648892-f58f78dd7-xf8xh" event={"ID":"174415ce-8818-44e6-b26e-f95c8c4b3521","Type":"ContainerStarted","Data":"875fce28a8a21c2b81b0eb833d25db982c8f0b1764bffe31713682385f132060"} Oct 30 09:19:26.697121 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:26.697006 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-648892-f58f78dd7-xf8xh" event={"ID":"174415ce-8818-44e6-b26e-f95c8c4b3521","Type":"ContainerStarted","Data":"da8778cfa990e3de51ae2990f744d8f88eb162cae626fd9794f21a102452ff07"} Oct 30 09:19:26.697186 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:26.697161 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-648892-f58f78dd7-xf8xh" Oct 30 09:19:26.762405 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:26.761866 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-648892-578444d976-xxlck" podStartSLOduration=1.761845939 podStartE2EDuration="1.761845939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:19:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:19:26.75734567 +0000 UTC m=+1527.018906662" watchObservedRunningTime="2026-10-30 09:19:26.761845939 +0000 UTC m=+1527.023406944" Oct 30 09:19:26.803288 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:26.803239 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-648892-f58f78dd7-xf8xh" podStartSLOduration=1.8032262079999999 podStartE2EDuration="1.803226208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:19:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:19:26.801761215 +0000 UTC m=+1527.063322225" watchObservedRunningTime="2026-10-30 09:19:26.803226208 +0000 UTC m=+1527.064787195" Oct 30 09:19:27.097792 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:27.097704 4562 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 09:19:27.097792 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:27.097766 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd" podUID="9ed52c04-442a-4a04-8a72-c40062a5e3ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 09:19:28.037007 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:28.036971 4562 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 09:19:28.037548 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:28.037512 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd" podUID="9ed52c04-442a-4a04-8a72-c40062a5e3ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 09:19:28.595796 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:19:28.595758 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:19:29.037373 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:29.037339 4562 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 09:19:29.037787 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:29.037433 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd" podUID="9ed52c04-442a-4a04-8a72-c40062a5e3ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 09:19:30.037566 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:30.037529 4562 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 09:19:30.037958 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:30.037585 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd" podUID="9ed52c04-442a-4a04-8a72-c40062a5e3ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 09:19:30.410169 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:30.410089 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-8eada0cf-5884b8bb8-lp9q6" Oct 30 09:19:30.411025 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:30.411003 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-8eada0cf-7c5b9fd4ff-9jzlk" Oct 30 09:19:31.037072 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:31.037038 4562 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 09:19:31.037249 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:31.037131 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd" podUID="9ed52c04-442a-4a04-8a72-c40062a5e3ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 09:19:32.036972 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:32.036929 4562 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 09:19:32.037435 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:32.036982 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd" podUID="9ed52c04-442a-4a04-8a72-c40062a5e3ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 09:19:33.042739 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:33.042708 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd" Oct 30 09:19:33.043159 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:33.043093 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd" Oct 30 09:19:33.043909 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:33.043890 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd" Oct 30 09:19:33.049287 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:33.049250 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ca0df2-57b8bf5d59-tbxzz"] Oct 30 09:19:33.053589 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:33.053573 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca0df2-57b8bf5d59-tbxzz" Oct 30 09:19:33.071913 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:33.071869 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-ca0df2-dockercfg-pv56z\"" Oct 30 09:19:33.076259 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:33.076228 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ca0df2-57b8bf5d59-tbxzz"] Oct 30 09:19:33.126797 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:33.126762 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-ca0df2\"" Oct 30 09:19:33.309054 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:33.308966 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ca0df2-7d67bcc9bb-b2brq"] Oct 30 09:19:33.334802 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:33.330366 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/349c4a5f-88cf-473a-9463-283db5e5f586-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ca0df2-57b8bf5d59-tbxzz\" (UID: \"349c4a5f-88cf-473a-9463-283db5e5f586\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca0df2-57b8bf5d59-tbxzz" Oct 30 09:19:33.334802 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:33.330461 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27wcx\" (UniqueName: \"kubernetes.io/projected/349c4a5f-88cf-473a-9463-283db5e5f586-kube-api-access-27wcx\") pod \"payload-pre-processing-e2e-api-a-ca0df2-57b8bf5d59-tbxzz\" (UID: \"349c4a5f-88cf-473a-9463-283db5e5f586\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca0df2-57b8bf5d59-tbxzz" Oct 30 09:19:33.341841 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:33.341811 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ca0df2-7d67bcc9bb-b2brq" Oct 30 09:19:33.352468 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:33.352441 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ca0df2-7d67bcc9bb-b2brq"] Oct 30 09:19:33.447577 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:33.443969 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/349c4a5f-88cf-473a-9463-283db5e5f586-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ca0df2-57b8bf5d59-tbxzz\" (UID: \"349c4a5f-88cf-473a-9463-283db5e5f586\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca0df2-57b8bf5d59-tbxzz" Oct 30 09:19:33.447577 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:33.444063 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27wcx\" (UniqueName: \"kubernetes.io/projected/349c4a5f-88cf-473a-9463-283db5e5f586-kube-api-access-27wcx\") pod \"payload-pre-processing-e2e-api-a-ca0df2-57b8bf5d59-tbxzz\" (UID: \"349c4a5f-88cf-473a-9463-283db5e5f586\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca0df2-57b8bf5d59-tbxzz" Oct 30 09:19:33.447577 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:33.447396 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/349c4a5f-88cf-473a-9463-283db5e5f586-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ca0df2-57b8bf5d59-tbxzz\" (UID: \"349c4a5f-88cf-473a-9463-283db5e5f586\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca0df2-57b8bf5d59-tbxzz" Oct 30 09:19:33.469939 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:33.469908 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27wcx\" (UniqueName: \"kubernetes.io/projected/349c4a5f-88cf-473a-9463-283db5e5f586-kube-api-access-27wcx\") pod \"payload-pre-processing-e2e-api-a-ca0df2-57b8bf5d59-tbxzz\" (UID: \"349c4a5f-88cf-473a-9463-283db5e5f586\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca0df2-57b8bf5d59-tbxzz" Oct 30 09:19:33.545184 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:33.545152 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d38fd70a-7b00-4c69-86d7-a6a58900a3bf-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ca0df2-7d67bcc9bb-b2brq\" (UID: \"d38fd70a-7b00-4c69-86d7-a6a58900a3bf\") " pod="openshift-ingress/payload-processing-e2e-api-a-ca0df2-7d67bcc9bb-b2brq" Oct 30 09:19:33.545348 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:33.545289 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6w4n\" (UniqueName: \"kubernetes.io/projected/d38fd70a-7b00-4c69-86d7-a6a58900a3bf-kube-api-access-k6w4n\") pod \"payload-processing-e2e-api-a-ca0df2-7d67bcc9bb-b2brq\" (UID: \"d38fd70a-7b00-4c69-86d7-a6a58900a3bf\") " pod="openshift-ingress/payload-processing-e2e-api-a-ca0df2-7d67bcc9bb-b2brq" Oct 30 09:19:33.646191 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:33.646115 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6w4n\" (UniqueName: \"kubernetes.io/projected/d38fd70a-7b00-4c69-86d7-a6a58900a3bf-kube-api-access-k6w4n\") pod \"payload-processing-e2e-api-a-ca0df2-7d67bcc9bb-b2brq\" (UID: \"d38fd70a-7b00-4c69-86d7-a6a58900a3bf\") " pod="openshift-ingress/payload-processing-e2e-api-a-ca0df2-7d67bcc9bb-b2brq" Oct 30 09:19:33.646324 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:33.646258 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d38fd70a-7b00-4c69-86d7-a6a58900a3bf-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ca0df2-7d67bcc9bb-b2brq\" (UID: \"d38fd70a-7b00-4c69-86d7-a6a58900a3bf\") " pod="openshift-ingress/payload-processing-e2e-api-a-ca0df2-7d67bcc9bb-b2brq" Oct 30 09:19:33.646933 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:33.646913 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d38fd70a-7b00-4c69-86d7-a6a58900a3bf-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ca0df2-7d67bcc9bb-b2brq\" (UID: \"d38fd70a-7b00-4c69-86d7-a6a58900a3bf\") " pod="openshift-ingress/payload-processing-e2e-api-a-ca0df2-7d67bcc9bb-b2brq" Oct 30 09:19:33.668915 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:33.668883 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6w4n\" (UniqueName: \"kubernetes.io/projected/d38fd70a-7b00-4c69-86d7-a6a58900a3bf-kube-api-access-k6w4n\") pod \"payload-processing-e2e-api-a-ca0df2-7d67bcc9bb-b2brq\" (UID: \"d38fd70a-7b00-4c69-86d7-a6a58900a3bf\") " pod="openshift-ingress/payload-processing-e2e-api-a-ca0df2-7d67bcc9bb-b2brq" Oct 30 09:19:33.669093 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:33.669056 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ca0df2-7d67bcc9bb-b2brq" Oct 30 09:19:33.676426 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:33.676372 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca0df2-57b8bf5d59-tbxzz" Oct 30 09:19:33.973343 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:19:33.973318 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd38fd70a_7b00_4c69_86d7_a6a58900a3bf.slice/crio-0edfa9b0032b4950ba7f093aa31f7275a744c78a485d57ef55eb19bf97504256 WatchSource:0}: Error finding container 0edfa9b0032b4950ba7f093aa31f7275a744c78a485d57ef55eb19bf97504256: Status 404 returned error can't find the container with id 0edfa9b0032b4950ba7f093aa31f7275a744c78a485d57ef55eb19bf97504256 Oct 30 09:19:33.974425 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:33.974403 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ca0df2-7d67bcc9bb-b2brq"] Oct 30 09:19:33.994765 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:19:33.994742 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod349c4a5f_88cf_473a_9463_283db5e5f586.slice/crio-59fa09d89132386c799849ba6fd04407ce5da4569cdaff87322bee91d446ef13 WatchSource:0}: Error finding container 59fa09d89132386c799849ba6fd04407ce5da4569cdaff87322bee91d446ef13: Status 404 returned error can't find the container with id 59fa09d89132386c799849ba6fd04407ce5da4569cdaff87322bee91d446ef13 Oct 30 09:19:33.995906 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:33.995879 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ca0df2-57b8bf5d59-tbxzz"] Oct 30 09:19:34.393054 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:34.392978 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh"] Oct 30 09:19:34.397048 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:34.397029 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh" Oct 30 09:19:34.466818 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:34.466787 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99f76ea3-743e-4536-9443-2454af9db849-workload-certs\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh\" (UID: \"99f76ea3-743e-4536-9443-2454af9db849\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh" Oct 30 09:19:34.466959 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:34.466854 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gjjn\" (UniqueName: \"kubernetes.io/projected/99f76ea3-743e-4536-9443-2454af9db849-kube-api-access-5gjjn\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh\" (UID: \"99f76ea3-743e-4536-9443-2454af9db849\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh" Oct 30 09:19:34.467008 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:34.466970 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99f76ea3-743e-4536-9443-2454af9db849-istio-podinfo\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh\" (UID: \"99f76ea3-743e-4536-9443-2454af9db849\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh" Oct 30 09:19:34.467085 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:34.467072 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99f76ea3-743e-4536-9443-2454af9db849-istio-envoy\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh\" (UID: \"99f76ea3-743e-4536-9443-2454af9db849\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh" Oct 30 09:19:34.467126 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:34.467105 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99f76ea3-743e-4536-9443-2454af9db849-istiod-ca-cert\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh\" (UID: \"99f76ea3-743e-4536-9443-2454af9db849\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh" Oct 30 09:19:34.467195 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:34.467183 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99f76ea3-743e-4536-9443-2454af9db849-credential-socket\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh\" (UID: \"99f76ea3-743e-4536-9443-2454af9db849\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh" Oct 30 09:19:34.467230 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:34.467217 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99f76ea3-743e-4536-9443-2454af9db849-istio-data\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh\" (UID: \"99f76ea3-743e-4536-9443-2454af9db849\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh" Oct 30 09:19:34.467325 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:34.467306 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99f76ea3-743e-4536-9443-2454af9db849-workload-socket\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh\" (UID: \"99f76ea3-743e-4536-9443-2454af9db849\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh" Oct 30 09:19:34.467397 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:34.467367 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99f76ea3-743e-4536-9443-2454af9db849-istio-token\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh\" (UID: \"99f76ea3-743e-4536-9443-2454af9db849\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh" Oct 30 09:19:34.493474 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:34.493447 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh"] Oct 30 09:19:34.568737 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:34.568708 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99f76ea3-743e-4536-9443-2454af9db849-istio-data\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh\" (UID: \"99f76ea3-743e-4536-9443-2454af9db849\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh" Oct 30 09:19:34.568897 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:34.568782 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99f76ea3-743e-4536-9443-2454af9db849-workload-socket\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh\" (UID: \"99f76ea3-743e-4536-9443-2454af9db849\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh" Oct 30 09:19:34.568897 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:34.568805 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99f76ea3-743e-4536-9443-2454af9db849-istio-token\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh\" (UID: \"99f76ea3-743e-4536-9443-2454af9db849\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh" Oct 30 09:19:34.568897 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:34.568831 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99f76ea3-743e-4536-9443-2454af9db849-workload-certs\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh\" (UID: \"99f76ea3-743e-4536-9443-2454af9db849\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh" Oct 30 09:19:34.568897 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:34.568885 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gjjn\" (UniqueName: \"kubernetes.io/projected/99f76ea3-743e-4536-9443-2454af9db849-kube-api-access-5gjjn\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh\" (UID: \"99f76ea3-743e-4536-9443-2454af9db849\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh" Oct 30 09:19:34.569158 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:34.568930 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99f76ea3-743e-4536-9443-2454af9db849-istio-podinfo\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh\" (UID: \"99f76ea3-743e-4536-9443-2454af9db849\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh" Oct 30 09:19:34.569158 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:34.569025 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99f76ea3-743e-4536-9443-2454af9db849-istio-envoy\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh\" (UID: \"99f76ea3-743e-4536-9443-2454af9db849\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh" Oct 30 09:19:34.569158 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:34.569073 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99f76ea3-743e-4536-9443-2454af9db849-istiod-ca-cert\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh\" (UID: \"99f76ea3-743e-4536-9443-2454af9db849\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh" Oct 30 09:19:34.569158 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:34.569137 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99f76ea3-743e-4536-9443-2454af9db849-credential-socket\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh\" (UID: \"99f76ea3-743e-4536-9443-2454af9db849\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh" Oct 30 09:19:34.569358 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:34.569166 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99f76ea3-743e-4536-9443-2454af9db849-istio-data\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh\" (UID: \"99f76ea3-743e-4536-9443-2454af9db849\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh" Oct 30 09:19:34.569603 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:34.569578 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99f76ea3-743e-4536-9443-2454af9db849-workload-certs\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh\" (UID: \"99f76ea3-743e-4536-9443-2454af9db849\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh" Oct 30 09:19:34.569735 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:34.569628 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99f76ea3-743e-4536-9443-2454af9db849-workload-socket\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh\" (UID: \"99f76ea3-743e-4536-9443-2454af9db849\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh" Oct 30 09:19:34.569815 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:34.569750 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99f76ea3-743e-4536-9443-2454af9db849-credential-socket\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh\" (UID: \"99f76ea3-743e-4536-9443-2454af9db849\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh" Oct 30 09:19:34.570043 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:34.570021 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99f76ea3-743e-4536-9443-2454af9db849-istiod-ca-cert\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh\" (UID: \"99f76ea3-743e-4536-9443-2454af9db849\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh" Oct 30 09:19:34.571409 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:34.571370 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99f76ea3-743e-4536-9443-2454af9db849-istio-envoy\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh\" (UID: \"99f76ea3-743e-4536-9443-2454af9db849\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh" Oct 30 09:19:34.571512 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:34.571494 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99f76ea3-743e-4536-9443-2454af9db849-istio-podinfo\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh\" (UID: \"99f76ea3-743e-4536-9443-2454af9db849\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh" Oct 30 09:19:34.589024 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:34.589002 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gjjn\" (UniqueName: \"kubernetes.io/projected/99f76ea3-743e-4536-9443-2454af9db849-kube-api-access-5gjjn\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh\" (UID: \"99f76ea3-743e-4536-9443-2454af9db849\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh" Oct 30 09:19:34.589280 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:34.589258 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99f76ea3-743e-4536-9443-2454af9db849-istio-token\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh\" (UID: \"99f76ea3-743e-4536-9443-2454af9db849\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh" Oct 30 09:19:34.710519 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:34.710490 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh" Oct 30 09:19:34.751416 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:34.751361 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca0df2-57b8bf5d59-tbxzz" event={"ID":"349c4a5f-88cf-473a-9463-283db5e5f586","Type":"ContainerStarted","Data":"a2b12d9ab1d9faa5c67642e205f2e220db100410b45311806a1fcbeff1363945"} Oct 30 09:19:34.751550 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:34.751444 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca0df2-57b8bf5d59-tbxzz" event={"ID":"349c4a5f-88cf-473a-9463-283db5e5f586","Type":"ContainerStarted","Data":"59fa09d89132386c799849ba6fd04407ce5da4569cdaff87322bee91d446ef13"} Oct 30 09:19:34.751612 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:34.751591 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca0df2-57b8bf5d59-tbxzz" Oct 30 09:19:34.753140 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:34.753117 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ca0df2-7d67bcc9bb-b2brq" event={"ID":"d38fd70a-7b00-4c69-86d7-a6a58900a3bf","Type":"ContainerStarted","Data":"ec15dc53d6caefe3d569d9664363e8f7d907a6a70419711b0eda8b8700a946aa"} Oct 30 09:19:34.753252 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:34.753153 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ca0df2-7d67bcc9bb-b2brq" event={"ID":"d38fd70a-7b00-4c69-86d7-a6a58900a3bf","Type":"ContainerStarted","Data":"0edfa9b0032b4950ba7f093aa31f7275a744c78a485d57ef55eb19bf97504256"} Oct 30 09:19:34.753252 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:34.753201 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-ca0df2-7d67bcc9bb-b2brq" Oct 30 09:19:34.799260 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:34.799186 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca0df2-57b8bf5d59-tbxzz" podStartSLOduration=1.79916499 podStartE2EDuration="1.79916499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:19:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:19:34.793865413 +0000 UTC m=+1535.055426409" watchObservedRunningTime="2026-10-30 09:19:34.79916499 +0000 UTC m=+1535.060725980" Oct 30 09:19:34.835958 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:34.835896 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-ca0df2-7d67bcc9bb-b2brq" podStartSLOduration=1.835876982 podStartE2EDuration="1.835876982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:19:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:19:34.834958825 +0000 UTC m=+1535.096519810" watchObservedRunningTime="2026-10-30 09:19:34.835876982 +0000 UTC m=+1535.097437972" Oct 30 09:19:35.052351 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:35.052318 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh"] Oct 30 09:19:35.056497 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:19:35.056462 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99f76ea3_743e_4536_9443_2454af9db849.slice/crio-b9460fe4ebcd89c4dc8259a803c452dc52d0ae8760d81d20639a2bad222e3a45 WatchSource:0}: Error finding container b9460fe4ebcd89c4dc8259a803c452dc52d0ae8760d81d20639a2bad222e3a45: Status 404 returned error can't find the container with id b9460fe4ebcd89c4dc8259a803c452dc52d0ae8760d81d20639a2bad222e3a45 Oct 30 09:19:35.058787 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:35.058746 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:19:35.058897 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:35.058835 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:19:35.058897 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:35.058886 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:19:35.772581 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:35.772546 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh" event={"ID":"99f76ea3-743e-4536-9443-2454af9db849","Type":"ContainerStarted","Data":"95bd10bbf27ff2e9d65865d0fef7b08e5d732c5544ed5b48fc36a8f5743a50bd"} Oct 30 09:19:35.772960 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:35.772626 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh" event={"ID":"99f76ea3-743e-4536-9443-2454af9db849","Type":"ContainerStarted","Data":"b9460fe4ebcd89c4dc8259a803c452dc52d0ae8760d81d20639a2bad222e3a45"} Oct 30 09:19:35.856406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:35.849143 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh" podStartSLOduration=1.849122324 podStartE2EDuration="1.849122324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:19:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:19:35.846866474 +0000 UTC m=+1536.108427463" watchObservedRunningTime="2026-10-30 09:19:35.849122324 +0000 UTC m=+1536.110683313" Oct 30 09:19:36.135119 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:36.135029 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vbklx"] Oct 30 09:19:36.135296 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:36.135265 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vbklx" podUID="051c87da-acbb-4941-8fea-57107abb4720" containerName="istio-proxy" containerID="cri-o://310c6d2b893a6aad248318e2cc5beac3acff5e06d7cb6df6b723a2acea71d756" gracePeriod=30 Oct 30 09:19:36.711340 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:36.711303 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh" Oct 30 09:19:36.712692 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:36.712666 4562 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 09:19:36.712817 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:36.712719 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh" podUID="99f76ea3-743e-4536-9443-2454af9db849" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 09:19:36.896137 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:36.896104 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-648892-69475bb5d5-sthgm"] Oct 30 09:19:36.900359 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:36.900337 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-648892-69475bb5d5-sthgm" Oct 30 09:19:36.918810 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:36.918784 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-648892-69475bb5d5-sthgm"] Oct 30 09:19:36.926590 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:36.926566 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-648892-serving-cert\"" Oct 30 09:19:37.006369 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:37.006288 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03e6f8da-1927-44fa-bab1-472ea5f845e0-openshift-service-ca\") pod \"maas-api-e2e-api-a-648892-69475bb5d5-sthgm\" (UID: \"03e6f8da-1927-44fa-bab1-472ea5f845e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-648892-69475bb5d5-sthgm" Oct 30 09:19:37.006369 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:37.006332 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03e6f8da-1927-44fa-bab1-472ea5f845e0-metrics-certs\") pod \"maas-api-e2e-api-a-648892-69475bb5d5-sthgm\" (UID: \"03e6f8da-1927-44fa-bab1-472ea5f845e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-648892-69475bb5d5-sthgm" Oct 30 09:19:37.006570 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:37.006479 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03e6f8da-1927-44fa-bab1-472ea5f845e0-maas-api-tls\") pod \"maas-api-e2e-api-a-648892-69475bb5d5-sthgm\" (UID: \"03e6f8da-1927-44fa-bab1-472ea5f845e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-648892-69475bb5d5-sthgm" Oct 30 09:19:37.006570 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:37.006521 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkx68\" (UniqueName: \"kubernetes.io/projected/03e6f8da-1927-44fa-bab1-472ea5f845e0-kube-api-access-hkx68\") pod \"maas-api-e2e-api-a-648892-69475bb5d5-sthgm\" (UID: \"03e6f8da-1927-44fa-bab1-472ea5f845e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-648892-69475bb5d5-sthgm" Oct 30 09:19:37.108247 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:37.108212 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03e6f8da-1927-44fa-bab1-472ea5f845e0-openshift-service-ca\") pod \"maas-api-e2e-api-a-648892-69475bb5d5-sthgm\" (UID: \"03e6f8da-1927-44fa-bab1-472ea5f845e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-648892-69475bb5d5-sthgm" Oct 30 09:19:37.108476 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:37.108255 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03e6f8da-1927-44fa-bab1-472ea5f845e0-metrics-certs\") pod \"maas-api-e2e-api-a-648892-69475bb5d5-sthgm\" (UID: \"03e6f8da-1927-44fa-bab1-472ea5f845e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-648892-69475bb5d5-sthgm" Oct 30 09:19:37.108476 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:37.108304 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03e6f8da-1927-44fa-bab1-472ea5f845e0-maas-api-tls\") pod \"maas-api-e2e-api-a-648892-69475bb5d5-sthgm\" (UID: \"03e6f8da-1927-44fa-bab1-472ea5f845e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-648892-69475bb5d5-sthgm" Oct 30 09:19:37.108476 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:37.108330 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkx68\" (UniqueName: \"kubernetes.io/projected/03e6f8da-1927-44fa-bab1-472ea5f845e0-kube-api-access-hkx68\") pod \"maas-api-e2e-api-a-648892-69475bb5d5-sthgm\" (UID: \"03e6f8da-1927-44fa-bab1-472ea5f845e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-648892-69475bb5d5-sthgm" Oct 30 09:19:37.109117 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:37.109093 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03e6f8da-1927-44fa-bab1-472ea5f845e0-openshift-service-ca\") pod \"maas-api-e2e-api-a-648892-69475bb5d5-sthgm\" (UID: \"03e6f8da-1927-44fa-bab1-472ea5f845e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-648892-69475bb5d5-sthgm" Oct 30 09:19:37.110924 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:37.110895 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03e6f8da-1927-44fa-bab1-472ea5f845e0-metrics-certs\") pod \"maas-api-e2e-api-a-648892-69475bb5d5-sthgm\" (UID: \"03e6f8da-1927-44fa-bab1-472ea5f845e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-648892-69475bb5d5-sthgm" Oct 30 09:19:37.111016 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:37.110922 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03e6f8da-1927-44fa-bab1-472ea5f845e0-maas-api-tls\") pod \"maas-api-e2e-api-a-648892-69475bb5d5-sthgm\" (UID: \"03e6f8da-1927-44fa-bab1-472ea5f845e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-648892-69475bb5d5-sthgm" Oct 30 09:19:37.129117 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:37.129092 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkx68\" (UniqueName: \"kubernetes.io/projected/03e6f8da-1927-44fa-bab1-472ea5f845e0-kube-api-access-hkx68\") pod \"maas-api-e2e-api-a-648892-69475bb5d5-sthgm\" (UID: \"03e6f8da-1927-44fa-bab1-472ea5f845e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-648892-69475bb5d5-sthgm" Oct 30 09:19:37.217406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:37.215680 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-648892-69475bb5d5-sthgm" Oct 30 09:19:37.635641 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:19:37.635602 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03e6f8da_1927_44fa_bab1_472ea5f845e0.slice/crio-93cc2440b7f752c278175c19845ab54e4001085629f7a286f323085f977e70dc WatchSource:0}: Error finding container 93cc2440b7f752c278175c19845ab54e4001085629f7a286f323085f977e70dc: Status 404 returned error can't find the container with id 93cc2440b7f752c278175c19845ab54e4001085629f7a286f323085f977e70dc Oct 30 09:19:37.636842 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:37.636815 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-648892-69475bb5d5-sthgm"] Oct 30 09:19:37.637562 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:37.637542 4562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 09:19:37.706469 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:37.706440 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-648892-f58f78dd7-xf8xh" Oct 30 09:19:37.706584 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:37.706515 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-648892-578444d976-xxlck" Oct 30 09:19:37.711316 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:37.711295 4562 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 09:19:37.711401 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:37.711342 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh" podUID="99f76ea3-743e-4536-9443-2454af9db849" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 09:19:37.791092 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:37.791058 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-648892-69475bb5d5-sthgm" event={"ID":"03e6f8da-1927-44fa-bab1-472ea5f845e0","Type":"ContainerStarted","Data":"93cc2440b7f752c278175c19845ab54e4001085629f7a286f323085f977e70dc"} Oct 30 09:19:38.711580 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:38.711548 4562 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 09:19:38.711969 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:38.711604 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh" podUID="99f76ea3-743e-4536-9443-2454af9db849" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 09:19:38.798854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:38.798812 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-648892-69475bb5d5-sthgm" event={"ID":"03e6f8da-1927-44fa-bab1-472ea5f845e0","Type":"ContainerStarted","Data":"8c2f0f1abaf66e4bcff8fb3b5b7ec6822479c75fcefebe12ea01fd6a90357fa0"} Oct 30 09:19:38.799038 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:38.798916 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-648892-69475bb5d5-sthgm" Oct 30 09:19:39.711015 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:39.710971 4562 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 09:19:39.711211 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:39.711029 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh" podUID="99f76ea3-743e-4536-9443-2454af9db849" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 09:19:40.237510 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:40.237456 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-648892-69475bb5d5-sthgm" podStartSLOduration=3.7833046489999997 podStartE2EDuration="4.237442027s" totalImagesPullingTime="454.137378ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:19:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:19:38.860650029 +0000 UTC m=+1539.122211045" watchObservedRunningTime="2026-10-30 09:19:40.237442027 +0000 UTC m=+1540.499003014" Oct 30 09:19:40.264486 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:40.264265 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g"] Oct 30 09:19:40.269821 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:40.269790 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g" Oct 30 09:19:40.278610 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:40.272275 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g"] Oct 30 09:19:40.367504 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:40.367467 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d2f5f60-9f83-49d0-aac6-94d56afc2d56-istio-envoy\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g\" (UID: \"0d2f5f60-9f83-49d0-aac6-94d56afc2d56\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g" Oct 30 09:19:40.367735 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:40.367710 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d2f5f60-9f83-49d0-aac6-94d56afc2d56-istio-token\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g\" (UID: \"0d2f5f60-9f83-49d0-aac6-94d56afc2d56\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g" Oct 30 09:19:40.367874 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:40.367857 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d2f5f60-9f83-49d0-aac6-94d56afc2d56-istiod-ca-cert\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g\" (UID: \"0d2f5f60-9f83-49d0-aac6-94d56afc2d56\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g" Oct 30 09:19:40.368104 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:40.368085 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d2f5f60-9f83-49d0-aac6-94d56afc2d56-istio-podinfo\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g\" (UID: \"0d2f5f60-9f83-49d0-aac6-94d56afc2d56\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g" Oct 30 09:19:40.369616 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:40.368987 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d2f5f60-9f83-49d0-aac6-94d56afc2d56-workload-socket\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g\" (UID: \"0d2f5f60-9f83-49d0-aac6-94d56afc2d56\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g" Oct 30 09:19:40.369616 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:40.369110 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d2f5f60-9f83-49d0-aac6-94d56afc2d56-workload-certs\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g\" (UID: \"0d2f5f60-9f83-49d0-aac6-94d56afc2d56\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g" Oct 30 09:19:40.369616 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:40.369173 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d2f5f60-9f83-49d0-aac6-94d56afc2d56-credential-socket\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g\" (UID: \"0d2f5f60-9f83-49d0-aac6-94d56afc2d56\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g" Oct 30 09:19:40.369616 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:40.369237 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d2f5f60-9f83-49d0-aac6-94d56afc2d56-istio-data\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g\" (UID: \"0d2f5f60-9f83-49d0-aac6-94d56afc2d56\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g" Oct 30 09:19:40.369616 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:40.369327 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjcr6\" (UniqueName: \"kubernetes.io/projected/0d2f5f60-9f83-49d0-aac6-94d56afc2d56-kube-api-access-pjcr6\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g\" (UID: \"0d2f5f60-9f83-49d0-aac6-94d56afc2d56\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g" Oct 30 09:19:40.470209 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:40.470177 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d2f5f60-9f83-49d0-aac6-94d56afc2d56-workload-certs\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g\" (UID: \"0d2f5f60-9f83-49d0-aac6-94d56afc2d56\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g" Oct 30 09:19:40.470509 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:40.470485 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d2f5f60-9f83-49d0-aac6-94d56afc2d56-credential-socket\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g\" (UID: \"0d2f5f60-9f83-49d0-aac6-94d56afc2d56\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g" Oct 30 09:19:40.470606 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:40.470547 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d2f5f60-9f83-49d0-aac6-94d56afc2d56-istio-data\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g\" (UID: \"0d2f5f60-9f83-49d0-aac6-94d56afc2d56\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g" Oct 30 09:19:40.470674 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:40.470626 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjcr6\" (UniqueName: \"kubernetes.io/projected/0d2f5f60-9f83-49d0-aac6-94d56afc2d56-kube-api-access-pjcr6\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g\" (UID: \"0d2f5f60-9f83-49d0-aac6-94d56afc2d56\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g" Oct 30 09:19:40.470674 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:40.470643 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d2f5f60-9f83-49d0-aac6-94d56afc2d56-workload-certs\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g\" (UID: \"0d2f5f60-9f83-49d0-aac6-94d56afc2d56\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g" Oct 30 09:19:40.470809 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:40.470671 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d2f5f60-9f83-49d0-aac6-94d56afc2d56-istio-envoy\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g\" (UID: \"0d2f5f60-9f83-49d0-aac6-94d56afc2d56\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g" Oct 30 09:19:40.470809 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:40.470720 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d2f5f60-9f83-49d0-aac6-94d56afc2d56-istio-token\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g\" (UID: \"0d2f5f60-9f83-49d0-aac6-94d56afc2d56\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g" Oct 30 09:19:40.470809 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:40.470761 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d2f5f60-9f83-49d0-aac6-94d56afc2d56-istiod-ca-cert\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g\" (UID: \"0d2f5f60-9f83-49d0-aac6-94d56afc2d56\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g" Oct 30 09:19:40.470809 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:40.470801 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d2f5f60-9f83-49d0-aac6-94d56afc2d56-credential-socket\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g\" (UID: \"0d2f5f60-9f83-49d0-aac6-94d56afc2d56\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g" Oct 30 09:19:40.471010 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:40.470928 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d2f5f60-9f83-49d0-aac6-94d56afc2d56-istio-data\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g\" (UID: \"0d2f5f60-9f83-49d0-aac6-94d56afc2d56\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g" Oct 30 09:19:40.471010 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:40.470946 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d2f5f60-9f83-49d0-aac6-94d56afc2d56-istio-podinfo\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g\" (UID: \"0d2f5f60-9f83-49d0-aac6-94d56afc2d56\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g" Oct 30 09:19:40.471010 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:40.470999 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d2f5f60-9f83-49d0-aac6-94d56afc2d56-workload-socket\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g\" (UID: \"0d2f5f60-9f83-49d0-aac6-94d56afc2d56\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g" Oct 30 09:19:40.471348 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:40.471323 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d2f5f60-9f83-49d0-aac6-94d56afc2d56-workload-socket\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g\" (UID: \"0d2f5f60-9f83-49d0-aac6-94d56afc2d56\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g" Oct 30 09:19:40.471485 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:40.471463 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d2f5f60-9f83-49d0-aac6-94d56afc2d56-istiod-ca-cert\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g\" (UID: \"0d2f5f60-9f83-49d0-aac6-94d56afc2d56\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g" Oct 30 09:19:40.473341 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:40.473318 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d2f5f60-9f83-49d0-aac6-94d56afc2d56-istio-podinfo\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g\" (UID: \"0d2f5f60-9f83-49d0-aac6-94d56afc2d56\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g" Oct 30 09:19:40.475681 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:40.475648 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d2f5f60-9f83-49d0-aac6-94d56afc2d56-istio-envoy\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g\" (UID: \"0d2f5f60-9f83-49d0-aac6-94d56afc2d56\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g" Oct 30 09:19:40.491136 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:40.491074 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d2f5f60-9f83-49d0-aac6-94d56afc2d56-istio-token\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g\" (UID: \"0d2f5f60-9f83-49d0-aac6-94d56afc2d56\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g" Oct 30 09:19:40.491831 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:40.491813 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjcr6\" (UniqueName: \"kubernetes.io/projected/0d2f5f60-9f83-49d0-aac6-94d56afc2d56-kube-api-access-pjcr6\") pod \"e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g\" (UID: \"0d2f5f60-9f83-49d0-aac6-94d56afc2d56\") " pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g" Oct 30 09:19:40.597438 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:40.597404 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g" Oct 30 09:19:40.711364 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:40.711329 4562 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 09:19:40.711512 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:40.711421 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh" podUID="99f76ea3-743e-4536-9443-2454af9db849" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 09:19:41.015680 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:41.015643 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g"] Oct 30 09:19:41.047574 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:41.045157 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:19:41.047574 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:41.045250 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:19:41.047574 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:41.045302 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:19:41.318795 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:41.318764 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vbklx" Oct 30 09:19:41.494327 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:41.494291 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/051c87da-acbb-4941-8fea-57107abb4720-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/051c87da-acbb-4941-8fea-57107abb4720-istio-envoy\") pod \"051c87da-acbb-4941-8fea-57107abb4720\" (UID: \"051c87da-acbb-4941-8fea-57107abb4720\") " Oct 30 09:19:41.494575 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:41.494546 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/051c87da-acbb-4941-8fea-57107abb4720-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/051c87da-acbb-4941-8fea-57107abb4720-istio-podinfo\") pod \"051c87da-acbb-4941-8fea-57107abb4720\" (UID: \"051c87da-acbb-4941-8fea-57107abb4720\") " Oct 30 09:19:41.494675 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:41.494621 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/051c87da-acbb-4941-8fea-57107abb4720-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/051c87da-acbb-4941-8fea-57107abb4720-workload-socket\") pod \"051c87da-acbb-4941-8fea-57107abb4720\" (UID: \"051c87da-acbb-4941-8fea-57107abb4720\") " Oct 30 09:19:41.494958 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:41.494929 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/051c87da-acbb-4941-8fea-57107abb4720-workload-socket" pod "051c87da-acbb-4941-8fea-57107abb4720" (UID: "051c87da-acbb-4941-8fea-57107abb4720"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:19:41.495068 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:41.495021 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/051c87da-acbb-4941-8fea-57107abb4720-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/051c87da-acbb-4941-8fea-57107abb4720-istiod-ca-cert\") pod \"051c87da-acbb-4941-8fea-57107abb4720\" (UID: \"051c87da-acbb-4941-8fea-57107abb4720\") " Oct 30 09:19:41.495141 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:41.495120 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/051c87da-acbb-4941-8fea-57107abb4720-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/051c87da-acbb-4941-8fea-57107abb4720-credential-socket\") pod \"051c87da-acbb-4941-8fea-57107abb4720\" (UID: \"051c87da-acbb-4941-8fea-57107abb4720\") " Oct 30 09:19:41.495260 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:41.495241 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/051c87da-acbb-4941-8fea-57107abb4720-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/051c87da-acbb-4941-8fea-57107abb4720-istio-data\") pod \"051c87da-acbb-4941-8fea-57107abb4720\" (UID: \"051c87da-acbb-4941-8fea-57107abb4720\") " Oct 30 09:19:41.495389 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:41.495357 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/051c87da-acbb-4941-8fea-57107abb4720-istio-token\" (UniqueName: \"kubernetes.io/projected/051c87da-acbb-4941-8fea-57107abb4720-istio-token\") pod \"051c87da-acbb-4941-8fea-57107abb4720\" (UID: \"051c87da-acbb-4941-8fea-57107abb4720\") " Oct 30 09:19:41.495473 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:41.495365 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/051c87da-acbb-4941-8fea-57107abb4720-istiod-ca-cert" pod "051c87da-acbb-4941-8fea-57107abb4720" (UID: "051c87da-acbb-4941-8fea-57107abb4720"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:19:41.495473 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:41.495453 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/051c87da-acbb-4941-8fea-57107abb4720-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/051c87da-acbb-4941-8fea-57107abb4720-workload-certs\") pod \"051c87da-acbb-4941-8fea-57107abb4720\" (UID: \"051c87da-acbb-4941-8fea-57107abb4720\") " Oct 30 09:19:41.495595 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:41.495501 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/051c87da-acbb-4941-8fea-57107abb4720-kube-api-access-k9sgg\" (UniqueName: \"kubernetes.io/projected/051c87da-acbb-4941-8fea-57107abb4720-kube-api-access-k9sgg\") pod \"051c87da-acbb-4941-8fea-57107abb4720\" (UID: \"051c87da-acbb-4941-8fea-57107abb4720\") " Oct 30 09:19:41.495820 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:41.495795 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/051c87da-acbb-4941-8fea-57107abb4720-istio-data" pod "051c87da-acbb-4941-8fea-57107abb4720" (UID: "051c87da-acbb-4941-8fea-57107abb4720"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:19:41.495892 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:41.495839 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/051c87da-acbb-4941-8fea-57107abb4720-workload-certs" pod "051c87da-acbb-4941-8fea-57107abb4720" (UID: "051c87da-acbb-4941-8fea-57107abb4720"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:19:41.496103 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:41.496082 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/051c87da-acbb-4941-8fea-57107abb4720-credential-socket" pod "051c87da-acbb-4941-8fea-57107abb4720" (UID: "051c87da-acbb-4941-8fea-57107abb4720"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:19:41.496424 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:41.496397 4562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/051c87da-acbb-4941-8fea-57107abb4720-workload-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:19:41.496511 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:41.496427 4562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/051c87da-acbb-4941-8fea-57107abb4720-istiod-ca-cert\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:19:41.496511 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:41.496442 4562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/051c87da-acbb-4941-8fea-57107abb4720-credential-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:19:41.496511 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:41.496458 4562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/051c87da-acbb-4941-8fea-57107abb4720-istio-data\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:19:41.496511 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:41.496470 4562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/051c87da-acbb-4941-8fea-57107abb4720-workload-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:19:41.497116 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:41.497091 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/051c87da-acbb-4941-8fea-57107abb4720-istio-envoy" pod "051c87da-acbb-4941-8fea-57107abb4720" (UID: "051c87da-acbb-4941-8fea-57107abb4720"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:19:41.499025 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:41.498998 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/051c87da-acbb-4941-8fea-57107abb4720-istio-token" pod "051c87da-acbb-4941-8fea-57107abb4720" (UID: "051c87da-acbb-4941-8fea-57107abb4720"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:19:41.499167 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:41.499146 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/051c87da-acbb-4941-8fea-57107abb4720-istio-podinfo" pod "051c87da-acbb-4941-8fea-57107abb4720" (UID: "051c87da-acbb-4941-8fea-57107abb4720"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:19:41.500707 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:41.500685 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/051c87da-acbb-4941-8fea-57107abb4720-kube-api-access-k9sgg" pod "051c87da-acbb-4941-8fea-57107abb4720" (UID: "051c87da-acbb-4941-8fea-57107abb4720"). InnerVolumeSpecName "kube-api-access-k9sgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:19:41.599280 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:41.598532 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9sgg\" (UniqueName: \"kubernetes.io/projected/051c87da-acbb-4941-8fea-57107abb4720-kube-api-access-k9sgg\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:19:41.599280 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:41.598589 4562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/051c87da-acbb-4941-8fea-57107abb4720-istio-envoy\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:19:41.599280 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:41.598607 4562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/051c87da-acbb-4941-8fea-57107abb4720-istio-podinfo\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:19:41.599280 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:41.598620 4562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/051c87da-acbb-4941-8fea-57107abb4720-istio-token\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:19:41.711135 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:41.711102 4562 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 09:19:41.711282 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:41.711151 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh" podUID="99f76ea3-743e-4536-9443-2454af9db849" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 09:19:41.820539 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:41.820502 4562 generic.go:356] "Generic (PLEG): container finished" podID="051c87da-acbb-4941-8fea-57107abb4720" containerID="310c6d2b893a6aad248318e2cc5beac3acff5e06d7cb6df6b723a2acea71d756" exitCode=0 Oct 30 09:19:41.820697 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:41.820587 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vbklx" event={"ID":"051c87da-acbb-4941-8fea-57107abb4720","Type":"ContainerDied","Data":"310c6d2b893a6aad248318e2cc5beac3acff5e06d7cb6df6b723a2acea71d756"} Oct 30 09:19:41.820697 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:41.820592 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vbklx" Oct 30 09:19:41.820697 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:41.820669 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vbklx" event={"ID":"051c87da-acbb-4941-8fea-57107abb4720","Type":"ContainerDied","Data":"383947c5c234a369635d32dea3d8450c5733a845728a4dded149e8ddc3ff9c19"} Oct 30 09:19:41.820880 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:41.820698 4562 scope.go:122] "RemoveContainer" containerID="310c6d2b893a6aad248318e2cc5beac3acff5e06d7cb6df6b723a2acea71d756" Oct 30 09:19:41.823461 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:41.823077 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g" event={"ID":"0d2f5f60-9f83-49d0-aac6-94d56afc2d56","Type":"ContainerStarted","Data":"07d2eae4fd38949486594f677cab355a35c411e53d804c78081372e1d0527f9b"} Oct 30 09:19:41.823461 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:41.823129 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g" event={"ID":"0d2f5f60-9f83-49d0-aac6-94d56afc2d56","Type":"ContainerStarted","Data":"2a408d1ac72a520551712466bf7b81a474b928ce7aed30a7f27c44b3a4bafed1"} Oct 30 09:19:41.832234 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:41.832215 4562 scope.go:122] "RemoveContainer" containerID="310c6d2b893a6aad248318e2cc5beac3acff5e06d7cb6df6b723a2acea71d756" Oct 30 09:19:41.832548 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:19:41.832529 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"310c6d2b893a6aad248318e2cc5beac3acff5e06d7cb6df6b723a2acea71d756\": container with ID starting with 310c6d2b893a6aad248318e2cc5beac3acff5e06d7cb6df6b723a2acea71d756 not found: ID does not exist" containerID="310c6d2b893a6aad248318e2cc5beac3acff5e06d7cb6df6b723a2acea71d756" Oct 30 09:19:41.832590 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:41.832557 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"310c6d2b893a6aad248318e2cc5beac3acff5e06d7cb6df6b723a2acea71d756"} err="failed to get container status \"310c6d2b893a6aad248318e2cc5beac3acff5e06d7cb6df6b723a2acea71d756\": rpc error: code = NotFound desc = could not find container \"310c6d2b893a6aad248318e2cc5beac3acff5e06d7cb6df6b723a2acea71d756\": container with ID starting with 310c6d2b893a6aad248318e2cc5beac3acff5e06d7cb6df6b723a2acea71d756 not found: ID does not exist" Oct 30 09:19:41.886115 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:41.886019 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g" podStartSLOduration=1.886006378 podStartE2EDuration="1.886006378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:19:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:19:41.883670503 +0000 UTC m=+1542.145231510" watchObservedRunningTime="2026-10-30 09:19:41.886006378 +0000 UTC m=+1542.147567368" Oct 30 09:19:41.922636 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:41.922607 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vbklx"] Oct 30 09:19:41.931909 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:41.931887 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vbklx"] Oct 30 09:19:42.598470 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:42.598430 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g" Oct 30 09:19:42.599982 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:42.599950 4562 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 09:19:42.600155 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:42.600003 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g" podUID="0d2f5f60-9f83-49d0-aac6-94d56afc2d56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 09:19:42.733580 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:42.733538 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh" Oct 30 09:19:42.735265 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:42.735242 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh" Oct 30 09:19:42.740440 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:42.736621 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh" Oct 30 09:19:43.597949 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:43.597910 4562 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 09:19:43.598118 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:43.597967 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g" podUID="0d2f5f60-9f83-49d0-aac6-94d56afc2d56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 09:19:43.874594 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:43.874505 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="051c87da-acbb-4941-8fea-57107abb4720" path="/var/lib/kubelet/pods/051c87da-acbb-4941-8fea-57107abb4720/volumes" Oct 30 09:19:44.598323 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:44.598285 4562 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 09:19:44.598524 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:44.598336 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g" podUID="0d2f5f60-9f83-49d0-aac6-94d56afc2d56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 09:19:44.812050 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:44.812021 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-648892-69475bb5d5-sthgm" Oct 30 09:19:45.598070 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:45.598031 4562 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 09:19:45.598466 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:45.598089 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g" podUID="0d2f5f60-9f83-49d0-aac6-94d56afc2d56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 09:19:45.809419 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:45.807683 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-ca0df2-7d67bcc9bb-b2brq" Oct 30 09:19:45.829403 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:45.825578 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca0df2-57b8bf5d59-tbxzz" Oct 30 09:19:46.611838 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:46.611800 4562 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 09:19:46.613223 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:46.613188 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g" podUID="0d2f5f60-9f83-49d0-aac6-94d56afc2d56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 09:19:46.857499 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:46.857457 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ca0df2-664449fb4-vshdt"] Oct 30 09:19:46.859105 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:46.859077 4562 cpu_manager.go:414] "containerMap: removing container" podUID="051c87da-acbb-4941-8fea-57107abb4720" containerName="istio-proxy" Oct 30 09:19:46.859279 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:46.859262 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="051c87da-acbb-4941-8fea-57107abb4720" containerName="istio-proxy" Oct 30 09:19:46.859590 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:46.859572 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="051c87da-acbb-4941-8fea-57107abb4720" containerName="istio-proxy" Oct 30 09:19:46.865988 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:46.865933 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca0df2-664449fb4-vshdt" Oct 30 09:19:46.884251 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:46.883861 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-ca0df2-serving-cert\"" Oct 30 09:19:46.892901 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:46.892872 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ca0df2-664449fb4-vshdt"] Oct 30 09:19:47.034412 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:47.031655 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9af7addc-7b8d-4204-892f-8225b90ab5aa-maas-api-tls\") pod \"maas-api-e2e-api-a-ca0df2-664449fb4-vshdt\" (UID: \"9af7addc-7b8d-4204-892f-8225b90ab5aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca0df2-664449fb4-vshdt" Oct 30 09:19:47.034412 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:47.031725 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnzp4\" (UniqueName: \"kubernetes.io/projected/9af7addc-7b8d-4204-892f-8225b90ab5aa-kube-api-access-dnzp4\") pod \"maas-api-e2e-api-a-ca0df2-664449fb4-vshdt\" (UID: \"9af7addc-7b8d-4204-892f-8225b90ab5aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca0df2-664449fb4-vshdt" Oct 30 09:19:47.034412 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:47.031849 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9af7addc-7b8d-4204-892f-8225b90ab5aa-openshift-service-ca\") pod \"maas-api-e2e-api-a-ca0df2-664449fb4-vshdt\" (UID: \"9af7addc-7b8d-4204-892f-8225b90ab5aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca0df2-664449fb4-vshdt" Oct 30 09:19:47.034412 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:47.031885 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9af7addc-7b8d-4204-892f-8225b90ab5aa-metrics-certs\") pod \"maas-api-e2e-api-a-ca0df2-664449fb4-vshdt\" (UID: \"9af7addc-7b8d-4204-892f-8225b90ab5aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca0df2-664449fb4-vshdt" Oct 30 09:19:47.289486 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:47.210137 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9af7addc-7b8d-4204-892f-8225b90ab5aa-maas-api-tls\") pod \"maas-api-e2e-api-a-ca0df2-664449fb4-vshdt\" (UID: \"9af7addc-7b8d-4204-892f-8225b90ab5aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca0df2-664449fb4-vshdt" Oct 30 09:19:47.289486 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:47.210206 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnzp4\" (UniqueName: \"kubernetes.io/projected/9af7addc-7b8d-4204-892f-8225b90ab5aa-kube-api-access-dnzp4\") pod \"maas-api-e2e-api-a-ca0df2-664449fb4-vshdt\" (UID: \"9af7addc-7b8d-4204-892f-8225b90ab5aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca0df2-664449fb4-vshdt" Oct 30 09:19:47.289486 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:47.210315 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9af7addc-7b8d-4204-892f-8225b90ab5aa-openshift-service-ca\") pod \"maas-api-e2e-api-a-ca0df2-664449fb4-vshdt\" (UID: \"9af7addc-7b8d-4204-892f-8225b90ab5aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca0df2-664449fb4-vshdt" Oct 30 09:19:47.289486 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:47.210357 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9af7addc-7b8d-4204-892f-8225b90ab5aa-metrics-certs\") pod \"maas-api-e2e-api-a-ca0df2-664449fb4-vshdt\" (UID: \"9af7addc-7b8d-4204-892f-8225b90ab5aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca0df2-664449fb4-vshdt" Oct 30 09:19:47.289486 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:19:47.210966 4562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-ca0df2-serving-cert: secret "maas-api-e2e-api-a-ca0df2-serving-cert" not found Oct 30 09:19:47.289486 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:19:47.211039 4562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9af7addc-7b8d-4204-892f-8225b90ab5aa-maas-api-tls podName:9af7addc-7b8d-4204-892f-8225b90ab5aa nodeName:}" failed. No retries permitted until 2026-10-30 09:19:47.711017462 +0000 UTC m=+1547.972578442 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9af7addc-7b8d-4204-892f-8225b90ab5aa-maas-api-tls") pod "maas-api-e2e-api-a-ca0df2-664449fb4-vshdt" (UID: "9af7addc-7b8d-4204-892f-8225b90ab5aa") : secret "maas-api-e2e-api-a-ca0df2-serving-cert" not found Oct 30 09:19:47.289486 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:47.211895 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9af7addc-7b8d-4204-892f-8225b90ab5aa-openshift-service-ca\") pod \"maas-api-e2e-api-a-ca0df2-664449fb4-vshdt\" (UID: \"9af7addc-7b8d-4204-892f-8225b90ab5aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca0df2-664449fb4-vshdt" Oct 30 09:19:47.468626 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:47.439142 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9af7addc-7b8d-4204-892f-8225b90ab5aa-metrics-certs\") pod \"maas-api-e2e-api-a-ca0df2-664449fb4-vshdt\" (UID: \"9af7addc-7b8d-4204-892f-8225b90ab5aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca0df2-664449fb4-vshdt" Oct 30 09:19:47.468626 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:47.442280 4562 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb" podUID="a540c86f-c856-42a8-9f73-b5b9b690f42b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:47.497407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:47.483847 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnzp4\" (UniqueName: \"kubernetes.io/projected/9af7addc-7b8d-4204-892f-8225b90ab5aa-kube-api-access-dnzp4\") pod \"maas-api-e2e-api-a-ca0df2-664449fb4-vshdt\" (UID: \"9af7addc-7b8d-4204-892f-8225b90ab5aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca0df2-664449fb4-vshdt" Oct 30 09:19:47.528833 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:47.527512 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb" podUID="a540c86f-c856-42a8-9f73-b5b9b690f42b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:47.605093 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:47.604996 4562 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 09:19:47.605353 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:47.605309 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g" podUID="0d2f5f60-9f83-49d0-aac6-94d56afc2d56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 09:19:47.793407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:47.785923 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9af7addc-7b8d-4204-892f-8225b90ab5aa-maas-api-tls\") pod \"maas-api-e2e-api-a-ca0df2-664449fb4-vshdt\" (UID: \"9af7addc-7b8d-4204-892f-8225b90ab5aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca0df2-664449fb4-vshdt" Oct 30 09:19:47.820008 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:47.819970 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9af7addc-7b8d-4204-892f-8225b90ab5aa-maas-api-tls\") pod \"maas-api-e2e-api-a-ca0df2-664449fb4-vshdt\" (UID: \"9af7addc-7b8d-4204-892f-8225b90ab5aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca0df2-664449fb4-vshdt" Oct 30 09:19:48.108448 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:48.107219 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca0df2-664449fb4-vshdt" Oct 30 09:19:48.566957 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:48.566900 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb" podUID="a540c86f-c856-42a8-9f73-b5b9b690f42b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:48.568228 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:48.568200 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb" Oct 30 09:19:48.617281 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:48.617225 4562 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 09:19:48.617599 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:48.617565 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g" podUID="0d2f5f60-9f83-49d0-aac6-94d56afc2d56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 09:19:48.771406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:48.765861 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/distinct-2-w1-c5447d-kserve-676749d548-xlmsf" podUID="c9ff897f-c520-4d0f-8820-e185b1d47dac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:48.792431 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:48.773548 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-176b6c/distinct-w3-176b6c-kserve-f57b7579c-k4pc5" podUID="28fd4072-399c-4be5-a0d8-1be6966da4af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Oct 30 09:19:48.933570 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:48.933487 4562 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 09:19:49.058658 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:49.058603 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf" podUID="cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:49.449923 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:49.449878 4562 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 09:19:49.450254 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:49.450216 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb" podUID="28431c22-78da-4145-9842-e7d92b360558" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:49.545553 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:49.545502 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55" podUID="88be34c3-a9e9-4489-a449-206a99ef1663" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:49.607710 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:49.591192 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ca0df2-664449fb4-vshdt"] Oct 30 09:19:49.607710 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:49.596511 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb" podUID="a540c86f-c856-42a8-9f73-b5b9b690f42b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:49.625437 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:49.616731 4562 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 09:19:49.625437 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:49.616802 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g" podUID="0d2f5f60-9f83-49d0-aac6-94d56afc2d56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 09:19:49.777643 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:49.777589 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-176b6c/distinct-w3-176b6c-kserve-f57b7579c-k4pc5" podUID="28fd4072-399c-4be5-a0d8-1be6966da4af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:49.851409 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:49.807614 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/distinct-2-w1-c5447d-kserve-676749d548-xlmsf" podUID="c9ff897f-c520-4d0f-8820-e185b1d47dac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Oct 30 09:19:50.011805 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:49.832567 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-176b6c/distinct-w3-176b6c-kserve-f57b7579c-k4pc5" Oct 30 09:19:50.163452 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:50.154402 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c5447d/distinct-2-w1-c5447d-kserve-676749d548-xlmsf" Oct 30 09:19:50.251455 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:50.251427 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb" Oct 30 09:19:50.583211 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:50.583162 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55" podUID="88be34c3-a9e9-4489-a449-206a99ef1663" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 09:19:50.638625 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:50.638585 4562 patch_prober.go:29] interesting pod/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 09:19:50.638937 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:50.638889 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g" podUID="0d2f5f60-9f83-49d0-aac6-94d56afc2d56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 09:19:50.812574 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:50.812523 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-176b6c/distinct-w3-176b6c-kserve-f57b7579c-k4pc5" podUID="28fd4072-399c-4be5-a0d8-1be6966da4af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:50.868047 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:50.867942 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55" Oct 30 09:19:50.899654 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:50.899602 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca0df2-664449fb4-vshdt" event={"ID":"9af7addc-7b8d-4204-892f-8225b90ab5aa","Type":"ContainerStarted","Data":"1a381e8f1ddd749626bd45dc831bd7c369ad84803701d7c8e103542d8c01f821"} Oct 30 09:19:51.008583 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:51.008530 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/distinct-2-w1-c5447d-kserve-676749d548-xlmsf" podUID="c9ff897f-c520-4d0f-8820-e185b1d47dac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:51.181977 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:51.180469 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb" podUID="a540c86f-c856-42a8-9f73-b5b9b690f42b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:51.587421 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:51.584631 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-176b6c/distinct-w3-176b6c-kserve-f57b7579c-k4pc5" Oct 30 09:19:51.587421 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:51.585724 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55" podUID="88be34c3-a9e9-4489-a449-206a99ef1663" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:19:51.716432 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:51.715802 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c5447d/distinct-2-w1-c5447d-kserve-676749d548-xlmsf" Oct 30 09:19:51.848546 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:51.848515 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55" Oct 30 09:19:51.936797 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:51.936768 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca0df2-664449fb4-vshdt" Oct 30 09:19:52.086456 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:52.059275 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca0df2-664449fb4-vshdt" podStartSLOduration=5.059781265 podStartE2EDuration="6.059247352s" totalImagesPullingTime="999.466087ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:19:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:19:52.037433604 +0000 UTC m=+1552.298994613" watchObservedRunningTime="2026-10-30 09:19:52.059247352 +0000 UTC m=+1552.320808343" Oct 30 09:19:52.427443 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:52.427311 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g" Oct 30 09:19:52.429365 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:52.429340 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g" Oct 30 09:19:52.430828 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:52.430807 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g" Oct 30 09:19:52.945157 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:52.945103 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca0df2-664449fb4-vshdt" event={"ID":"9af7addc-7b8d-4204-892f-8225b90ab5aa","Type":"ContainerStarted","Data":"fdfc96da80ae38619963654ec022b46cb93175a1a55f8f92e8b7a8403cb9ff71"} Oct 30 09:19:58.667004 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:19:58.666958 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:19:58.958476 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:19:58.958448 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca0df2-664449fb4-vshdt" Oct 30 09:20:04.396630 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:04.396599 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn"] Oct 30 09:20:04.400822 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:04.400802 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn" Oct 30 09:20:04.418401 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:04.418360 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-6c9de5-openshift-default-dockercfg-dsj4l\"" Oct 30 09:20:04.435219 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:04.435193 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn"] Oct 30 09:20:04.530408 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:04.511032 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/357b42b1-a9e1-423c-96b4-3bb155fed864-workload-socket\") pod \"e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn\" (UID: \"357b42b1-a9e1-423c-96b4-3bb155fed864\") " pod="openshift-ingress/e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn" Oct 30 09:20:04.546756 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:04.546716 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/357b42b1-a9e1-423c-96b4-3bb155fed864-istio-data\") pod \"e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn\" (UID: \"357b42b1-a9e1-423c-96b4-3bb155fed864\") " pod="openshift-ingress/e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn" Oct 30 09:20:04.547078 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:04.547058 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/357b42b1-a9e1-423c-96b4-3bb155fed864-istio-token\") pod \"e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn\" (UID: \"357b42b1-a9e1-423c-96b4-3bb155fed864\") " pod="openshift-ingress/e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn" Oct 30 09:20:04.547296 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:04.547277 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/357b42b1-a9e1-423c-96b4-3bb155fed864-credential-socket\") pod \"e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn\" (UID: \"357b42b1-a9e1-423c-96b4-3bb155fed864\") " pod="openshift-ingress/e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn" Oct 30 09:20:04.547544 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:04.547522 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/357b42b1-a9e1-423c-96b4-3bb155fed864-istio-podinfo\") pod \"e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn\" (UID: \"357b42b1-a9e1-423c-96b4-3bb155fed864\") " pod="openshift-ingress/e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn" Oct 30 09:20:04.570615 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:04.570579 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/357b42b1-a9e1-423c-96b4-3bb155fed864-workload-certs\") pod \"e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn\" (UID: \"357b42b1-a9e1-423c-96b4-3bb155fed864\") " pod="openshift-ingress/e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn" Oct 30 09:20:04.571018 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:04.570989 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/357b42b1-a9e1-423c-96b4-3bb155fed864-istio-envoy\") pod \"e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn\" (UID: \"357b42b1-a9e1-423c-96b4-3bb155fed864\") " pod="openshift-ingress/e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn" Oct 30 09:20:04.571272 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:04.571253 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgbvb\" (UniqueName: \"kubernetes.io/projected/357b42b1-a9e1-423c-96b4-3bb155fed864-kube-api-access-qgbvb\") pod \"e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn\" (UID: \"357b42b1-a9e1-423c-96b4-3bb155fed864\") " pod="openshift-ingress/e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn" Oct 30 09:20:04.571541 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:04.571522 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/357b42b1-a9e1-423c-96b4-3bb155fed864-istiod-ca-cert\") pod \"e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn\" (UID: \"357b42b1-a9e1-423c-96b4-3bb155fed864\") " pod="openshift-ingress/e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn" Oct 30 09:20:04.727414 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:04.727338 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/357b42b1-a9e1-423c-96b4-3bb155fed864-istiod-ca-cert\") pod \"e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn\" (UID: \"357b42b1-a9e1-423c-96b4-3bb155fed864\") " pod="openshift-ingress/e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn" Oct 30 09:20:04.727608 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:04.727476 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/357b42b1-a9e1-423c-96b4-3bb155fed864-workload-socket\") pod \"e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn\" (UID: \"357b42b1-a9e1-423c-96b4-3bb155fed864\") " pod="openshift-ingress/e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn" Oct 30 09:20:04.727608 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:04.727528 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/357b42b1-a9e1-423c-96b4-3bb155fed864-istio-data\") pod \"e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn\" (UID: \"357b42b1-a9e1-423c-96b4-3bb155fed864\") " pod="openshift-ingress/e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn" Oct 30 09:20:04.727608 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:04.727572 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/357b42b1-a9e1-423c-96b4-3bb155fed864-istio-token\") pod \"e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn\" (UID: \"357b42b1-a9e1-423c-96b4-3bb155fed864\") " pod="openshift-ingress/e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn" Oct 30 09:20:04.727786 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:04.727641 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/357b42b1-a9e1-423c-96b4-3bb155fed864-credential-socket\") pod \"e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn\" (UID: \"357b42b1-a9e1-423c-96b4-3bb155fed864\") " pod="openshift-ingress/e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn" Oct 30 09:20:04.727786 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:04.727718 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/357b42b1-a9e1-423c-96b4-3bb155fed864-istio-podinfo\") pod \"e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn\" (UID: \"357b42b1-a9e1-423c-96b4-3bb155fed864\") " pod="openshift-ingress/e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn" Oct 30 09:20:04.727786 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:04.727763 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/357b42b1-a9e1-423c-96b4-3bb155fed864-workload-certs\") pod \"e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn\" (UID: \"357b42b1-a9e1-423c-96b4-3bb155fed864\") " pod="openshift-ingress/e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn" Oct 30 09:20:04.727932 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:04.727879 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/357b42b1-a9e1-423c-96b4-3bb155fed864-istio-envoy\") pod \"e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn\" (UID: \"357b42b1-a9e1-423c-96b4-3bb155fed864\") " pod="openshift-ingress/e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn" Oct 30 09:20:04.727988 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:04.727965 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgbvb\" (UniqueName: \"kubernetes.io/projected/357b42b1-a9e1-423c-96b4-3bb155fed864-kube-api-access-qgbvb\") pod \"e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn\" (UID: \"357b42b1-a9e1-423c-96b4-3bb155fed864\") " pod="openshift-ingress/e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn" Oct 30 09:20:04.733403 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:04.729701 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/357b42b1-a9e1-423c-96b4-3bb155fed864-istio-data\") pod \"e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn\" (UID: \"357b42b1-a9e1-423c-96b4-3bb155fed864\") " pod="openshift-ingress/e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn" Oct 30 09:20:04.733403 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:04.729981 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/357b42b1-a9e1-423c-96b4-3bb155fed864-credential-socket\") pod \"e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn\" (UID: \"357b42b1-a9e1-423c-96b4-3bb155fed864\") " pod="openshift-ingress/e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn" Oct 30 09:20:04.733403 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:04.730119 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/357b42b1-a9e1-423c-96b4-3bb155fed864-istiod-ca-cert\") pod \"e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn\" (UID: \"357b42b1-a9e1-423c-96b4-3bb155fed864\") " pod="openshift-ingress/e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn" Oct 30 09:20:04.733403 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:04.730306 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/357b42b1-a9e1-423c-96b4-3bb155fed864-workload-certs\") pod \"e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn\" (UID: \"357b42b1-a9e1-423c-96b4-3bb155fed864\") " pod="openshift-ingress/e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn" Oct 30 09:20:04.748944 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:04.748864 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/357b42b1-a9e1-423c-96b4-3bb155fed864-workload-socket\") pod \"e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn\" (UID: \"357b42b1-a9e1-423c-96b4-3bb155fed864\") " pod="openshift-ingress/e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn" Oct 30 09:20:04.752114 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:04.752078 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgbvb\" (UniqueName: \"kubernetes.io/projected/357b42b1-a9e1-423c-96b4-3bb155fed864-kube-api-access-qgbvb\") pod \"e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn\" (UID: \"357b42b1-a9e1-423c-96b4-3bb155fed864\") " pod="openshift-ingress/e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn" Oct 30 09:20:04.778979 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:04.778943 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/357b42b1-a9e1-423c-96b4-3bb155fed864-istio-podinfo\") pod \"e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn\" (UID: \"357b42b1-a9e1-423c-96b4-3bb155fed864\") " pod="openshift-ingress/e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn" Oct 30 09:20:04.863201 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:04.863154 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/357b42b1-a9e1-423c-96b4-3bb155fed864-istio-envoy\") pod \"e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn\" (UID: \"357b42b1-a9e1-423c-96b4-3bb155fed864\") " pod="openshift-ingress/e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn" Oct 30 09:20:04.864891 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:04.864835 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/357b42b1-a9e1-423c-96b4-3bb155fed864-istio-token\") pod \"e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn\" (UID: \"357b42b1-a9e1-423c-96b4-3bb155fed864\") " pod="openshift-ingress/e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn" Oct 30 09:20:05.015441 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:05.015347 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn" Oct 30 09:20:05.359712 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:20:05.359682 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod357b42b1_a9e1_423c_96b4_3bb155fed864.slice/crio-a2b66918eb8f378c094a17e2aebf673a0bb91e5a521f9a97511cac5a22419c0c WatchSource:0}: Error finding container a2b66918eb8f378c094a17e2aebf673a0bb91e5a521f9a97511cac5a22419c0c: Status 404 returned error can't find the container with id a2b66918eb8f378c094a17e2aebf673a0bb91e5a521f9a97511cac5a22419c0c Oct 30 09:20:05.360777 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:05.360747 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn"] Oct 30 09:20:05.362361 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:05.362325 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:20:05.362518 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:05.362439 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:20:05.362518 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:05.362494 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:20:06.122885 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:06.122850 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn" event={"ID":"357b42b1-a9e1-423c-96b4-3bb155fed864","Type":"ContainerStarted","Data":"774209cf60805d8c3fed9f3186e0ca6185e67bba1e26e9f85705c968d4c6b23c"} Oct 30 09:20:06.123347 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:06.122906 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn" event={"ID":"357b42b1-a9e1-423c-96b4-3bb155fed864","Type":"ContainerStarted","Data":"a2b66918eb8f378c094a17e2aebf673a0bb91e5a521f9a97511cac5a22419c0c"} Oct 30 09:20:06.183109 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:06.183054 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn" podStartSLOduration=2.183038109 podStartE2EDuration="2.183038109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:20:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:20:06.18131776 +0000 UTC m=+1566.442878768" watchObservedRunningTime="2026-10-30 09:20:06.183038109 +0000 UTC m=+1566.444599096" Oct 30 09:20:06.263569 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:06.263538 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fh6qf"] Oct 30 09:20:06.268290 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:06.268266 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fh6qf" Oct 30 09:20:06.293423 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:06.290677 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fh6qf"] Oct 30 09:20:06.388582 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:06.388500 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beaa426c-dd69-4ab1-8776-c2512f0ff19d-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-fh6qf\" (UID: \"beaa426c-dd69-4ab1-8776-c2512f0ff19d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fh6qf" Oct 30 09:20:06.388754 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:06.388583 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beaa426c-dd69-4ab1-8776-c2512f0ff19d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-fh6qf\" (UID: \"beaa426c-dd69-4ab1-8776-c2512f0ff19d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fh6qf" Oct 30 09:20:06.388754 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:06.388600 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6gf2\" (UniqueName: \"kubernetes.io/projected/beaa426c-dd69-4ab1-8776-c2512f0ff19d-kube-api-access-j6gf2\") pod \"maas-default-gateway-openshift-default-77f556d647-fh6qf\" (UID: \"beaa426c-dd69-4ab1-8776-c2512f0ff19d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fh6qf" Oct 30 09:20:06.388754 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:06.388619 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beaa426c-dd69-4ab1-8776-c2512f0ff19d-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-fh6qf\" (UID: \"beaa426c-dd69-4ab1-8776-c2512f0ff19d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fh6qf" Oct 30 09:20:06.388754 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:06.388653 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beaa426c-dd69-4ab1-8776-c2512f0ff19d-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-fh6qf\" (UID: \"beaa426c-dd69-4ab1-8776-c2512f0ff19d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fh6qf" Oct 30 09:20:06.388754 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:06.388733 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beaa426c-dd69-4ab1-8776-c2512f0ff19d-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-fh6qf\" (UID: \"beaa426c-dd69-4ab1-8776-c2512f0ff19d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fh6qf" Oct 30 09:20:06.388998 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:06.388759 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beaa426c-dd69-4ab1-8776-c2512f0ff19d-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-fh6qf\" (UID: \"beaa426c-dd69-4ab1-8776-c2512f0ff19d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fh6qf" Oct 30 09:20:06.388998 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:06.388799 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beaa426c-dd69-4ab1-8776-c2512f0ff19d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-fh6qf\" (UID: \"beaa426c-dd69-4ab1-8776-c2512f0ff19d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fh6qf" Oct 30 09:20:06.388998 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:06.388822 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beaa426c-dd69-4ab1-8776-c2512f0ff19d-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-fh6qf\" (UID: \"beaa426c-dd69-4ab1-8776-c2512f0ff19d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fh6qf" Oct 30 09:20:06.489714 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:06.489684 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beaa426c-dd69-4ab1-8776-c2512f0ff19d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-fh6qf\" (UID: \"beaa426c-dd69-4ab1-8776-c2512f0ff19d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fh6qf" Oct 30 09:20:06.489881 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:06.489717 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6gf2\" (UniqueName: \"kubernetes.io/projected/beaa426c-dd69-4ab1-8776-c2512f0ff19d-kube-api-access-j6gf2\") pod \"maas-default-gateway-openshift-default-77f556d647-fh6qf\" (UID: \"beaa426c-dd69-4ab1-8776-c2512f0ff19d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fh6qf" Oct 30 09:20:06.489881 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:06.489742 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beaa426c-dd69-4ab1-8776-c2512f0ff19d-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-fh6qf\" (UID: \"beaa426c-dd69-4ab1-8776-c2512f0ff19d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fh6qf" Oct 30 09:20:06.489881 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:06.489801 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beaa426c-dd69-4ab1-8776-c2512f0ff19d-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-fh6qf\" (UID: \"beaa426c-dd69-4ab1-8776-c2512f0ff19d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fh6qf" Oct 30 09:20:06.489997 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:06.489938 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beaa426c-dd69-4ab1-8776-c2512f0ff19d-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-fh6qf\" (UID: \"beaa426c-dd69-4ab1-8776-c2512f0ff19d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fh6qf" Oct 30 09:20:06.489997 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:06.489980 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beaa426c-dd69-4ab1-8776-c2512f0ff19d-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-fh6qf\" (UID: \"beaa426c-dd69-4ab1-8776-c2512f0ff19d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fh6qf" Oct 30 09:20:06.490091 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:06.490043 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beaa426c-dd69-4ab1-8776-c2512f0ff19d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-fh6qf\" (UID: \"beaa426c-dd69-4ab1-8776-c2512f0ff19d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fh6qf" Oct 30 09:20:06.490091 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:06.490069 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beaa426c-dd69-4ab1-8776-c2512f0ff19d-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-fh6qf\" (UID: \"beaa426c-dd69-4ab1-8776-c2512f0ff19d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fh6qf" Oct 30 09:20:06.490170 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:06.490110 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beaa426c-dd69-4ab1-8776-c2512f0ff19d-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-fh6qf\" (UID: \"beaa426c-dd69-4ab1-8776-c2512f0ff19d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fh6qf" Oct 30 09:20:06.490314 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:06.490261 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beaa426c-dd69-4ab1-8776-c2512f0ff19d-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-fh6qf\" (UID: \"beaa426c-dd69-4ab1-8776-c2512f0ff19d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fh6qf" Oct 30 09:20:06.490407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:06.490320 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beaa426c-dd69-4ab1-8776-c2512f0ff19d-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-fh6qf\" (UID: \"beaa426c-dd69-4ab1-8776-c2512f0ff19d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fh6qf" Oct 30 09:20:06.490463 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:06.490430 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beaa426c-dd69-4ab1-8776-c2512f0ff19d-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-fh6qf\" (UID: \"beaa426c-dd69-4ab1-8776-c2512f0ff19d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fh6qf" Oct 30 09:20:06.490537 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:06.490517 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beaa426c-dd69-4ab1-8776-c2512f0ff19d-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-fh6qf\" (UID: \"beaa426c-dd69-4ab1-8776-c2512f0ff19d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fh6qf" Oct 30 09:20:06.490799 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:06.490772 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beaa426c-dd69-4ab1-8776-c2512f0ff19d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-fh6qf\" (UID: \"beaa426c-dd69-4ab1-8776-c2512f0ff19d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fh6qf" Oct 30 09:20:06.492394 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:06.492348 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beaa426c-dd69-4ab1-8776-c2512f0ff19d-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-fh6qf\" (UID: \"beaa426c-dd69-4ab1-8776-c2512f0ff19d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fh6qf" Oct 30 09:20:06.492488 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:06.492397 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beaa426c-dd69-4ab1-8776-c2512f0ff19d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-fh6qf\" (UID: \"beaa426c-dd69-4ab1-8776-c2512f0ff19d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fh6qf" Oct 30 09:20:06.510531 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:06.510506 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beaa426c-dd69-4ab1-8776-c2512f0ff19d-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-fh6qf\" (UID: \"beaa426c-dd69-4ab1-8776-c2512f0ff19d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fh6qf" Oct 30 09:20:06.510758 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:06.510737 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6gf2\" (UniqueName: \"kubernetes.io/projected/beaa426c-dd69-4ab1-8776-c2512f0ff19d-kube-api-access-j6gf2\") pod \"maas-default-gateway-openshift-default-77f556d647-fh6qf\" (UID: \"beaa426c-dd69-4ab1-8776-c2512f0ff19d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fh6qf" Oct 30 09:20:06.583129 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:06.583097 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fh6qf" Oct 30 09:20:06.919436 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:06.919403 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fh6qf"] Oct 30 09:20:06.928615 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:06.928581 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:20:06.928707 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:06.928650 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:20:06.928707 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:06.928678 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:20:07.016414 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:07.016339 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn" Oct 30 09:20:07.023175 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:07.023150 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn" Oct 30 09:20:07.174359 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:07.173454 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fh6qf" event={"ID":"beaa426c-dd69-4ab1-8776-c2512f0ff19d","Type":"ContainerStarted","Data":"e50a38edde0f298df418766d799f723e9eb98535b06411fd585ac0b63762564c"} Oct 30 09:20:07.174359 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:07.173513 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fh6qf" event={"ID":"beaa426c-dd69-4ab1-8776-c2512f0ff19d","Type":"ContainerStarted","Data":"ed1595ab6867c252981a1b83900495f2ad1508a090f5c61720a06f6b9e51f7ba"} Oct 30 09:20:07.174359 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:07.173541 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn" Oct 30 09:20:07.174946 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:07.174852 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn" Oct 30 09:20:07.221542 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:07.221481 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fh6qf" podStartSLOduration=1.221464177 podStartE2EDuration="1.221464177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:20:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:20:07.217748827 +0000 UTC m=+1567.479309814" watchObservedRunningTime="2026-10-30 09:20:07.221464177 +0000 UTC m=+1567.483025167" Oct 30 09:20:07.583749 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:07.583708 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fh6qf" Oct 30 09:20:07.585658 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:07.585626 4562 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-fh6qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 09:20:07.585803 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:07.585682 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fh6qf" podUID="beaa426c-dd69-4ab1-8776-c2512f0ff19d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 09:20:08.583760 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:08.583723 4562 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-fh6qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 09:20:08.584170 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:08.583778 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fh6qf" podUID="beaa426c-dd69-4ab1-8776-c2512f0ff19d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 09:20:09.583703 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:09.583663 4562 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-fh6qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 09:20:09.583911 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:09.583721 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fh6qf" podUID="beaa426c-dd69-4ab1-8776-c2512f0ff19d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 09:20:10.584188 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:10.584155 4562 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-fh6qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 09:20:10.584684 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:10.584205 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fh6qf" podUID="beaa426c-dd69-4ab1-8776-c2512f0ff19d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 09:20:11.425436 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:11.425403 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6c9de5-6fbf95ccff-sc8sg"] Oct 30 09:20:11.429916 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:11.429895 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6c9de5-6fbf95ccff-sc8sg" Oct 30 09:20:11.568417 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:11.565849 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-6c9de5-dockercfg-zkrw7\"" Oct 30 09:20:11.568417 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:11.566221 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-6c9de5\"" Oct 30 09:20:11.641633 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:11.578894 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6c9de5-6fbf95ccff-sc8sg"] Oct 30 09:20:11.641633 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:11.604008 4562 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-fh6qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 09:20:11.641633 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:11.604072 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fh6qf" podUID="beaa426c-dd69-4ab1-8776-c2512f0ff19d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 09:20:11.641633 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:11.618439 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szfvd\" (UniqueName: \"kubernetes.io/projected/096665bc-2bb1-46b5-8561-1371424b699d-kube-api-access-szfvd\") pod \"payload-pre-processing-e2e-api-b-6c9de5-6fbf95ccff-sc8sg\" (UID: \"096665bc-2bb1-46b5-8561-1371424b699d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6c9de5-6fbf95ccff-sc8sg" Oct 30 09:20:11.641633 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:11.618528 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/096665bc-2bb1-46b5-8561-1371424b699d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-6c9de5-6fbf95ccff-sc8sg\" (UID: \"096665bc-2bb1-46b5-8561-1371424b699d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6c9de5-6fbf95ccff-sc8sg" Oct 30 09:20:11.802440 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:11.719862 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szfvd\" (UniqueName: \"kubernetes.io/projected/096665bc-2bb1-46b5-8561-1371424b699d-kube-api-access-szfvd\") pod \"payload-pre-processing-e2e-api-b-6c9de5-6fbf95ccff-sc8sg\" (UID: \"096665bc-2bb1-46b5-8561-1371424b699d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6c9de5-6fbf95ccff-sc8sg" Oct 30 09:20:11.802653 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:11.802613 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/096665bc-2bb1-46b5-8561-1371424b699d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-6c9de5-6fbf95ccff-sc8sg\" (UID: \"096665bc-2bb1-46b5-8561-1371424b699d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6c9de5-6fbf95ccff-sc8sg" Oct 30 09:20:11.803702 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:11.803677 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/096665bc-2bb1-46b5-8561-1371424b699d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-6c9de5-6fbf95ccff-sc8sg\" (UID: \"096665bc-2bb1-46b5-8561-1371424b699d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6c9de5-6fbf95ccff-sc8sg" Oct 30 09:20:11.806529 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:11.806457 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6c9de5-65c4494864-c5fmz"] Oct 30 09:20:11.811898 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:11.811874 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6c9de5-65c4494864-c5fmz"] Oct 30 09:20:11.812036 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:11.811982 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6c9de5-65c4494864-c5fmz" Oct 30 09:20:11.815201 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:11.815177 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szfvd\" (UniqueName: \"kubernetes.io/projected/096665bc-2bb1-46b5-8561-1371424b699d-kube-api-access-szfvd\") pod \"payload-pre-processing-e2e-api-b-6c9de5-6fbf95ccff-sc8sg\" (UID: \"096665bc-2bb1-46b5-8561-1371424b699d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6c9de5-6fbf95ccff-sc8sg" Oct 30 09:20:11.928232 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:11.927073 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swkgk\" (UniqueName: \"kubernetes.io/projected/6091ee8a-f7ce-4ea4-a8eb-44d13b5b1bdc-kube-api-access-swkgk\") pod \"payload-processing-e2e-api-b-6c9de5-65c4494864-c5fmz\" (UID: \"6091ee8a-f7ce-4ea4-a8eb-44d13b5b1bdc\") " pod="openshift-ingress/payload-processing-e2e-api-b-6c9de5-65c4494864-c5fmz" Oct 30 09:20:11.930118 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:11.929302 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6091ee8a-f7ce-4ea4-a8eb-44d13b5b1bdc-plugins-config-volume\") pod \"payload-processing-e2e-api-b-6c9de5-65c4494864-c5fmz\" (UID: \"6091ee8a-f7ce-4ea4-a8eb-44d13b5b1bdc\") " pod="openshift-ingress/payload-processing-e2e-api-b-6c9de5-65c4494864-c5fmz" Oct 30 09:20:12.030909 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:12.030840 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swkgk\" (UniqueName: \"kubernetes.io/projected/6091ee8a-f7ce-4ea4-a8eb-44d13b5b1bdc-kube-api-access-swkgk\") pod \"payload-processing-e2e-api-b-6c9de5-65c4494864-c5fmz\" (UID: \"6091ee8a-f7ce-4ea4-a8eb-44d13b5b1bdc\") " pod="openshift-ingress/payload-processing-e2e-api-b-6c9de5-65c4494864-c5fmz" Oct 30 09:20:12.031362 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:12.031339 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6091ee8a-f7ce-4ea4-a8eb-44d13b5b1bdc-plugins-config-volume\") pod \"payload-processing-e2e-api-b-6c9de5-65c4494864-c5fmz\" (UID: \"6091ee8a-f7ce-4ea4-a8eb-44d13b5b1bdc\") " pod="openshift-ingress/payload-processing-e2e-api-b-6c9de5-65c4494864-c5fmz" Oct 30 09:20:12.032289 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:12.032262 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6091ee8a-f7ce-4ea4-a8eb-44d13b5b1bdc-plugins-config-volume\") pod \"payload-processing-e2e-api-b-6c9de5-65c4494864-c5fmz\" (UID: \"6091ee8a-f7ce-4ea4-a8eb-44d13b5b1bdc\") " pod="openshift-ingress/payload-processing-e2e-api-b-6c9de5-65c4494864-c5fmz" Oct 30 09:20:12.069481 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:12.069343 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swkgk\" (UniqueName: \"kubernetes.io/projected/6091ee8a-f7ce-4ea4-a8eb-44d13b5b1bdc-kube-api-access-swkgk\") pod \"payload-processing-e2e-api-b-6c9de5-65c4494864-c5fmz\" (UID: \"6091ee8a-f7ce-4ea4-a8eb-44d13b5b1bdc\") " pod="openshift-ingress/payload-processing-e2e-api-b-6c9de5-65c4494864-c5fmz" Oct 30 09:20:12.103950 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:12.103917 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6c9de5-6fbf95ccff-sc8sg" Oct 30 09:20:12.164405 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:12.160647 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6c9de5-65c4494864-c5fmz" Oct 30 09:20:12.472519 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:12.472494 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6c9de5-6fbf95ccff-sc8sg"] Oct 30 09:20:12.517835 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:12.517628 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6c9de5-65c4494864-c5fmz"] Oct 30 09:20:12.583951 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:12.583901 4562 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-fh6qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 09:20:12.584100 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:12.583951 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fh6qf" podUID="beaa426c-dd69-4ab1-8776-c2512f0ff19d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 09:20:13.295556 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:13.295451 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6c9de5-65c4494864-c5fmz" event={"ID":"6091ee8a-f7ce-4ea4-a8eb-44d13b5b1bdc","Type":"ContainerStarted","Data":"cf052a94d8c78e6dcb14155e26d445dc551e641d42fe7a1198566f2d8b794b4e"} Oct 30 09:20:13.296091 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:13.296066 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6c9de5-65c4494864-c5fmz" event={"ID":"6091ee8a-f7ce-4ea4-a8eb-44d13b5b1bdc","Type":"ContainerStarted","Data":"530b50c4d904fb18a174dff9802629ef5faafca3018d7ebbfd5a7344550de690"} Oct 30 09:20:13.301545 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:13.297174 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-6c9de5-65c4494864-c5fmz" Oct 30 09:20:13.313269 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:13.311074 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6c9de5-6fbf95ccff-sc8sg" event={"ID":"096665bc-2bb1-46b5-8561-1371424b699d","Type":"ContainerStarted","Data":"0a60717dbb3a6836bdeb923055daff61451b92706799b2cacaf064e79b75784f"} Oct 30 09:20:13.313269 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:13.311132 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6c9de5-6fbf95ccff-sc8sg" event={"ID":"096665bc-2bb1-46b5-8561-1371424b699d","Type":"ContainerStarted","Data":"ee6f584715d83b23d67021d2ea9e3b182d1011747a4f96cd686e3b600fb4f5ec"} Oct 30 09:20:13.313269 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:13.312016 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6c9de5-6fbf95ccff-sc8sg" Oct 30 09:20:13.356698 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:13.350946 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-6c9de5-65c4494864-c5fmz" podStartSLOduration=2.350923134 podStartE2EDuration="2.350923134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:20:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:20:13.339013723 +0000 UTC m=+1573.600574713" watchObservedRunningTime="2026-10-30 09:20:13.350923134 +0000 UTC m=+1573.612484124" Oct 30 09:20:13.585118 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:13.585021 4562 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-fh6qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 09:20:13.585424 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:13.585356 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fh6qf" podUID="beaa426c-dd69-4ab1-8776-c2512f0ff19d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 09:20:14.588004 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:14.587964 4562 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-fh6qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 09:20:14.588568 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:14.588025 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fh6qf" podUID="beaa426c-dd69-4ab1-8776-c2512f0ff19d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 09:20:16.365513 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:16.365454 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fh6qf" Oct 30 09:20:16.366214 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:16.366186 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fh6qf" Oct 30 09:20:16.367759 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:16.367718 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fh6qf" Oct 30 09:20:16.475570 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:16.475513 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6c9de5-6fbf95ccff-sc8sg" podStartSLOduration=5.475499412 podStartE2EDuration="5.475499412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:20:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:20:13.395665508 +0000 UTC m=+1573.657226500" watchObservedRunningTime="2026-10-30 09:20:16.475499412 +0000 UTC m=+1576.737060399" Oct 30 09:20:19.460096 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:19.460053 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8d1106-7447558595-2fhm8"] Oct 30 09:20:19.465854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:19.465832 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8d1106-7447558595-2fhm8" Oct 30 09:20:19.487338 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:19.487308 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8d1106-7447558595-2fhm8"] Oct 30 09:20:19.494988 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:19.494962 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-8d1106\"" Oct 30 09:20:19.495202 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:19.495024 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-8d1106-dockercfg-n7h6z\"" Oct 30 09:20:19.594412 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:19.572991 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8b544fe-02d1-4d36-8122-ec15d90f14c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8d1106-7447558595-2fhm8\" (UID: \"d8b544fe-02d1-4d36-8122-ec15d90f14c7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8d1106-7447558595-2fhm8" Oct 30 09:20:19.595061 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:19.595018 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9knt\" (UniqueName: \"kubernetes.io/projected/d8b544fe-02d1-4d36-8122-ec15d90f14c7-kube-api-access-z9knt\") pod \"payload-pre-processing-e2e-api-b-8d1106-7447558595-2fhm8\" (UID: \"d8b544fe-02d1-4d36-8122-ec15d90f14c7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8d1106-7447558595-2fhm8" Oct 30 09:20:19.660768 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:19.660720 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8d1106-78c5b8b468-zkwsh"] Oct 30 09:20:19.700404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:19.696338 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8d1106-78c5b8b468-zkwsh"] Oct 30 09:20:19.700404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:19.696493 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8d1106-78c5b8b468-zkwsh" Oct 30 09:20:19.700787 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:19.700555 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8b544fe-02d1-4d36-8122-ec15d90f14c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8d1106-7447558595-2fhm8\" (UID: \"d8b544fe-02d1-4d36-8122-ec15d90f14c7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8d1106-7447558595-2fhm8" Oct 30 09:20:19.700787 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:19.700733 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9knt\" (UniqueName: \"kubernetes.io/projected/d8b544fe-02d1-4d36-8122-ec15d90f14c7-kube-api-access-z9knt\") pod \"payload-pre-processing-e2e-api-b-8d1106-7447558595-2fhm8\" (UID: \"d8b544fe-02d1-4d36-8122-ec15d90f14c7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8d1106-7447558595-2fhm8" Oct 30 09:20:19.702131 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:19.702093 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8b544fe-02d1-4d36-8122-ec15d90f14c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8d1106-7447558595-2fhm8\" (UID: \"d8b544fe-02d1-4d36-8122-ec15d90f14c7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8d1106-7447558595-2fhm8" Oct 30 09:20:19.734092 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:19.734026 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9knt\" (UniqueName: \"kubernetes.io/projected/d8b544fe-02d1-4d36-8122-ec15d90f14c7-kube-api-access-z9knt\") pod \"payload-pre-processing-e2e-api-b-8d1106-7447558595-2fhm8\" (UID: \"d8b544fe-02d1-4d36-8122-ec15d90f14c7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8d1106-7447558595-2fhm8" Oct 30 09:20:19.792060 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:19.792010 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8d1106-7447558595-2fhm8" Oct 30 09:20:19.828707 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:19.828643 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzn5j\" (UniqueName: \"kubernetes.io/projected/57af6030-ca4c-48ff-b271-2917affcede5-kube-api-access-dzn5j\") pod \"payload-processing-e2e-api-b-8d1106-78c5b8b468-zkwsh\" (UID: \"57af6030-ca4c-48ff-b271-2917affcede5\") " pod="openshift-ingress/payload-processing-e2e-api-b-8d1106-78c5b8b468-zkwsh" Oct 30 09:20:19.829078 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:19.829039 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57af6030-ca4c-48ff-b271-2917affcede5-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8d1106-78c5b8b468-zkwsh\" (UID: \"57af6030-ca4c-48ff-b271-2917affcede5\") " pod="openshift-ingress/payload-processing-e2e-api-b-8d1106-78c5b8b468-zkwsh" Oct 30 09:20:19.932333 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:19.932302 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57af6030-ca4c-48ff-b271-2917affcede5-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8d1106-78c5b8b468-zkwsh\" (UID: \"57af6030-ca4c-48ff-b271-2917affcede5\") " pod="openshift-ingress/payload-processing-e2e-api-b-8d1106-78c5b8b468-zkwsh" Oct 30 09:20:19.932510 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:19.932448 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzn5j\" (UniqueName: \"kubernetes.io/projected/57af6030-ca4c-48ff-b271-2917affcede5-kube-api-access-dzn5j\") pod \"payload-processing-e2e-api-b-8d1106-78c5b8b468-zkwsh\" (UID: \"57af6030-ca4c-48ff-b271-2917affcede5\") " pod="openshift-ingress/payload-processing-e2e-api-b-8d1106-78c5b8b468-zkwsh" Oct 30 09:20:19.933157 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:19.933137 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57af6030-ca4c-48ff-b271-2917affcede5-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8d1106-78c5b8b468-zkwsh\" (UID: \"57af6030-ca4c-48ff-b271-2917affcede5\") " pod="openshift-ingress/payload-processing-e2e-api-b-8d1106-78c5b8b468-zkwsh" Oct 30 09:20:19.953429 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:19.953230 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzn5j\" (UniqueName: \"kubernetes.io/projected/57af6030-ca4c-48ff-b271-2917affcede5-kube-api-access-dzn5j\") pod \"payload-processing-e2e-api-b-8d1106-78c5b8b468-zkwsh\" (UID: \"57af6030-ca4c-48ff-b271-2917affcede5\") " pod="openshift-ingress/payload-processing-e2e-api-b-8d1106-78c5b8b468-zkwsh" Oct 30 09:20:20.134647 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:20.134554 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8d1106-78c5b8b468-zkwsh" Oct 30 09:20:20.161619 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:20:20.161587 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8b544fe_02d1_4d36_8122_ec15d90f14c7.slice/crio-24bf2214100bd9c165dc75fc5bbe2a14b8e4d6c0dc1df8d84864a49f5ad833ad WatchSource:0}: Error finding container 24bf2214100bd9c165dc75fc5bbe2a14b8e4d6c0dc1df8d84864a49f5ad833ad: Status 404 returned error can't find the container with id 24bf2214100bd9c165dc75fc5bbe2a14b8e4d6c0dc1df8d84864a49f5ad833ad Oct 30 09:20:20.165264 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:20.164861 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8d1106-7447558595-2fhm8"] Oct 30 09:20:20.361877 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:20.361842 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8d1106-7447558595-2fhm8" event={"ID":"d8b544fe-02d1-4d36-8122-ec15d90f14c7","Type":"ContainerStarted","Data":"f5488470922a70905cc8c860355ddd0dd099a8bd4966dc507a92d84395dc827b"} Oct 30 09:20:20.362065 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:20.361896 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8d1106-7447558595-2fhm8" event={"ID":"d8b544fe-02d1-4d36-8122-ec15d90f14c7","Type":"ContainerStarted","Data":"24bf2214100bd9c165dc75fc5bbe2a14b8e4d6c0dc1df8d84864a49f5ad833ad"} Oct 30 09:20:20.362065 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:20.362034 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8d1106-7447558595-2fhm8" Oct 30 09:20:20.413499 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:20.413294 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8d1106-7447558595-2fhm8" podStartSLOduration=1.413277629 podStartE2EDuration="1.413277629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:20:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:20:20.408651623 +0000 UTC m=+1580.670212612" watchObservedRunningTime="2026-10-30 09:20:20.413277629 +0000 UTC m=+1580.674838618" Oct 30 09:20:20.502298 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:20.502242 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8d1106-78c5b8b468-zkwsh"] Oct 30 09:20:21.317042 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:21.317004 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2pgpc"] Oct 30 09:20:21.321537 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:21.321509 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2pgpc" Oct 30 09:20:21.344425 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:21.344393 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2pgpc"] Oct 30 09:20:21.391482 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:21.390444 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8d1106-78c5b8b468-zkwsh" event={"ID":"57af6030-ca4c-48ff-b271-2917affcede5","Type":"ContainerStarted","Data":"11ca51ee8761310678ad15961bc9f2a3036a98f94920ea2cb762c44bf3d3e5a4"} Oct 30 09:20:21.391482 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:21.390548 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8d1106-78c5b8b468-zkwsh" event={"ID":"57af6030-ca4c-48ff-b271-2917affcede5","Type":"ContainerStarted","Data":"7a738dcd8b9fd8d780625f5014dda34988c9e884ac4fb8221fafc3ef8d842ec1"} Oct 30 09:20:21.391482 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:21.390783 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-8d1106-78c5b8b468-zkwsh" Oct 30 09:20:21.432496 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:21.432439 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-8d1106-78c5b8b468-zkwsh" podStartSLOduration=2.432425528 podStartE2EDuration="2.432425528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:20:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:20:21.42990212 +0000 UTC m=+1581.691463109" watchObservedRunningTime="2026-10-30 09:20:21.432425528 +0000 UTC m=+1581.693986515" Oct 30 09:20:21.495728 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:21.495692 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12bbbb56-5c34-4297-8ab0-2d25958adcf5-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-2pgpc\" (UID: \"12bbbb56-5c34-4297-8ab0-2d25958adcf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2pgpc" Oct 30 09:20:21.495941 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:21.495914 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12bbbb56-5c34-4297-8ab0-2d25958adcf5-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-2pgpc\" (UID: \"12bbbb56-5c34-4297-8ab0-2d25958adcf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2pgpc" Oct 30 09:20:21.496079 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:21.496061 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12bbbb56-5c34-4297-8ab0-2d25958adcf5-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-2pgpc\" (UID: \"12bbbb56-5c34-4297-8ab0-2d25958adcf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2pgpc" Oct 30 09:20:21.496146 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:21.496104 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqv82\" (UniqueName: \"kubernetes.io/projected/12bbbb56-5c34-4297-8ab0-2d25958adcf5-kube-api-access-vqv82\") pod \"maas-default-gateway-openshift-default-77f556d647-2pgpc\" (UID: \"12bbbb56-5c34-4297-8ab0-2d25958adcf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2pgpc" Oct 30 09:20:21.496146 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:21.496132 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12bbbb56-5c34-4297-8ab0-2d25958adcf5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-2pgpc\" (UID: \"12bbbb56-5c34-4297-8ab0-2d25958adcf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2pgpc" Oct 30 09:20:21.496235 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:21.496216 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12bbbb56-5c34-4297-8ab0-2d25958adcf5-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-2pgpc\" (UID: \"12bbbb56-5c34-4297-8ab0-2d25958adcf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2pgpc" Oct 30 09:20:21.496392 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:21.496363 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12bbbb56-5c34-4297-8ab0-2d25958adcf5-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-2pgpc\" (UID: \"12bbbb56-5c34-4297-8ab0-2d25958adcf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2pgpc" Oct 30 09:20:21.496449 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:21.496422 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12bbbb56-5c34-4297-8ab0-2d25958adcf5-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-2pgpc\" (UID: \"12bbbb56-5c34-4297-8ab0-2d25958adcf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2pgpc" Oct 30 09:20:21.496486 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:21.496474 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12bbbb56-5c34-4297-8ab0-2d25958adcf5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-2pgpc\" (UID: \"12bbbb56-5c34-4297-8ab0-2d25958adcf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2pgpc" Oct 30 09:20:21.606461 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:21.603415 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12bbbb56-5c34-4297-8ab0-2d25958adcf5-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-2pgpc\" (UID: \"12bbbb56-5c34-4297-8ab0-2d25958adcf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2pgpc" Oct 30 09:20:21.606461 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:21.603683 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12bbbb56-5c34-4297-8ab0-2d25958adcf5-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-2pgpc\" (UID: \"12bbbb56-5c34-4297-8ab0-2d25958adcf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2pgpc" Oct 30 09:20:21.606461 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:21.603717 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqv82\" (UniqueName: \"kubernetes.io/projected/12bbbb56-5c34-4297-8ab0-2d25958adcf5-kube-api-access-vqv82\") pod \"maas-default-gateway-openshift-default-77f556d647-2pgpc\" (UID: \"12bbbb56-5c34-4297-8ab0-2d25958adcf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2pgpc" Oct 30 09:20:21.606461 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:21.603746 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12bbbb56-5c34-4297-8ab0-2d25958adcf5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-2pgpc\" (UID: \"12bbbb56-5c34-4297-8ab0-2d25958adcf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2pgpc" Oct 30 09:20:21.606461 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:21.603793 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12bbbb56-5c34-4297-8ab0-2d25958adcf5-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-2pgpc\" (UID: \"12bbbb56-5c34-4297-8ab0-2d25958adcf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2pgpc" Oct 30 09:20:21.606461 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:21.603874 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12bbbb56-5c34-4297-8ab0-2d25958adcf5-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-2pgpc\" (UID: \"12bbbb56-5c34-4297-8ab0-2d25958adcf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2pgpc" Oct 30 09:20:21.606461 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:21.603913 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12bbbb56-5c34-4297-8ab0-2d25958adcf5-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-2pgpc\" (UID: \"12bbbb56-5c34-4297-8ab0-2d25958adcf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2pgpc" Oct 30 09:20:21.606461 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:21.603938 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12bbbb56-5c34-4297-8ab0-2d25958adcf5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-2pgpc\" (UID: \"12bbbb56-5c34-4297-8ab0-2d25958adcf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2pgpc" Oct 30 09:20:21.606461 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:21.604027 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12bbbb56-5c34-4297-8ab0-2d25958adcf5-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-2pgpc\" (UID: \"12bbbb56-5c34-4297-8ab0-2d25958adcf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2pgpc" Oct 30 09:20:21.606461 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:21.604571 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12bbbb56-5c34-4297-8ab0-2d25958adcf5-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-2pgpc\" (UID: \"12bbbb56-5c34-4297-8ab0-2d25958adcf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2pgpc" Oct 30 09:20:21.606461 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:21.604836 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12bbbb56-5c34-4297-8ab0-2d25958adcf5-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-2pgpc\" (UID: \"12bbbb56-5c34-4297-8ab0-2d25958adcf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2pgpc" Oct 30 09:20:21.606461 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:21.606004 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12bbbb56-5c34-4297-8ab0-2d25958adcf5-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-2pgpc\" (UID: \"12bbbb56-5c34-4297-8ab0-2d25958adcf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2pgpc" Oct 30 09:20:21.606461 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:21.606240 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12bbbb56-5c34-4297-8ab0-2d25958adcf5-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-2pgpc\" (UID: \"12bbbb56-5c34-4297-8ab0-2d25958adcf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2pgpc" Oct 30 09:20:21.613351 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:21.613316 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12bbbb56-5c34-4297-8ab0-2d25958adcf5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-2pgpc\" (UID: \"12bbbb56-5c34-4297-8ab0-2d25958adcf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2pgpc" Oct 30 09:20:21.615744 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:21.615715 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12bbbb56-5c34-4297-8ab0-2d25958adcf5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-2pgpc\" (UID: \"12bbbb56-5c34-4297-8ab0-2d25958adcf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2pgpc" Oct 30 09:20:21.620953 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:21.620907 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12bbbb56-5c34-4297-8ab0-2d25958adcf5-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-2pgpc\" (UID: \"12bbbb56-5c34-4297-8ab0-2d25958adcf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2pgpc" Oct 30 09:20:21.634606 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:21.627075 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqv82\" (UniqueName: \"kubernetes.io/projected/12bbbb56-5c34-4297-8ab0-2d25958adcf5-kube-api-access-vqv82\") pod \"maas-default-gateway-openshift-default-77f556d647-2pgpc\" (UID: \"12bbbb56-5c34-4297-8ab0-2d25958adcf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2pgpc" Oct 30 09:20:21.639472 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:21.638857 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12bbbb56-5c34-4297-8ab0-2d25958adcf5-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-2pgpc\" (UID: \"12bbbb56-5c34-4297-8ab0-2d25958adcf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2pgpc" Oct 30 09:20:21.936318 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:21.936238 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2pgpc" Oct 30 09:20:22.258437 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:20:22.258365 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12bbbb56_5c34_4297_8ab0_2d25958adcf5.slice/crio-503d158f6f6fff14178423f32e63f1fdc3cc49794df8eaa8fc6eed5b27ac03ef WatchSource:0}: Error finding container 503d158f6f6fff14178423f32e63f1fdc3cc49794df8eaa8fc6eed5b27ac03ef: Status 404 returned error can't find the container with id 503d158f6f6fff14178423f32e63f1fdc3cc49794df8eaa8fc6eed5b27ac03ef Oct 30 09:20:22.260458 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:22.260433 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2pgpc"] Oct 30 09:20:22.261348 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:22.261317 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:20:22.261451 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:22.261434 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:20:22.261501 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:22.261492 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:20:22.397857 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:22.397813 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2pgpc" event={"ID":"12bbbb56-5c34-4297-8ab0-2d25958adcf5","Type":"ContainerStarted","Data":"d2637bf8160a36f897b1c8a96ae391f7bd86a920bdea62aa2565264dcae218ea"} Oct 30 09:20:22.398041 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:22.397879 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2pgpc" event={"ID":"12bbbb56-5c34-4297-8ab0-2d25958adcf5","Type":"ContainerStarted","Data":"503d158f6f6fff14178423f32e63f1fdc3cc49794df8eaa8fc6eed5b27ac03ef"} Oct 30 09:20:22.442555 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:22.442505 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2pgpc" podStartSLOduration=1.442492586 podStartE2EDuration="1.442492586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:20:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:20:22.441311797 +0000 UTC m=+1582.702872785" watchObservedRunningTime="2026-10-30 09:20:22.442492586 +0000 UTC m=+1582.704053573" Oct 30 09:20:22.936652 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:22.936620 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2pgpc" Oct 30 09:20:23.954558 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:23.954517 4562 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-2pgpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 30 09:20:23.955166 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:23.955131 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2pgpc" podUID="12bbbb56-5c34-4297-8ab0-2d25958adcf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 30 09:20:24.937449 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:24.937419 4562 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-2pgpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 30 09:20:24.937629 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:24.937467 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2pgpc" podUID="12bbbb56-5c34-4297-8ab0-2d25958adcf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 30 09:20:25.269520 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:25.269486 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6c9de5-895f9fb7-48dpg"] Oct 30 09:20:25.274031 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:25.274003 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6c9de5-895f9fb7-48dpg" Oct 30 09:20:25.288895 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:25.288866 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3764eee1-fcd1-4d69-8654-0d71e22845e4-openshift-service-ca\") pod \"maas-api-e2e-api-b-6c9de5-895f9fb7-48dpg\" (UID: \"3764eee1-fcd1-4d69-8654-0d71e22845e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6c9de5-895f9fb7-48dpg" Oct 30 09:20:25.289025 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:25.288966 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3764eee1-fcd1-4d69-8654-0d71e22845e4-metrics-certs\") pod \"maas-api-e2e-api-b-6c9de5-895f9fb7-48dpg\" (UID: \"3764eee1-fcd1-4d69-8654-0d71e22845e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6c9de5-895f9fb7-48dpg" Oct 30 09:20:25.289025 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:25.288986 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7rtp\" (UniqueName: \"kubernetes.io/projected/3764eee1-fcd1-4d69-8654-0d71e22845e4-kube-api-access-r7rtp\") pod \"maas-api-e2e-api-b-6c9de5-895f9fb7-48dpg\" (UID: \"3764eee1-fcd1-4d69-8654-0d71e22845e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6c9de5-895f9fb7-48dpg" Oct 30 09:20:25.289144 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:25.289079 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3764eee1-fcd1-4d69-8654-0d71e22845e4-maas-api-tls\") pod \"maas-api-e2e-api-b-6c9de5-895f9fb7-48dpg\" (UID: \"3764eee1-fcd1-4d69-8654-0d71e22845e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6c9de5-895f9fb7-48dpg" Oct 30 09:20:25.291540 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:25.291517 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6c9de5-895f9fb7-48dpg"] Oct 30 09:20:25.304162 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:25.301040 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-6c9de5-serving-cert\"" Oct 30 09:20:25.331286 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:25.331258 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-6c9de5-65c4494864-c5fmz" Oct 30 09:20:25.331702 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:25.331679 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6c9de5-6fbf95ccff-sc8sg" Oct 30 09:20:25.390896 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:25.390857 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3764eee1-fcd1-4d69-8654-0d71e22845e4-openshift-service-ca\") pod \"maas-api-e2e-api-b-6c9de5-895f9fb7-48dpg\" (UID: \"3764eee1-fcd1-4d69-8654-0d71e22845e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6c9de5-895f9fb7-48dpg" Oct 30 09:20:25.391063 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:25.391021 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3764eee1-fcd1-4d69-8654-0d71e22845e4-metrics-certs\") pod \"maas-api-e2e-api-b-6c9de5-895f9fb7-48dpg\" (UID: \"3764eee1-fcd1-4d69-8654-0d71e22845e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6c9de5-895f9fb7-48dpg" Oct 30 09:20:25.391063 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:25.391049 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7rtp\" (UniqueName: \"kubernetes.io/projected/3764eee1-fcd1-4d69-8654-0d71e22845e4-kube-api-access-r7rtp\") pod \"maas-api-e2e-api-b-6c9de5-895f9fb7-48dpg\" (UID: \"3764eee1-fcd1-4d69-8654-0d71e22845e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6c9de5-895f9fb7-48dpg" Oct 30 09:20:25.391258 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:25.391240 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3764eee1-fcd1-4d69-8654-0d71e22845e4-maas-api-tls\") pod \"maas-api-e2e-api-b-6c9de5-895f9fb7-48dpg\" (UID: \"3764eee1-fcd1-4d69-8654-0d71e22845e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6c9de5-895f9fb7-48dpg" Oct 30 09:20:25.392213 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:25.391803 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3764eee1-fcd1-4d69-8654-0d71e22845e4-openshift-service-ca\") pod \"maas-api-e2e-api-b-6c9de5-895f9fb7-48dpg\" (UID: \"3764eee1-fcd1-4d69-8654-0d71e22845e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6c9de5-895f9fb7-48dpg" Oct 30 09:20:25.393895 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:25.393869 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3764eee1-fcd1-4d69-8654-0d71e22845e4-maas-api-tls\") pod \"maas-api-e2e-api-b-6c9de5-895f9fb7-48dpg\" (UID: \"3764eee1-fcd1-4d69-8654-0d71e22845e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6c9de5-895f9fb7-48dpg" Oct 30 09:20:25.394245 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:25.394223 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3764eee1-fcd1-4d69-8654-0d71e22845e4-metrics-certs\") pod \"maas-api-e2e-api-b-6c9de5-895f9fb7-48dpg\" (UID: \"3764eee1-fcd1-4d69-8654-0d71e22845e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6c9de5-895f9fb7-48dpg" Oct 30 09:20:25.417349 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:25.412969 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7rtp\" (UniqueName: \"kubernetes.io/projected/3764eee1-fcd1-4d69-8654-0d71e22845e4-kube-api-access-r7rtp\") pod \"maas-api-e2e-api-b-6c9de5-895f9fb7-48dpg\" (UID: \"3764eee1-fcd1-4d69-8654-0d71e22845e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6c9de5-895f9fb7-48dpg" Oct 30 09:20:25.600458 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:25.598431 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6c9de5-895f9fb7-48dpg" Oct 30 09:20:25.936592 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:25.936507 4562 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-2pgpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 30 09:20:25.936592 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:25.936560 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2pgpc" podUID="12bbbb56-5c34-4297-8ab0-2d25958adcf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 30 09:20:26.022440 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:20:26.022402 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3764eee1_fcd1_4d69_8654_0d71e22845e4.slice/crio-29f417ae9ce00799b1a1b87b1bd54365dacb8f95555d44382762f802c57a911a WatchSource:0}: Error finding container 29f417ae9ce00799b1a1b87b1bd54365dacb8f95555d44382762f802c57a911a: Status 404 returned error can't find the container with id 29f417ae9ce00799b1a1b87b1bd54365dacb8f95555d44382762f802c57a911a Oct 30 09:20:26.024728 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:26.024703 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6c9de5-895f9fb7-48dpg"] Oct 30 09:20:26.427581 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:26.427553 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6c9de5-895f9fb7-48dpg" event={"ID":"3764eee1-fcd1-4d69-8654-0d71e22845e4","Type":"ContainerStarted","Data":"29f417ae9ce00799b1a1b87b1bd54365dacb8f95555d44382762f802c57a911a"} Oct 30 09:20:26.937072 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:26.937043 4562 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-2pgpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 30 09:20:26.937241 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:26.937089 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2pgpc" podUID="12bbbb56-5c34-4297-8ab0-2d25958adcf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 30 09:20:27.482438 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:27.482200 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6c9de5-895f9fb7-48dpg" event={"ID":"3764eee1-fcd1-4d69-8654-0d71e22845e4","Type":"ContainerStarted","Data":"fb959c2c6aabc1998748458c5e000ea1eca8d5a6f28b76ba74398a7de904706a"} Oct 30 09:20:27.497492 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:27.487275 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6c9de5-895f9fb7-48dpg" Oct 30 09:20:27.554491 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:27.531992 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6c9de5-895f9fb7-48dpg" podStartSLOduration=2.183521242 podStartE2EDuration="2.531973227s" totalImagesPullingTime="348.451985ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:20:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:20:27.528465537 +0000 UTC m=+1587.790026573" watchObservedRunningTime="2026-10-30 09:20:27.531973227 +0000 UTC m=+1587.793534216" Oct 30 09:20:27.937063 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:27.936982 4562 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-2pgpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 30 09:20:27.937063 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:27.937034 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2pgpc" podUID="12bbbb56-5c34-4297-8ab0-2d25958adcf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 30 09:20:28.596658 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:20:28.596625 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:20:28.936865 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:28.936750 4562 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-2pgpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 30 09:20:28.936865 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:28.936835 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2pgpc" podUID="12bbbb56-5c34-4297-8ab0-2d25958adcf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 30 09:20:29.937353 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:29.937321 4562 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-2pgpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 30 09:20:29.937851 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:29.937374 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2pgpc" podUID="12bbbb56-5c34-4297-8ab0-2d25958adcf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 30 09:20:31.357676 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:31.357645 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2pgpc" Oct 30 09:20:31.358172 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:31.358142 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2pgpc" Oct 30 09:20:31.358874 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:31.358855 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2pgpc" Oct 30 09:20:31.394073 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:31.394050 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8d1106-7447558595-2fhm8" Oct 30 09:20:32.411633 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:32.411600 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-8d1106-78c5b8b468-zkwsh" Oct 30 09:20:34.530898 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:34.530869 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6c9de5-895f9fb7-48dpg" Oct 30 09:20:58.596670 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:20:58.596588 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:20:58.867997 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:58.867960 4562 generic.go:356] "Generic (PLEG): container finished" podID="357b42b1-a9e1-423c-96b4-3bb155fed864" containerID="774209cf60805d8c3fed9f3186e0ca6185e67bba1e26e9f85705c968d4c6b23c" exitCode=0 Oct 30 09:20:58.868190 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:58.868032 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn" event={"ID":"357b42b1-a9e1-423c-96b4-3bb155fed864","Type":"ContainerDied","Data":"774209cf60805d8c3fed9f3186e0ca6185e67bba1e26e9f85705c968d4c6b23c"} Oct 30 09:20:58.868461 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:58.868445 4562 scope.go:122] "RemoveContainer" containerID="774209cf60805d8c3fed9f3186e0ca6185e67bba1e26e9f85705c968d4c6b23c" Oct 30 09:20:58.869851 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:58.869822 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:20:58.869960 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:58.869900 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:20:58.869960 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:58.869930 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:20:59.879178 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:20:59.879138 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn" event={"ID":"357b42b1-a9e1-423c-96b4-3bb155fed864","Type":"ContainerStarted","Data":"0afd5c4c97f378b7d68b9d98a65566cb37d1256b31cc99d51875c95f6ff2dcef"} Oct 30 09:21:00.015914 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:00.015881 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn" Oct 30 09:21:00.016114 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:00.016050 4562 patch_prober.go:29] interesting pod/e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 30 09:21:00.016114 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:00.016099 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn" podUID="357b42b1-a9e1-423c-96b4-3bb155fed864" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 30 09:21:01.019797 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:01.019757 4562 patch_prober.go:29] interesting pod/e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 30 09:21:01.020428 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:01.020372 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn" podUID="357b42b1-a9e1-423c-96b4-3bb155fed864" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 30 09:21:01.596611 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:01.596572 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-648892-578444d976-xxlck"] Oct 30 09:21:01.629003 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:01.628972 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-648892-f58f78dd7-xf8xh"] Oct 30 09:21:02.015694 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:02.015657 4562 patch_prober.go:29] interesting pod/e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 30 09:21:02.017923 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:02.017889 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn" podUID="357b42b1-a9e1-423c-96b4-3bb155fed864" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 30 09:21:03.016013 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:03.015981 4562 patch_prober.go:29] interesting pod/e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 30 09:21:03.016496 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:03.016145 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn" podUID="357b42b1-a9e1-423c-96b4-3bb155fed864" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 30 09:21:04.015772 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:04.015738 4562 patch_prober.go:29] interesting pod/e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 30 09:21:04.015939 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:04.015791 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn" podUID="357b42b1-a9e1-423c-96b4-3bb155fed864" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 30 09:21:05.015676 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:05.015640 4562 patch_prober.go:29] interesting pod/e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 30 09:21:05.016078 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:05.015702 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn" podUID="357b42b1-a9e1-423c-96b4-3bb155fed864" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 30 09:21:06.020747 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:06.020708 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn" Oct 30 09:21:06.021209 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:06.021001 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn" Oct 30 09:21:06.021785 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:06.021765 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn" Oct 30 09:21:06.597486 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:06.597442 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-648892-578444d976-xxlck" podUID="56d7083b-d7cf-46d7-9469-6bcf9ef1d02a" containerName="payload-processing" containerID="cri-o://efbc0162f569d63bd4c10db2c9a92844ffceccb8e45d9ff8d23599fdfcd03107" gracePeriod=25 Oct 30 09:21:06.629988 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:06.629958 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-648892-f58f78dd7-xf8xh" podUID="174415ce-8818-44e6-b26e-f95c8c4b3521" containerName="payload-pre-processing" containerID="cri-o://875fce28a8a21c2b81b0eb833d25db982c8f0b1764bffe31713682385f132060" gracePeriod=25 Oct 30 09:21:06.911942 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:06.911915 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-648892-578444d976-xxlck" Oct 30 09:21:06.942276 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:06.942208 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-648892-f58f78dd7-xf8xh" Oct 30 09:21:07.064931 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:07.064894 4562 generic.go:356] "Generic (PLEG): container finished" podID="174415ce-8818-44e6-b26e-f95c8c4b3521" containerID="875fce28a8a21c2b81b0eb833d25db982c8f0b1764bffe31713682385f132060" exitCode=0 Oct 30 09:21:07.065403 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:07.064954 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-648892-f58f78dd7-xf8xh" Oct 30 09:21:07.065403 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:07.064975 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-648892-f58f78dd7-xf8xh" event={"ID":"174415ce-8818-44e6-b26e-f95c8c4b3521","Type":"ContainerDied","Data":"875fce28a8a21c2b81b0eb833d25db982c8f0b1764bffe31713682385f132060"} Oct 30 09:21:07.065403 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:07.065035 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-648892-f58f78dd7-xf8xh" event={"ID":"174415ce-8818-44e6-b26e-f95c8c4b3521","Type":"ContainerDied","Data":"da8778cfa990e3de51ae2990f744d8f88eb162cae626fd9794f21a102452ff07"} Oct 30 09:21:07.065403 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:07.065059 4562 scope.go:122] "RemoveContainer" containerID="875fce28a8a21c2b81b0eb833d25db982c8f0b1764bffe31713682385f132060" Oct 30 09:21:07.066822 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:07.066800 4562 generic.go:356] "Generic (PLEG): container finished" podID="56d7083b-d7cf-46d7-9469-6bcf9ef1d02a" containerID="efbc0162f569d63bd4c10db2c9a92844ffceccb8e45d9ff8d23599fdfcd03107" exitCode=0 Oct 30 09:21:07.066943 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:07.066858 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-648892-578444d976-xxlck" Oct 30 09:21:07.066943 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:07.066889 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-648892-578444d976-xxlck" event={"ID":"56d7083b-d7cf-46d7-9469-6bcf9ef1d02a","Type":"ContainerDied","Data":"efbc0162f569d63bd4c10db2c9a92844ffceccb8e45d9ff8d23599fdfcd03107"} Oct 30 09:21:07.067038 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:07.066933 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-648892-578444d976-xxlck" event={"ID":"56d7083b-d7cf-46d7-9469-6bcf9ef1d02a","Type":"ContainerDied","Data":"fd3bd20f21f8242df2d023614deaf1bd769936938ae1634e61065c12848b2c22"} Oct 30 09:21:07.077910 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:07.077888 4562 scope.go:122] "RemoveContainer" containerID="875fce28a8a21c2b81b0eb833d25db982c8f0b1764bffe31713682385f132060" Oct 30 09:21:07.078244 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:21:07.078216 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"875fce28a8a21c2b81b0eb833d25db982c8f0b1764bffe31713682385f132060\": container with ID starting with 875fce28a8a21c2b81b0eb833d25db982c8f0b1764bffe31713682385f132060 not found: ID does not exist" containerID="875fce28a8a21c2b81b0eb833d25db982c8f0b1764bffe31713682385f132060" Oct 30 09:21:07.078359 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:07.078252 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"875fce28a8a21c2b81b0eb833d25db982c8f0b1764bffe31713682385f132060"} err="failed to get container status \"875fce28a8a21c2b81b0eb833d25db982c8f0b1764bffe31713682385f132060\": rpc error: code = NotFound desc = could not find container \"875fce28a8a21c2b81b0eb833d25db982c8f0b1764bffe31713682385f132060\": container with ID starting with 875fce28a8a21c2b81b0eb833d25db982c8f0b1764bffe31713682385f132060 not found: ID does not exist" Oct 30 09:21:07.078359 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:07.078274 4562 scope.go:122] "RemoveContainer" containerID="efbc0162f569d63bd4c10db2c9a92844ffceccb8e45d9ff8d23599fdfcd03107" Oct 30 09:21:07.091202 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:07.091183 4562 scope.go:122] "RemoveContainer" containerID="efbc0162f569d63bd4c10db2c9a92844ffceccb8e45d9ff8d23599fdfcd03107" Oct 30 09:21:07.091493 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:21:07.091464 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efbc0162f569d63bd4c10db2c9a92844ffceccb8e45d9ff8d23599fdfcd03107\": container with ID starting with efbc0162f569d63bd4c10db2c9a92844ffceccb8e45d9ff8d23599fdfcd03107 not found: ID does not exist" containerID="efbc0162f569d63bd4c10db2c9a92844ffceccb8e45d9ff8d23599fdfcd03107" Oct 30 09:21:07.091602 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:07.091502 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efbc0162f569d63bd4c10db2c9a92844ffceccb8e45d9ff8d23599fdfcd03107"} err="failed to get container status \"efbc0162f569d63bd4c10db2c9a92844ffceccb8e45d9ff8d23599fdfcd03107\": rpc error: code = NotFound desc = could not find container \"efbc0162f569d63bd4c10db2c9a92844ffceccb8e45d9ff8d23599fdfcd03107\": container with ID starting with efbc0162f569d63bd4c10db2c9a92844ffceccb8e45d9ff8d23599fdfcd03107 not found: ID does not exist" Oct 30 09:21:07.105827 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:07.105771 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56d7083b-d7cf-46d7-9469-6bcf9ef1d02a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56d7083b-d7cf-46d7-9469-6bcf9ef1d02a-plugins-config-volume\") pod \"56d7083b-d7cf-46d7-9469-6bcf9ef1d02a\" (UID: \"56d7083b-d7cf-46d7-9469-6bcf9ef1d02a\") " Oct 30 09:21:07.105827 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:07.105811 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56d7083b-d7cf-46d7-9469-6bcf9ef1d02a-kube-api-access-g744h\" (UniqueName: \"kubernetes.io/projected/56d7083b-d7cf-46d7-9469-6bcf9ef1d02a-kube-api-access-g744h\") pod \"56d7083b-d7cf-46d7-9469-6bcf9ef1d02a\" (UID: \"56d7083b-d7cf-46d7-9469-6bcf9ef1d02a\") " Oct 30 09:21:07.105992 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:07.105855 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/174415ce-8818-44e6-b26e-f95c8c4b3521-kube-api-access-5ps9w\" (UniqueName: \"kubernetes.io/projected/174415ce-8818-44e6-b26e-f95c8c4b3521-kube-api-access-5ps9w\") pod \"174415ce-8818-44e6-b26e-f95c8c4b3521\" (UID: \"174415ce-8818-44e6-b26e-f95c8c4b3521\") " Oct 30 09:21:07.105992 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:07.105917 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/174415ce-8818-44e6-b26e-f95c8c4b3521-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/174415ce-8818-44e6-b26e-f95c8c4b3521-plugins-config-volume\") pod \"174415ce-8818-44e6-b26e-f95c8c4b3521\" (UID: \"174415ce-8818-44e6-b26e-f95c8c4b3521\") " Oct 30 09:21:07.106134 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:07.106113 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56d7083b-d7cf-46d7-9469-6bcf9ef1d02a-plugins-config-volume" pod "56d7083b-d7cf-46d7-9469-6bcf9ef1d02a" (UID: "56d7083b-d7cf-46d7-9469-6bcf9ef1d02a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:21:07.106450 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:07.106416 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/174415ce-8818-44e6-b26e-f95c8c4b3521-plugins-config-volume" pod "174415ce-8818-44e6-b26e-f95c8c4b3521" (UID: "174415ce-8818-44e6-b26e-f95c8c4b3521"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:21:07.106547 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:07.106511 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/174415ce-8818-44e6-b26e-f95c8c4b3521-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:21:07.106547 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:07.106524 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56d7083b-d7cf-46d7-9469-6bcf9ef1d02a-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:21:07.108738 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:07.108718 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56d7083b-d7cf-46d7-9469-6bcf9ef1d02a-kube-api-access-g744h" pod "56d7083b-d7cf-46d7-9469-6bcf9ef1d02a" (UID: "56d7083b-d7cf-46d7-9469-6bcf9ef1d02a"). InnerVolumeSpecName "kube-api-access-g744h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:21:07.109032 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:07.109013 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/174415ce-8818-44e6-b26e-f95c8c4b3521-kube-api-access-5ps9w" pod "174415ce-8818-44e6-b26e-f95c8c4b3521" (UID: "174415ce-8818-44e6-b26e-f95c8c4b3521"). InnerVolumeSpecName "kube-api-access-5ps9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:21:07.208376 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:07.208344 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g744h\" (UniqueName: \"kubernetes.io/projected/56d7083b-d7cf-46d7-9469-6bcf9ef1d02a-kube-api-access-g744h\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:21:07.208527 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:07.208377 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ps9w\" (UniqueName: \"kubernetes.io/projected/174415ce-8818-44e6-b26e-f95c8c4b3521-kube-api-access-5ps9w\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:21:07.486579 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:07.481039 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-648892-f58f78dd7-xf8xh"] Oct 30 09:21:07.507407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:07.490206 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-648892-f58f78dd7-xf8xh"] Oct 30 09:21:07.507407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:07.494376 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-648892-578444d976-xxlck"] Oct 30 09:21:07.511498 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:07.511450 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-648892-578444d976-xxlck"] Oct 30 09:21:07.876636 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:07.876596 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="174415ce-8818-44e6-b26e-f95c8c4b3521" path="/var/lib/kubelet/pods/174415ce-8818-44e6-b26e-f95c8c4b3521/volumes" Oct 30 09:21:07.877345 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:07.877322 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56d7083b-d7cf-46d7-9469-6bcf9ef1d02a" path="/var/lib/kubelet/pods/56d7083b-d7cf-46d7-9469-6bcf9ef1d02a/volumes" Oct 30 09:21:09.679790 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:09.679755 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ca0df2-7d67bcc9bb-b2brq"] Oct 30 09:21:09.714103 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:09.714075 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ca0df2-57b8bf5d59-tbxzz"] Oct 30 09:21:14.680510 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:14.680441 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-ca0df2-7d67bcc9bb-b2brq" podUID="d38fd70a-7b00-4c69-86d7-a6a58900a3bf" containerName="payload-processing" containerID="cri-o://ec15dc53d6caefe3d569d9664363e8f7d907a6a70419711b0eda8b8700a946aa" gracePeriod=25 Oct 30 09:21:14.715129 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:14.715094 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca0df2-57b8bf5d59-tbxzz" podUID="349c4a5f-88cf-473a-9463-283db5e5f586" containerName="payload-pre-processing" containerID="cri-o://a2b12d9ab1d9faa5c67642e205f2e220db100410b45311806a1fcbeff1363945" gracePeriod=25 Oct 30 09:21:14.995708 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:14.995680 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ca0df2-7d67bcc9bb-b2brq" Oct 30 09:21:15.016567 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:15.016540 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca0df2-57b8bf5d59-tbxzz" Oct 30 09:21:15.036494 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:15.036467 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/349c4a5f-88cf-473a-9463-283db5e5f586-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/349c4a5f-88cf-473a-9463-283db5e5f586-plugins-config-volume\") pod \"349c4a5f-88cf-473a-9463-283db5e5f586\" (UID: \"349c4a5f-88cf-473a-9463-283db5e5f586\") " Oct 30 09:21:15.036623 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:15.036524 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d38fd70a-7b00-4c69-86d7-a6a58900a3bf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d38fd70a-7b00-4c69-86d7-a6a58900a3bf-plugins-config-volume\") pod \"d38fd70a-7b00-4c69-86d7-a6a58900a3bf\" (UID: \"d38fd70a-7b00-4c69-86d7-a6a58900a3bf\") " Oct 30 09:21:15.036770 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:15.036750 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d38fd70a-7b00-4c69-86d7-a6a58900a3bf-kube-api-access-k6w4n\" (UniqueName: \"kubernetes.io/projected/d38fd70a-7b00-4c69-86d7-a6a58900a3bf-kube-api-access-k6w4n\") pod \"d38fd70a-7b00-4c69-86d7-a6a58900a3bf\" (UID: \"d38fd70a-7b00-4c69-86d7-a6a58900a3bf\") " Oct 30 09:21:15.036837 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:15.036818 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/349c4a5f-88cf-473a-9463-283db5e5f586-kube-api-access-27wcx\" (UniqueName: \"kubernetes.io/projected/349c4a5f-88cf-473a-9463-283db5e5f586-kube-api-access-27wcx\") pod \"349c4a5f-88cf-473a-9463-283db5e5f586\" (UID: \"349c4a5f-88cf-473a-9463-283db5e5f586\") " Oct 30 09:21:15.036916 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:15.036893 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/349c4a5f-88cf-473a-9463-283db5e5f586-plugins-config-volume" pod "349c4a5f-88cf-473a-9463-283db5e5f586" (UID: "349c4a5f-88cf-473a-9463-283db5e5f586"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:21:15.037517 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:15.037496 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/349c4a5f-88cf-473a-9463-283db5e5f586-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:21:15.037697 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:15.037675 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d38fd70a-7b00-4c69-86d7-a6a58900a3bf-plugins-config-volume" pod "d38fd70a-7b00-4c69-86d7-a6a58900a3bf" (UID: "d38fd70a-7b00-4c69-86d7-a6a58900a3bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:21:15.040162 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:15.040139 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d38fd70a-7b00-4c69-86d7-a6a58900a3bf-kube-api-access-k6w4n" pod "d38fd70a-7b00-4c69-86d7-a6a58900a3bf" (UID: "d38fd70a-7b00-4c69-86d7-a6a58900a3bf"). InnerVolumeSpecName "kube-api-access-k6w4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:21:15.040268 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:15.040186 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/349c4a5f-88cf-473a-9463-283db5e5f586-kube-api-access-27wcx" pod "349c4a5f-88cf-473a-9463-283db5e5f586" (UID: "349c4a5f-88cf-473a-9463-283db5e5f586"). InnerVolumeSpecName "kube-api-access-27wcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:21:15.139002 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:15.138973 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27wcx\" (UniqueName: \"kubernetes.io/projected/349c4a5f-88cf-473a-9463-283db5e5f586-kube-api-access-27wcx\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:21:15.139002 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:15.139000 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d38fd70a-7b00-4c69-86d7-a6a58900a3bf-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:21:15.139002 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:15.139009 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6w4n\" (UniqueName: \"kubernetes.io/projected/d38fd70a-7b00-4c69-86d7-a6a58900a3bf-kube-api-access-k6w4n\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:21:15.153492 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:15.153461 4562 generic.go:356] "Generic (PLEG): container finished" podID="d38fd70a-7b00-4c69-86d7-a6a58900a3bf" containerID="ec15dc53d6caefe3d569d9664363e8f7d907a6a70419711b0eda8b8700a946aa" exitCode=0 Oct 30 09:21:15.153629 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:15.153506 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ca0df2-7d67bcc9bb-b2brq" Oct 30 09:21:15.153629 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:15.153548 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ca0df2-7d67bcc9bb-b2brq" event={"ID":"d38fd70a-7b00-4c69-86d7-a6a58900a3bf","Type":"ContainerDied","Data":"ec15dc53d6caefe3d569d9664363e8f7d907a6a70419711b0eda8b8700a946aa"} Oct 30 09:21:15.153629 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:15.153604 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ca0df2-7d67bcc9bb-b2brq" event={"ID":"d38fd70a-7b00-4c69-86d7-a6a58900a3bf","Type":"ContainerDied","Data":"0edfa9b0032b4950ba7f093aa31f7275a744c78a485d57ef55eb19bf97504256"} Oct 30 09:21:15.153629 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:15.153625 4562 scope.go:122] "RemoveContainer" containerID="ec15dc53d6caefe3d569d9664363e8f7d907a6a70419711b0eda8b8700a946aa" Oct 30 09:21:15.155025 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:15.155000 4562 generic.go:356] "Generic (PLEG): container finished" podID="349c4a5f-88cf-473a-9463-283db5e5f586" containerID="a2b12d9ab1d9faa5c67642e205f2e220db100410b45311806a1fcbeff1363945" exitCode=0 Oct 30 09:21:15.155129 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:15.155053 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca0df2-57b8bf5d59-tbxzz" Oct 30 09:21:15.155129 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:15.155082 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca0df2-57b8bf5d59-tbxzz" event={"ID":"349c4a5f-88cf-473a-9463-283db5e5f586","Type":"ContainerDied","Data":"a2b12d9ab1d9faa5c67642e205f2e220db100410b45311806a1fcbeff1363945"} Oct 30 09:21:15.155218 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:15.155136 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca0df2-57b8bf5d59-tbxzz" event={"ID":"349c4a5f-88cf-473a-9463-283db5e5f586","Type":"ContainerDied","Data":"59fa09d89132386c799849ba6fd04407ce5da4569cdaff87322bee91d446ef13"} Oct 30 09:21:15.165255 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:15.165233 4562 scope.go:122] "RemoveContainer" containerID="ec15dc53d6caefe3d569d9664363e8f7d907a6a70419711b0eda8b8700a946aa" Oct 30 09:21:15.165564 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:21:15.165541 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec15dc53d6caefe3d569d9664363e8f7d907a6a70419711b0eda8b8700a946aa\": container with ID starting with ec15dc53d6caefe3d569d9664363e8f7d907a6a70419711b0eda8b8700a946aa not found: ID does not exist" containerID="ec15dc53d6caefe3d569d9664363e8f7d907a6a70419711b0eda8b8700a946aa" Oct 30 09:21:15.165651 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:15.165570 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec15dc53d6caefe3d569d9664363e8f7d907a6a70419711b0eda8b8700a946aa"} err="failed to get container status \"ec15dc53d6caefe3d569d9664363e8f7d907a6a70419711b0eda8b8700a946aa\": rpc error: code = NotFound desc = could not find container \"ec15dc53d6caefe3d569d9664363e8f7d907a6a70419711b0eda8b8700a946aa\": container with ID starting with ec15dc53d6caefe3d569d9664363e8f7d907a6a70419711b0eda8b8700a946aa not found: ID does not exist" Oct 30 09:21:15.165651 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:15.165587 4562 scope.go:122] "RemoveContainer" containerID="a2b12d9ab1d9faa5c67642e205f2e220db100410b45311806a1fcbeff1363945" Oct 30 09:21:15.174848 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:15.174832 4562 scope.go:122] "RemoveContainer" containerID="a2b12d9ab1d9faa5c67642e205f2e220db100410b45311806a1fcbeff1363945" Oct 30 09:21:15.175113 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:21:15.175092 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2b12d9ab1d9faa5c67642e205f2e220db100410b45311806a1fcbeff1363945\": container with ID starting with a2b12d9ab1d9faa5c67642e205f2e220db100410b45311806a1fcbeff1363945 not found: ID does not exist" containerID="a2b12d9ab1d9faa5c67642e205f2e220db100410b45311806a1fcbeff1363945" Oct 30 09:21:15.175169 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:15.175123 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2b12d9ab1d9faa5c67642e205f2e220db100410b45311806a1fcbeff1363945"} err="failed to get container status \"a2b12d9ab1d9faa5c67642e205f2e220db100410b45311806a1fcbeff1363945\": rpc error: code = NotFound desc = could not find container \"a2b12d9ab1d9faa5c67642e205f2e220db100410b45311806a1fcbeff1363945\": container with ID starting with a2b12d9ab1d9faa5c67642e205f2e220db100410b45311806a1fcbeff1363945 not found: ID does not exist" Oct 30 09:21:15.211336 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:15.211290 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ca0df2-7d67bcc9bb-b2brq"] Oct 30 09:21:15.215316 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:15.215294 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ca0df2-7d67bcc9bb-b2brq"] Oct 30 09:21:15.247903 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:15.247876 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ca0df2-57b8bf5d59-tbxzz"] Oct 30 09:21:15.251164 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:15.251139 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ca0df2-57b8bf5d59-tbxzz"] Oct 30 09:21:15.888203 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:15.888168 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="349c4a5f-88cf-473a-9463-283db5e5f586" path="/var/lib/kubelet/pods/349c4a5f-88cf-473a-9463-283db5e5f586/volumes" Oct 30 09:21:15.888653 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:15.888638 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d38fd70a-7b00-4c69-86d7-a6a58900a3bf" path="/var/lib/kubelet/pods/d38fd70a-7b00-4c69-86d7-a6a58900a3bf/volumes" Oct 30 09:21:26.479085 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:26.479053 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-648892-69475bb5d5-sthgm"] Oct 30 09:21:26.479925 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:26.479263 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-648892-69475bb5d5-sthgm" podUID="03e6f8da-1927-44fa-bab1-472ea5f845e0" containerName="maas-api" containerID="cri-o://8c2f0f1abaf66e4bcff8fb3b5b7ec6822479c75fcefebe12ea01fd6a90357fa0" gracePeriod=30 Oct 30 09:21:27.157495 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:27.157464 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-648892-69475bb5d5-sthgm" Oct 30 09:21:27.227619 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:27.227587 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03e6f8da-1927-44fa-bab1-472ea5f845e0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03e6f8da-1927-44fa-bab1-472ea5f845e0-maas-api-tls\") pod \"03e6f8da-1927-44fa-bab1-472ea5f845e0\" (UID: \"03e6f8da-1927-44fa-bab1-472ea5f845e0\") " Oct 30 09:21:27.227785 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:27.227694 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03e6f8da-1927-44fa-bab1-472ea5f845e0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/03e6f8da-1927-44fa-bab1-472ea5f845e0-metrics-certs\") pod \"03e6f8da-1927-44fa-bab1-472ea5f845e0\" (UID: \"03e6f8da-1927-44fa-bab1-472ea5f845e0\") " Oct 30 09:21:27.227785 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:27.227740 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03e6f8da-1927-44fa-bab1-472ea5f845e0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03e6f8da-1927-44fa-bab1-472ea5f845e0-openshift-service-ca\") pod \"03e6f8da-1927-44fa-bab1-472ea5f845e0\" (UID: \"03e6f8da-1927-44fa-bab1-472ea5f845e0\") " Oct 30 09:21:27.227935 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:27.227919 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03e6f8da-1927-44fa-bab1-472ea5f845e0-kube-api-access-hkx68\" (UniqueName: \"kubernetes.io/projected/03e6f8da-1927-44fa-bab1-472ea5f845e0-kube-api-access-hkx68\") pod \"03e6f8da-1927-44fa-bab1-472ea5f845e0\" (UID: \"03e6f8da-1927-44fa-bab1-472ea5f845e0\") " Oct 30 09:21:27.228192 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:27.228160 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03e6f8da-1927-44fa-bab1-472ea5f845e0-openshift-service-ca" pod "03e6f8da-1927-44fa-bab1-472ea5f845e0" (UID: "03e6f8da-1927-44fa-bab1-472ea5f845e0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:21:27.228597 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:27.228572 4562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03e6f8da-1927-44fa-bab1-472ea5f845e0-openshift-service-ca\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:21:27.229958 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:27.229930 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03e6f8da-1927-44fa-bab1-472ea5f845e0-metrics-certs" pod "03e6f8da-1927-44fa-bab1-472ea5f845e0" (UID: "03e6f8da-1927-44fa-bab1-472ea5f845e0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:21:27.229958 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:27.229940 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03e6f8da-1927-44fa-bab1-472ea5f845e0-maas-api-tls" pod "03e6f8da-1927-44fa-bab1-472ea5f845e0" (UID: "03e6f8da-1927-44fa-bab1-472ea5f845e0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:21:27.230147 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:27.230019 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03e6f8da-1927-44fa-bab1-472ea5f845e0-kube-api-access-hkx68" pod "03e6f8da-1927-44fa-bab1-472ea5f845e0" (UID: "03e6f8da-1927-44fa-bab1-472ea5f845e0"). InnerVolumeSpecName "kube-api-access-hkx68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:21:27.303006 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:27.302933 4562 generic.go:356] "Generic (PLEG): container finished" podID="03e6f8da-1927-44fa-bab1-472ea5f845e0" containerID="8c2f0f1abaf66e4bcff8fb3b5b7ec6822479c75fcefebe12ea01fd6a90357fa0" exitCode=0 Oct 30 09:21:27.303006 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:27.302985 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-648892-69475bb5d5-sthgm" Oct 30 09:21:27.303170 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:27.303012 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-648892-69475bb5d5-sthgm" event={"ID":"03e6f8da-1927-44fa-bab1-472ea5f845e0","Type":"ContainerDied","Data":"8c2f0f1abaf66e4bcff8fb3b5b7ec6822479c75fcefebe12ea01fd6a90357fa0"} Oct 30 09:21:27.303170 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:27.303065 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-648892-69475bb5d5-sthgm" event={"ID":"03e6f8da-1927-44fa-bab1-472ea5f845e0","Type":"ContainerDied","Data":"93cc2440b7f752c278175c19845ab54e4001085629f7a286f323085f977e70dc"} Oct 30 09:21:27.303170 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:27.303087 4562 scope.go:122] "RemoveContainer" containerID="8c2f0f1abaf66e4bcff8fb3b5b7ec6822479c75fcefebe12ea01fd6a90357fa0" Oct 30 09:21:27.313822 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:27.313796 4562 scope.go:122] "RemoveContainer" containerID="8c2f0f1abaf66e4bcff8fb3b5b7ec6822479c75fcefebe12ea01fd6a90357fa0" Oct 30 09:21:27.314111 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:21:27.314090 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c2f0f1abaf66e4bcff8fb3b5b7ec6822479c75fcefebe12ea01fd6a90357fa0\": container with ID starting with 8c2f0f1abaf66e4bcff8fb3b5b7ec6822479c75fcefebe12ea01fd6a90357fa0 not found: ID does not exist" containerID="8c2f0f1abaf66e4bcff8fb3b5b7ec6822479c75fcefebe12ea01fd6a90357fa0" Oct 30 09:21:27.314172 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:27.314122 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c2f0f1abaf66e4bcff8fb3b5b7ec6822479c75fcefebe12ea01fd6a90357fa0"} err="failed to get container status \"8c2f0f1abaf66e4bcff8fb3b5b7ec6822479c75fcefebe12ea01fd6a90357fa0\": rpc error: code = NotFound desc = could not find container \"8c2f0f1abaf66e4bcff8fb3b5b7ec6822479c75fcefebe12ea01fd6a90357fa0\": container with ID starting with 8c2f0f1abaf66e4bcff8fb3b5b7ec6822479c75fcefebe12ea01fd6a90357fa0 not found: ID does not exist" Oct 30 09:21:27.330454 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:27.330429 4562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03e6f8da-1927-44fa-bab1-472ea5f845e0-metrics-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:21:27.330454 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:27.330451 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkx68\" (UniqueName: \"kubernetes.io/projected/03e6f8da-1927-44fa-bab1-472ea5f845e0-kube-api-access-hkx68\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:21:27.330590 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:27.330461 4562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03e6f8da-1927-44fa-bab1-472ea5f845e0-maas-api-tls\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:21:27.357561 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:27.357537 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-648892-69475bb5d5-sthgm"] Oct 30 09:21:27.363661 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:27.363640 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-648892-69475bb5d5-sthgm"] Oct 30 09:21:27.875798 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:27.875764 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03e6f8da-1927-44fa-bab1-472ea5f845e0" path="/var/lib/kubelet/pods/03e6f8da-1927-44fa-bab1-472ea5f845e0/volumes" Oct 30 09:21:28.595147 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:21:28.595115 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:21:29.575895 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:29.575859 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11378-fb8d68"] Oct 30 09:21:29.576539 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:29.576523 4562 cpu_manager.go:414] "containerMap: removing container" podUID="d38fd70a-7b00-4c69-86d7-a6a58900a3bf" containerName="payload-processing" Oct 30 09:21:29.576599 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:29.576539 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d38fd70a-7b00-4c69-86d7-a6a58900a3bf" containerName="payload-processing" Oct 30 09:21:29.576599 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:29.576568 4562 cpu_manager.go:414] "containerMap: removing container" podUID="349c4a5f-88cf-473a-9463-283db5e5f586" containerName="payload-pre-processing" Oct 30 09:21:29.576599 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:29.576573 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="349c4a5f-88cf-473a-9463-283db5e5f586" containerName="payload-pre-processing" Oct 30 09:21:29.576599 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:29.576581 4562 cpu_manager.go:414] "containerMap: removing container" podUID="03e6f8da-1927-44fa-bab1-472ea5f845e0" containerName="maas-api" Oct 30 09:21:29.576599 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:29.576585 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03e6f8da-1927-44fa-bab1-472ea5f845e0" containerName="maas-api" Oct 30 09:21:29.576752 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:29.576601 4562 cpu_manager.go:414] "containerMap: removing container" podUID="56d7083b-d7cf-46d7-9469-6bcf9ef1d02a" containerName="payload-processing" Oct 30 09:21:29.576752 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:29.576607 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56d7083b-d7cf-46d7-9469-6bcf9ef1d02a" containerName="payload-processing" Oct 30 09:21:29.576752 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:29.576622 4562 cpu_manager.go:414] "containerMap: removing container" podUID="174415ce-8818-44e6-b26e-f95c8c4b3521" containerName="payload-pre-processing" Oct 30 09:21:29.576752 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:29.576627 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="174415ce-8818-44e6-b26e-f95c8c4b3521" containerName="payload-pre-processing" Oct 30 09:21:29.576752 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:29.576710 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="d38fd70a-7b00-4c69-86d7-a6a58900a3bf" containerName="payload-processing" Oct 30 09:21:29.576752 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:29.576718 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="349c4a5f-88cf-473a-9463-283db5e5f586" containerName="payload-pre-processing" Oct 30 09:21:29.576752 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:29.576723 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="03e6f8da-1927-44fa-bab1-472ea5f845e0" containerName="maas-api" Oct 30 09:21:29.576752 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:29.576731 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="56d7083b-d7cf-46d7-9469-6bcf9ef1d02a" containerName="payload-processing" Oct 30 09:21:29.576752 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:29.576744 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="174415ce-8818-44e6-b26e-f95c8c4b3521" containerName="payload-pre-processing" Oct 30 09:21:29.579990 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:29.579972 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11378-fb8d68" Oct 30 09:21:29.593836 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:29.593805 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11378-fb8d68"] Oct 30 09:21:29.602089 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:29.602057 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-jjt9f\"" Oct 30 09:21:29.668244 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:29.668208 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8k2z\" (UniqueName: \"kubernetes.io/projected/24a44609-ca7b-4ddd-9987-8480ebda4ae0-kube-api-access-n8k2z\") pod \"test-curl-11378-fb8d68\" (UID: \"24a44609-ca7b-4ddd-9987-8480ebda4ae0\") " pod="opendatahub/test-curl-11378-fb8d68" Oct 30 09:21:29.769794 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:29.769756 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8k2z\" (UniqueName: \"kubernetes.io/projected/24a44609-ca7b-4ddd-9987-8480ebda4ae0-kube-api-access-n8k2z\") pod \"test-curl-11378-fb8d68\" (UID: \"24a44609-ca7b-4ddd-9987-8480ebda4ae0\") " pod="opendatahub/test-curl-11378-fb8d68" Oct 30 09:21:29.789835 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:29.789808 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8k2z\" (UniqueName: \"kubernetes.io/projected/24a44609-ca7b-4ddd-9987-8480ebda4ae0-kube-api-access-n8k2z\") pod \"test-curl-11378-fb8d68\" (UID: \"24a44609-ca7b-4ddd-9987-8480ebda4ae0\") " pod="opendatahub/test-curl-11378-fb8d68" Oct 30 09:21:29.893271 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:29.893197 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11378-fb8d68" Oct 30 09:21:30.191132 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:21:30.191101 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod24a44609_ca7b_4ddd_9987_8480ebda4ae0.slice/crio-2418a42a4302c17bb402ed22e80ee333ac3650d85c25a200c1a7c2c38709634a WatchSource:0}: Error finding container 2418a42a4302c17bb402ed22e80ee333ac3650d85c25a200c1a7c2c38709634a: Status 404 returned error can't find the container with id 2418a42a4302c17bb402ed22e80ee333ac3650d85c25a200c1a7c2c38709634a Oct 30 09:21:30.192076 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:30.192050 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11378-fb8d68"] Oct 30 09:21:30.330462 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:30.330430 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11378-fb8d68" event={"ID":"24a44609-ca7b-4ddd-9987-8480ebda4ae0","Type":"ContainerStarted","Data":"2418a42a4302c17bb402ed22e80ee333ac3650d85c25a200c1a7c2c38709634a"} Oct 30 09:21:32.346746 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:32.346711 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11378-fb8d68" event={"ID":"24a44609-ca7b-4ddd-9987-8480ebda4ae0","Type":"ContainerStarted","Data":"a9b529f4a4e2bcc89edb24c0f558365356f8dcae67064bab8abdccfd0806be6c"} Oct 30 09:21:32.386047 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:32.385998 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11378-fb8d68" podStartSLOduration=1.392687589 podStartE2EDuration="3.385986119s" totalImagesPullingTime="1.99329853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:21:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:21:32.385182206 +0000 UTC m=+1652.646743211" watchObservedRunningTime="2026-10-30 09:21:32.385986119 +0000 UTC m=+1652.647547107" Oct 30 09:21:33.235032 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:33.234998 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11378-fb8d68"] Oct 30 09:21:33.239341 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:33.239312 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11378-fb8d68"] Oct 30 09:21:34.363802 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:34.361786 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11378-fb8d68" podUID="24a44609-ca7b-4ddd-9987-8480ebda4ae0" containerName="test-curl-11378-fb8d68" containerID="cri-o://a9b529f4a4e2bcc89edb24c0f558365356f8dcae67064bab8abdccfd0806be6c" gracePeriod=2 Oct 30 09:21:35.462364 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:35.462330 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ca0df2-664449fb4-vshdt"] Oct 30 09:21:35.462770 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:35.462585 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca0df2-664449fb4-vshdt" podUID="9af7addc-7b8d-4204-892f-8225b90ab5aa" containerName="maas-api" containerID="cri-o://fdfc96da80ae38619963654ec022b46cb93175a1a55f8f92e8b7a8403cb9ff71" gracePeriod=30 Oct 30 09:21:35.921623 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:35.921492 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca0df2-664449fb4-vshdt" Oct 30 09:21:36.015724 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:36.012022 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9af7addc-7b8d-4204-892f-8225b90ab5aa-kube-api-access-dnzp4\" (UniqueName: \"kubernetes.io/projected/9af7addc-7b8d-4204-892f-8225b90ab5aa-kube-api-access-dnzp4\") pod \"9af7addc-7b8d-4204-892f-8225b90ab5aa\" (UID: \"9af7addc-7b8d-4204-892f-8225b90ab5aa\") " Oct 30 09:21:36.015724 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:36.013538 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9af7addc-7b8d-4204-892f-8225b90ab5aa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9af7addc-7b8d-4204-892f-8225b90ab5aa-openshift-service-ca\") pod \"9af7addc-7b8d-4204-892f-8225b90ab5aa\" (UID: \"9af7addc-7b8d-4204-892f-8225b90ab5aa\") " Oct 30 09:21:36.015724 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:36.013739 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9af7addc-7b8d-4204-892f-8225b90ab5aa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9af7addc-7b8d-4204-892f-8225b90ab5aa-metrics-certs\") pod \"9af7addc-7b8d-4204-892f-8225b90ab5aa\" (UID: \"9af7addc-7b8d-4204-892f-8225b90ab5aa\") " Oct 30 09:21:36.015724 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:36.013936 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9af7addc-7b8d-4204-892f-8225b90ab5aa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9af7addc-7b8d-4204-892f-8225b90ab5aa-maas-api-tls\") pod \"9af7addc-7b8d-4204-892f-8225b90ab5aa\" (UID: \"9af7addc-7b8d-4204-892f-8225b90ab5aa\") " Oct 30 09:21:36.130454 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:36.126654 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9af7addc-7b8d-4204-892f-8225b90ab5aa-openshift-service-ca" pod "9af7addc-7b8d-4204-892f-8225b90ab5aa" (UID: "9af7addc-7b8d-4204-892f-8225b90ab5aa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:21:36.130454 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:36.126881 4562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9af7addc-7b8d-4204-892f-8225b90ab5aa-openshift-service-ca\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:21:36.130454 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:36.129105 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9af7addc-7b8d-4204-892f-8225b90ab5aa-kube-api-access-dnzp4" pod "9af7addc-7b8d-4204-892f-8225b90ab5aa" (UID: "9af7addc-7b8d-4204-892f-8225b90ab5aa"). InnerVolumeSpecName "kube-api-access-dnzp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:21:36.130454 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:36.129106 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9af7addc-7b8d-4204-892f-8225b90ab5aa-metrics-certs" pod "9af7addc-7b8d-4204-892f-8225b90ab5aa" (UID: "9af7addc-7b8d-4204-892f-8225b90ab5aa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:21:36.135776 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:36.135518 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9af7addc-7b8d-4204-892f-8225b90ab5aa-maas-api-tls" pod "9af7addc-7b8d-4204-892f-8225b90ab5aa" (UID: "9af7addc-7b8d-4204-892f-8225b90ab5aa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:21:36.228081 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:36.228048 4562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9af7addc-7b8d-4204-892f-8225b90ab5aa-metrics-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:21:36.228081 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:36.228076 4562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9af7addc-7b8d-4204-892f-8225b90ab5aa-maas-api-tls\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:21:36.228081 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:36.228085 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnzp4\" (UniqueName: \"kubernetes.io/projected/9af7addc-7b8d-4204-892f-8225b90ab5aa-kube-api-access-dnzp4\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:21:36.384925 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:36.384858 4562 generic.go:356] "Generic (PLEG): container finished" podID="9af7addc-7b8d-4204-892f-8225b90ab5aa" containerID="fdfc96da80ae38619963654ec022b46cb93175a1a55f8f92e8b7a8403cb9ff71" exitCode=0 Oct 30 09:21:36.385043 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:36.384934 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca0df2-664449fb4-vshdt" Oct 30 09:21:36.385043 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:36.384943 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca0df2-664449fb4-vshdt" event={"ID":"9af7addc-7b8d-4204-892f-8225b90ab5aa","Type":"ContainerDied","Data":"fdfc96da80ae38619963654ec022b46cb93175a1a55f8f92e8b7a8403cb9ff71"} Oct 30 09:21:36.385043 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:36.385031 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca0df2-664449fb4-vshdt" event={"ID":"9af7addc-7b8d-4204-892f-8225b90ab5aa","Type":"ContainerDied","Data":"1a381e8f1ddd749626bd45dc831bd7c369ad84803701d7c8e103542d8c01f821"} Oct 30 09:21:36.385147 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:36.385054 4562 scope.go:122] "RemoveContainer" containerID="fdfc96da80ae38619963654ec022b46cb93175a1a55f8f92e8b7a8403cb9ff71" Oct 30 09:21:36.420951 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:36.420933 4562 scope.go:122] "RemoveContainer" containerID="fdfc96da80ae38619963654ec022b46cb93175a1a55f8f92e8b7a8403cb9ff71" Oct 30 09:21:36.421259 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:21:36.421233 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdfc96da80ae38619963654ec022b46cb93175a1a55f8f92e8b7a8403cb9ff71\": container with ID starting with fdfc96da80ae38619963654ec022b46cb93175a1a55f8f92e8b7a8403cb9ff71 not found: ID does not exist" containerID="fdfc96da80ae38619963654ec022b46cb93175a1a55f8f92e8b7a8403cb9ff71" Oct 30 09:21:36.421310 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:36.421273 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdfc96da80ae38619963654ec022b46cb93175a1a55f8f92e8b7a8403cb9ff71"} err="failed to get container status \"fdfc96da80ae38619963654ec022b46cb93175a1a55f8f92e8b7a8403cb9ff71\": rpc error: code = NotFound desc = could not find container \"fdfc96da80ae38619963654ec022b46cb93175a1a55f8f92e8b7a8403cb9ff71\": container with ID starting with fdfc96da80ae38619963654ec022b46cb93175a1a55f8f92e8b7a8403cb9ff71 not found: ID does not exist" Oct 30 09:21:36.466010 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:36.465966 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ca0df2-664449fb4-vshdt"] Oct 30 09:21:36.478829 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:36.478799 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ca0df2-664449fb4-vshdt"] Oct 30 09:21:36.655612 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:36.655550 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11378-fb8d68" Oct 30 09:21:36.747412 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:36.747366 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24a44609-ca7b-4ddd-9987-8480ebda4ae0-kube-api-access-n8k2z\" (UniqueName: \"kubernetes.io/projected/24a44609-ca7b-4ddd-9987-8480ebda4ae0-kube-api-access-n8k2z\") pod \"24a44609-ca7b-4ddd-9987-8480ebda4ae0\" (UID: \"24a44609-ca7b-4ddd-9987-8480ebda4ae0\") " Oct 30 09:21:36.749622 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:36.749596 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24a44609-ca7b-4ddd-9987-8480ebda4ae0-kube-api-access-n8k2z" pod "24a44609-ca7b-4ddd-9987-8480ebda4ae0" (UID: "24a44609-ca7b-4ddd-9987-8480ebda4ae0"). InnerVolumeSpecName "kube-api-access-n8k2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:21:36.849102 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:36.849068 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8k2z\" (UniqueName: \"kubernetes.io/projected/24a44609-ca7b-4ddd-9987-8480ebda4ae0-kube-api-access-n8k2z\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:21:37.393877 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:37.393839 4562 generic.go:356] "Generic (PLEG): container finished" podID="24a44609-ca7b-4ddd-9987-8480ebda4ae0" containerID="a9b529f4a4e2bcc89edb24c0f558365356f8dcae67064bab8abdccfd0806be6c" exitCode=137 Oct 30 09:21:37.394082 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:37.393884 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11378-fb8d68" Oct 30 09:21:37.394697 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:37.394665 4562 scope.go:122] "RemoveContainer" containerID="a9b529f4a4e2bcc89edb24c0f558365356f8dcae67064bab8abdccfd0806be6c" Oct 30 09:21:37.404828 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:37.404809 4562 scope.go:122] "RemoveContainer" containerID="a9b529f4a4e2bcc89edb24c0f558365356f8dcae67064bab8abdccfd0806be6c" Oct 30 09:21:37.405152 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:21:37.405077 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9b529f4a4e2bcc89edb24c0f558365356f8dcae67064bab8abdccfd0806be6c\": container with ID starting with a9b529f4a4e2bcc89edb24c0f558365356f8dcae67064bab8abdccfd0806be6c not found: ID does not exist" containerID="a9b529f4a4e2bcc89edb24c0f558365356f8dcae67064bab8abdccfd0806be6c" Oct 30 09:21:37.405152 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:37.405111 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9b529f4a4e2bcc89edb24c0f558365356f8dcae67064bab8abdccfd0806be6c"} err="failed to get container status \"a9b529f4a4e2bcc89edb24c0f558365356f8dcae67064bab8abdccfd0806be6c\": rpc error: code = NotFound desc = could not find container \"a9b529f4a4e2bcc89edb24c0f558365356f8dcae67064bab8abdccfd0806be6c\": container with ID starting with a9b529f4a4e2bcc89edb24c0f558365356f8dcae67064bab8abdccfd0806be6c not found: ID does not exist" Oct 30 09:21:37.416862 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:21:37.416830 4562 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11378-fb8d68\" is forbidden: User \"system:node:ip-10-0-143-244.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-244.ec2.internal' and this object" podUID="24a44609-ca7b-4ddd-9987-8480ebda4ae0" pod="opendatahub/test-curl-11378-fb8d68" Oct 30 09:21:37.877620 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:37.877579 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24a44609-ca7b-4ddd-9987-8480ebda4ae0" path="/var/lib/kubelet/pods/24a44609-ca7b-4ddd-9987-8480ebda4ae0/volumes" Oct 30 09:21:37.878251 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:37.878228 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9af7addc-7b8d-4204-892f-8225b90ab5aa" path="/var/lib/kubelet/pods/9af7addc-7b8d-4204-892f-8225b90ab5aa/volumes" Oct 30 09:21:44.142990 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:44.142898 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8eada0cf-5884b8bb8-lp9q6"] Oct 30 09:21:44.175272 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:44.175243 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8eada0cf-7c5b9fd4ff-9jzlk"] Oct 30 09:21:45.431782 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:45.431748 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6c9de5-65c4494864-c5fmz"] Oct 30 09:21:45.466748 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:45.466709 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6c9de5-6fbf95ccff-sc8sg"] Oct 30 09:21:49.143983 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:49.143944 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-8eada0cf-5884b8bb8-lp9q6" podUID="79e02fb4-8a91-48ee-8fce-7fb16274008d" containerName="payload-processing" containerID="cri-o://7fc14de99612c11ff44a5329298c52a2cd6a0c00530c64ea0ca84993dd95275b" gracePeriod=25 Oct 30 09:21:49.176400 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:49.176343 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-8eada0cf-7c5b9fd4ff-9jzlk" podUID="58bd2646-2220-4f53-9133-c27c4f8b0c15" containerName="payload-pre-processing" containerID="cri-o://bd03c52899d66ab056f94c1cd59c14d1a06f38c7fcf5f68d9a508d0ccb5611d3" gracePeriod=25 Oct 30 09:21:50.408912 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:50.408849 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-8eada0cf-5884b8bb8-lp9q6" podUID="79e02fb4-8a91-48ee-8fce-7fb16274008d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:21:50.409844 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:50.409817 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-8eada0cf-7c5b9fd4ff-9jzlk" podUID="58bd2646-2220-4f53-9133-c27c4f8b0c15" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:21:50.432344 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:50.432294 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-6c9de5-65c4494864-c5fmz" podUID="6091ee8a-f7ce-4ea4-a8eb-44d13b5b1bdc" containerName="payload-processing" containerID="cri-o://cf052a94d8c78e6dcb14155e26d445dc551e641d42fe7a1198566f2d8b794b4e" gracePeriod=25 Oct 30 09:21:50.467960 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:50.467916 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6c9de5-6fbf95ccff-sc8sg" podUID="096665bc-2bb1-46b5-8561-1371424b699d" containerName="payload-pre-processing" containerID="cri-o://0a60717dbb3a6836bdeb923055daff61451b92706799b2cacaf064e79b75784f" gracePeriod=25 Oct 30 09:21:50.587364 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:50.587328 4562 generic.go:356] "Generic (PLEG): container finished" podID="6091ee8a-f7ce-4ea4-a8eb-44d13b5b1bdc" containerID="cf052a94d8c78e6dcb14155e26d445dc551e641d42fe7a1198566f2d8b794b4e" exitCode=0 Oct 30 09:21:50.587557 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:50.587534 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6c9de5-65c4494864-c5fmz" event={"ID":"6091ee8a-f7ce-4ea4-a8eb-44d13b5b1bdc","Type":"ContainerDied","Data":"cf052a94d8c78e6dcb14155e26d445dc551e641d42fe7a1198566f2d8b794b4e"} Oct 30 09:21:50.795274 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:50.795243 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6c9de5-65c4494864-c5fmz" Oct 30 09:21:50.826447 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:50.826426 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6c9de5-6fbf95ccff-sc8sg" Oct 30 09:21:50.888127 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:50.888095 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/096665bc-2bb1-46b5-8561-1371424b699d-kube-api-access-szfvd\" (UniqueName: \"kubernetes.io/projected/096665bc-2bb1-46b5-8561-1371424b699d-kube-api-access-szfvd\") pod \"096665bc-2bb1-46b5-8561-1371424b699d\" (UID: \"096665bc-2bb1-46b5-8561-1371424b699d\") " Oct 30 09:21:50.888320 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:50.888293 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6091ee8a-f7ce-4ea4-a8eb-44d13b5b1bdc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6091ee8a-f7ce-4ea4-a8eb-44d13b5b1bdc-plugins-config-volume\") pod \"6091ee8a-f7ce-4ea4-a8eb-44d13b5b1bdc\" (UID: \"6091ee8a-f7ce-4ea4-a8eb-44d13b5b1bdc\") " Oct 30 09:21:50.888417 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:50.888353 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/096665bc-2bb1-46b5-8561-1371424b699d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/096665bc-2bb1-46b5-8561-1371424b699d-plugins-config-volume\") pod \"096665bc-2bb1-46b5-8561-1371424b699d\" (UID: \"096665bc-2bb1-46b5-8561-1371424b699d\") " Oct 30 09:21:50.888505 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:50.888487 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6091ee8a-f7ce-4ea4-a8eb-44d13b5b1bdc-kube-api-access-swkgk\" (UniqueName: \"kubernetes.io/projected/6091ee8a-f7ce-4ea4-a8eb-44d13b5b1bdc-kube-api-access-swkgk\") pod \"6091ee8a-f7ce-4ea4-a8eb-44d13b5b1bdc\" (UID: \"6091ee8a-f7ce-4ea4-a8eb-44d13b5b1bdc\") " Oct 30 09:21:50.888767 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:50.888740 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6091ee8a-f7ce-4ea4-a8eb-44d13b5b1bdc-plugins-config-volume" pod "6091ee8a-f7ce-4ea4-a8eb-44d13b5b1bdc" (UID: "6091ee8a-f7ce-4ea4-a8eb-44d13b5b1bdc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:21:50.888767 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:50.888740 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/096665bc-2bb1-46b5-8561-1371424b699d-plugins-config-volume" pod "096665bc-2bb1-46b5-8561-1371424b699d" (UID: "096665bc-2bb1-46b5-8561-1371424b699d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:21:50.889184 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:50.889161 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6091ee8a-f7ce-4ea4-a8eb-44d13b5b1bdc-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:21:50.889268 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:50.889189 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/096665bc-2bb1-46b5-8561-1371424b699d-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:21:50.890476 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:50.890449 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/096665bc-2bb1-46b5-8561-1371424b699d-kube-api-access-szfvd" pod "096665bc-2bb1-46b5-8561-1371424b699d" (UID: "096665bc-2bb1-46b5-8561-1371424b699d"). InnerVolumeSpecName "kube-api-access-szfvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:21:50.890793 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:50.890771 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6091ee8a-f7ce-4ea4-a8eb-44d13b5b1bdc-kube-api-access-swkgk" pod "6091ee8a-f7ce-4ea4-a8eb-44d13b5b1bdc" (UID: "6091ee8a-f7ce-4ea4-a8eb-44d13b5b1bdc"). InnerVolumeSpecName "kube-api-access-swkgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:21:50.990526 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:50.990489 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szfvd\" (UniqueName: \"kubernetes.io/projected/096665bc-2bb1-46b5-8561-1371424b699d-kube-api-access-szfvd\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:21:50.990526 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:50.990527 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swkgk\" (UniqueName: \"kubernetes.io/projected/6091ee8a-f7ce-4ea4-a8eb-44d13b5b1bdc-kube-api-access-swkgk\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:21:51.595002 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:51.594969 4562 generic.go:356] "Generic (PLEG): container finished" podID="096665bc-2bb1-46b5-8561-1371424b699d" containerID="0a60717dbb3a6836bdeb923055daff61451b92706799b2cacaf064e79b75784f" exitCode=0 Oct 30 09:21:51.595475 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:51.595021 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6c9de5-6fbf95ccff-sc8sg" Oct 30 09:21:51.595475 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:51.595051 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6c9de5-6fbf95ccff-sc8sg" event={"ID":"096665bc-2bb1-46b5-8561-1371424b699d","Type":"ContainerDied","Data":"0a60717dbb3a6836bdeb923055daff61451b92706799b2cacaf064e79b75784f"} Oct 30 09:21:51.595475 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:51.595114 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6c9de5-6fbf95ccff-sc8sg" event={"ID":"096665bc-2bb1-46b5-8561-1371424b699d","Type":"ContainerDied","Data":"ee6f584715d83b23d67021d2ea9e3b182d1011747a4f96cd686e3b600fb4f5ec"} Oct 30 09:21:51.595475 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:51.595149 4562 scope.go:122] "RemoveContainer" containerID="0a60717dbb3a6836bdeb923055daff61451b92706799b2cacaf064e79b75784f" Oct 30 09:21:51.596836 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:51.596810 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6c9de5-65c4494864-c5fmz" event={"ID":"6091ee8a-f7ce-4ea4-a8eb-44d13b5b1bdc","Type":"ContainerDied","Data":"530b50c4d904fb18a174dff9802629ef5faafca3018d7ebbfd5a7344550de690"} Oct 30 09:21:51.596836 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:51.596828 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6c9de5-65c4494864-c5fmz" Oct 30 09:21:51.606127 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:51.606111 4562 scope.go:122] "RemoveContainer" containerID="0a60717dbb3a6836bdeb923055daff61451b92706799b2cacaf064e79b75784f" Oct 30 09:21:51.606503 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:21:51.606483 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a60717dbb3a6836bdeb923055daff61451b92706799b2cacaf064e79b75784f\": container with ID starting with 0a60717dbb3a6836bdeb923055daff61451b92706799b2cacaf064e79b75784f not found: ID does not exist" containerID="0a60717dbb3a6836bdeb923055daff61451b92706799b2cacaf064e79b75784f" Oct 30 09:21:51.606565 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:51.606510 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a60717dbb3a6836bdeb923055daff61451b92706799b2cacaf064e79b75784f"} err="failed to get container status \"0a60717dbb3a6836bdeb923055daff61451b92706799b2cacaf064e79b75784f\": rpc error: code = NotFound desc = could not find container \"0a60717dbb3a6836bdeb923055daff61451b92706799b2cacaf064e79b75784f\": container with ID starting with 0a60717dbb3a6836bdeb923055daff61451b92706799b2cacaf064e79b75784f not found: ID does not exist" Oct 30 09:21:51.606565 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:51.606527 4562 scope.go:122] "RemoveContainer" containerID="cf052a94d8c78e6dcb14155e26d445dc551e641d42fe7a1198566f2d8b794b4e" Oct 30 09:21:51.659830 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:51.659803 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6c9de5-65c4494864-c5fmz"] Oct 30 09:21:51.671143 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:51.671119 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6c9de5-65c4494864-c5fmz"] Oct 30 09:21:51.718944 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:51.718880 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6c9de5-6fbf95ccff-sc8sg"] Oct 30 09:21:51.727423 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:51.723316 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6c9de5-6fbf95ccff-sc8sg"] Oct 30 09:21:51.875674 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:51.875636 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="096665bc-2bb1-46b5-8561-1371424b699d" path="/var/lib/kubelet/pods/096665bc-2bb1-46b5-8561-1371424b699d/volumes" Oct 30 09:21:51.876110 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:51.876092 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6091ee8a-f7ce-4ea4-a8eb-44d13b5b1bdc" path="/var/lib/kubelet/pods/6091ee8a-f7ce-4ea4-a8eb-44d13b5b1bdc/volumes" Oct 30 09:21:54.266742 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:54.266708 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8d1106-78c5b8b468-zkwsh"] Oct 30 09:21:54.295782 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:54.295749 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8d1106-7447558595-2fhm8"] Oct 30 09:21:58.625964 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:21:58.625925 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:21:59.269298 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:59.269078 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-8d1106-78c5b8b468-zkwsh" podUID="57af6030-ca4c-48ff-b271-2917affcede5" containerName="payload-processing" containerID="cri-o://11ca51ee8761310678ad15961bc9f2a3036a98f94920ea2cb762c44bf3d3e5a4" gracePeriod=25 Oct 30 09:21:59.296211 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:59.296154 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8d1106-7447558595-2fhm8" podUID="d8b544fe-02d1-4d36-8122-ec15d90f14c7" containerName="payload-pre-processing" containerID="cri-o://f5488470922a70905cc8c860355ddd0dd099a8bd4966dc507a92d84395dc827b" gracePeriod=25 Oct 30 09:21:59.609842 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:59.609817 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8d1106-78c5b8b468-zkwsh" Oct 30 09:21:59.647858 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:59.647837 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8d1106-7447558595-2fhm8" Oct 30 09:21:59.665953 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:59.665918 4562 generic.go:356] "Generic (PLEG): container finished" podID="57af6030-ca4c-48ff-b271-2917affcede5" containerID="11ca51ee8761310678ad15961bc9f2a3036a98f94920ea2cb762c44bf3d3e5a4" exitCode=0 Oct 30 09:21:59.666097 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:59.665968 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8d1106-78c5b8b468-zkwsh" Oct 30 09:21:59.666097 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:59.666032 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8d1106-78c5b8b468-zkwsh" event={"ID":"57af6030-ca4c-48ff-b271-2917affcede5","Type":"ContainerDied","Data":"11ca51ee8761310678ad15961bc9f2a3036a98f94920ea2cb762c44bf3d3e5a4"} Oct 30 09:21:59.666215 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:59.666100 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8d1106-78c5b8b468-zkwsh" event={"ID":"57af6030-ca4c-48ff-b271-2917affcede5","Type":"ContainerDied","Data":"7a738dcd8b9fd8d780625f5014dda34988c9e884ac4fb8221fafc3ef8d842ec1"} Oct 30 09:21:59.666215 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:59.666143 4562 scope.go:122] "RemoveContainer" containerID="11ca51ee8761310678ad15961bc9f2a3036a98f94920ea2cb762c44bf3d3e5a4" Oct 30 09:21:59.668072 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:59.668044 4562 generic.go:356] "Generic (PLEG): container finished" podID="d8b544fe-02d1-4d36-8122-ec15d90f14c7" containerID="f5488470922a70905cc8c860355ddd0dd099a8bd4966dc507a92d84395dc827b" exitCode=0 Oct 30 09:21:59.668207 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:59.668187 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8d1106-7447558595-2fhm8" event={"ID":"d8b544fe-02d1-4d36-8122-ec15d90f14c7","Type":"ContainerDied","Data":"f5488470922a70905cc8c860355ddd0dd099a8bd4966dc507a92d84395dc827b"} Oct 30 09:21:59.668296 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:59.668238 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8d1106-7447558595-2fhm8" event={"ID":"d8b544fe-02d1-4d36-8122-ec15d90f14c7","Type":"ContainerDied","Data":"24bf2214100bd9c165dc75fc5bbe2a14b8e4d6c0dc1df8d84864a49f5ad833ad"} Oct 30 09:21:59.668359 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:59.668310 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8d1106-7447558595-2fhm8" Oct 30 09:21:59.676515 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:59.676488 4562 scope.go:122] "RemoveContainer" containerID="11ca51ee8761310678ad15961bc9f2a3036a98f94920ea2cb762c44bf3d3e5a4" Oct 30 09:21:59.676812 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:21:59.676784 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11ca51ee8761310678ad15961bc9f2a3036a98f94920ea2cb762c44bf3d3e5a4\": container with ID starting with 11ca51ee8761310678ad15961bc9f2a3036a98f94920ea2cb762c44bf3d3e5a4 not found: ID does not exist" containerID="11ca51ee8761310678ad15961bc9f2a3036a98f94920ea2cb762c44bf3d3e5a4" Oct 30 09:21:59.676902 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:59.676813 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11ca51ee8761310678ad15961bc9f2a3036a98f94920ea2cb762c44bf3d3e5a4"} err="failed to get container status \"11ca51ee8761310678ad15961bc9f2a3036a98f94920ea2cb762c44bf3d3e5a4\": rpc error: code = NotFound desc = could not find container \"11ca51ee8761310678ad15961bc9f2a3036a98f94920ea2cb762c44bf3d3e5a4\": container with ID starting with 11ca51ee8761310678ad15961bc9f2a3036a98f94920ea2cb762c44bf3d3e5a4 not found: ID does not exist" Oct 30 09:21:59.676902 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:59.676834 4562 scope.go:122] "RemoveContainer" containerID="f5488470922a70905cc8c860355ddd0dd099a8bd4966dc507a92d84395dc827b" Oct 30 09:21:59.688781 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:59.688759 4562 scope.go:122] "RemoveContainer" containerID="f5488470922a70905cc8c860355ddd0dd099a8bd4966dc507a92d84395dc827b" Oct 30 09:21:59.689140 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:21:59.689112 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5488470922a70905cc8c860355ddd0dd099a8bd4966dc507a92d84395dc827b\": container with ID starting with f5488470922a70905cc8c860355ddd0dd099a8bd4966dc507a92d84395dc827b not found: ID does not exist" containerID="f5488470922a70905cc8c860355ddd0dd099a8bd4966dc507a92d84395dc827b" Oct 30 09:21:59.689250 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:59.689150 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5488470922a70905cc8c860355ddd0dd099a8bd4966dc507a92d84395dc827b"} err="failed to get container status \"f5488470922a70905cc8c860355ddd0dd099a8bd4966dc507a92d84395dc827b\": rpc error: code = NotFound desc = could not find container \"f5488470922a70905cc8c860355ddd0dd099a8bd4966dc507a92d84395dc827b\": container with ID starting with f5488470922a70905cc8c860355ddd0dd099a8bd4966dc507a92d84395dc827b not found: ID does not exist" Oct 30 09:21:59.768753 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:59.768722 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57af6030-ca4c-48ff-b271-2917affcede5-kube-api-access-dzn5j\" (UniqueName: \"kubernetes.io/projected/57af6030-ca4c-48ff-b271-2917affcede5-kube-api-access-dzn5j\") pod \"57af6030-ca4c-48ff-b271-2917affcede5\" (UID: \"57af6030-ca4c-48ff-b271-2917affcede5\") " Oct 30 09:21:59.768930 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:59.768782 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8b544fe-02d1-4d36-8122-ec15d90f14c7-kube-api-access-z9knt\" (UniqueName: \"kubernetes.io/projected/d8b544fe-02d1-4d36-8122-ec15d90f14c7-kube-api-access-z9knt\") pod \"d8b544fe-02d1-4d36-8122-ec15d90f14c7\" (UID: \"d8b544fe-02d1-4d36-8122-ec15d90f14c7\") " Oct 30 09:21:59.768930 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:59.768845 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8b544fe-02d1-4d36-8122-ec15d90f14c7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8b544fe-02d1-4d36-8122-ec15d90f14c7-plugins-config-volume\") pod \"d8b544fe-02d1-4d36-8122-ec15d90f14c7\" (UID: \"d8b544fe-02d1-4d36-8122-ec15d90f14c7\") " Oct 30 09:21:59.768930 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:59.768907 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57af6030-ca4c-48ff-b271-2917affcede5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57af6030-ca4c-48ff-b271-2917affcede5-plugins-config-volume\") pod \"57af6030-ca4c-48ff-b271-2917affcede5\" (UID: \"57af6030-ca4c-48ff-b271-2917affcede5\") " Oct 30 09:21:59.769453 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:59.769291 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8b544fe-02d1-4d36-8122-ec15d90f14c7-plugins-config-volume" pod "d8b544fe-02d1-4d36-8122-ec15d90f14c7" (UID: "d8b544fe-02d1-4d36-8122-ec15d90f14c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:21:59.769598 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:59.769497 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57af6030-ca4c-48ff-b271-2917affcede5-plugins-config-volume" pod "57af6030-ca4c-48ff-b271-2917affcede5" (UID: "57af6030-ca4c-48ff-b271-2917affcede5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:21:59.769598 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:59.769525 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8b544fe-02d1-4d36-8122-ec15d90f14c7-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:21:59.771026 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:59.771001 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57af6030-ca4c-48ff-b271-2917affcede5-kube-api-access-dzn5j" pod "57af6030-ca4c-48ff-b271-2917affcede5" (UID: "57af6030-ca4c-48ff-b271-2917affcede5"). InnerVolumeSpecName "kube-api-access-dzn5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:21:59.771131 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:59.771066 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8b544fe-02d1-4d36-8122-ec15d90f14c7-kube-api-access-z9knt" pod "d8b544fe-02d1-4d36-8122-ec15d90f14c7" (UID: "d8b544fe-02d1-4d36-8122-ec15d90f14c7"). InnerVolumeSpecName "kube-api-access-z9knt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:21:59.870760 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:59.870679 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57af6030-ca4c-48ff-b271-2917affcede5-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:21:59.870760 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:59.870719 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzn5j\" (UniqueName: \"kubernetes.io/projected/57af6030-ca4c-48ff-b271-2917affcede5-kube-api-access-dzn5j\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:21:59.870760 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:21:59.870736 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9knt\" (UniqueName: \"kubernetes.io/projected/d8b544fe-02d1-4d36-8122-ec15d90f14c7-kube-api-access-z9knt\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:22:00.006800 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:00.006773 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8d1106-78c5b8b468-zkwsh"] Oct 30 09:22:00.019828 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:00.019806 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8d1106-78c5b8b468-zkwsh"] Oct 30 09:22:00.063939 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:00.059160 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8d1106-7447558595-2fhm8"] Oct 30 09:22:00.068978 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:00.068939 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8d1106-7447558595-2fhm8"] Oct 30 09:22:00.408820 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:00.408785 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-8eada0cf-5884b8bb8-lp9q6" podUID="79e02fb4-8a91-48ee-8fce-7fb16274008d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:22:00.409472 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:00.409440 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-8eada0cf-7c5b9fd4ff-9jzlk" podUID="58bd2646-2220-4f53-9133-c27c4f8b0c15" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:22:01.878315 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:01.878274 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57af6030-ca4c-48ff-b271-2917affcede5" path="/var/lib/kubelet/pods/57af6030-ca4c-48ff-b271-2917affcede5/volumes" Oct 30 09:22:01.878936 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:01.878915 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8b544fe-02d1-4d36-8122-ec15d90f14c7" path="/var/lib/kubelet/pods/d8b544fe-02d1-4d36-8122-ec15d90f14c7/volumes" Oct 30 09:22:04.569716 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:04.556690 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6c9de5-895f9fb7-48dpg"] Oct 30 09:22:04.569716 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:04.557013 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6c9de5-895f9fb7-48dpg" podUID="3764eee1-fcd1-4d69-8654-0d71e22845e4" containerName="maas-api" containerID="cri-o://fb959c2c6aabc1998748458c5e000ea1eca8d5a6f28b76ba74398a7de904706a" gracePeriod=30 Oct 30 09:22:05.162531 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:05.162505 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6c9de5-895f9fb7-48dpg" Oct 30 09:22:05.339569 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:05.336830 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3764eee1-fcd1-4d69-8654-0d71e22845e4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3764eee1-fcd1-4d69-8654-0d71e22845e4-maas-api-tls\") pod \"3764eee1-fcd1-4d69-8654-0d71e22845e4\" (UID: \"3764eee1-fcd1-4d69-8654-0d71e22845e4\") " Oct 30 09:22:05.339569 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:05.336896 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3764eee1-fcd1-4d69-8654-0d71e22845e4-kube-api-access-r7rtp\" (UniqueName: \"kubernetes.io/projected/3764eee1-fcd1-4d69-8654-0d71e22845e4-kube-api-access-r7rtp\") pod \"3764eee1-fcd1-4d69-8654-0d71e22845e4\" (UID: \"3764eee1-fcd1-4d69-8654-0d71e22845e4\") " Oct 30 09:22:05.339569 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:05.336992 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3764eee1-fcd1-4d69-8654-0d71e22845e4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3764eee1-fcd1-4d69-8654-0d71e22845e4-openshift-service-ca\") pod \"3764eee1-fcd1-4d69-8654-0d71e22845e4\" (UID: \"3764eee1-fcd1-4d69-8654-0d71e22845e4\") " Oct 30 09:22:05.339569 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:05.337188 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3764eee1-fcd1-4d69-8654-0d71e22845e4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3764eee1-fcd1-4d69-8654-0d71e22845e4-metrics-certs\") pod \"3764eee1-fcd1-4d69-8654-0d71e22845e4\" (UID: \"3764eee1-fcd1-4d69-8654-0d71e22845e4\") " Oct 30 09:22:05.377739 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:05.377678 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3764eee1-fcd1-4d69-8654-0d71e22845e4-openshift-service-ca" pod "3764eee1-fcd1-4d69-8654-0d71e22845e4" (UID: "3764eee1-fcd1-4d69-8654-0d71e22845e4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:22:05.379119 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:05.379085 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3764eee1-fcd1-4d69-8654-0d71e22845e4-metrics-certs" pod "3764eee1-fcd1-4d69-8654-0d71e22845e4" (UID: "3764eee1-fcd1-4d69-8654-0d71e22845e4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:22:05.380031 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:05.380000 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3764eee1-fcd1-4d69-8654-0d71e22845e4-maas-api-tls" pod "3764eee1-fcd1-4d69-8654-0d71e22845e4" (UID: "3764eee1-fcd1-4d69-8654-0d71e22845e4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:22:05.397007 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:05.396946 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3764eee1-fcd1-4d69-8654-0d71e22845e4-kube-api-access-r7rtp" pod "3764eee1-fcd1-4d69-8654-0d71e22845e4" (UID: "3764eee1-fcd1-4d69-8654-0d71e22845e4"). InnerVolumeSpecName "kube-api-access-r7rtp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:22:05.439318 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:05.439276 4562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3764eee1-fcd1-4d69-8654-0d71e22845e4-maas-api-tls\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:22:05.439318 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:05.439312 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7rtp\" (UniqueName: \"kubernetes.io/projected/3764eee1-fcd1-4d69-8654-0d71e22845e4-kube-api-access-r7rtp\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:22:05.439576 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:05.439327 4562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3764eee1-fcd1-4d69-8654-0d71e22845e4-openshift-service-ca\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:22:05.439576 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:05.439339 4562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3764eee1-fcd1-4d69-8654-0d71e22845e4-metrics-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:22:05.812963 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:05.812928 4562 generic.go:356] "Generic (PLEG): container finished" podID="3764eee1-fcd1-4d69-8654-0d71e22845e4" containerID="fb959c2c6aabc1998748458c5e000ea1eca8d5a6f28b76ba74398a7de904706a" exitCode=0 Oct 30 09:22:05.813427 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:05.812972 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6c9de5-895f9fb7-48dpg" Oct 30 09:22:05.813427 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:05.813009 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6c9de5-895f9fb7-48dpg" event={"ID":"3764eee1-fcd1-4d69-8654-0d71e22845e4","Type":"ContainerDied","Data":"fb959c2c6aabc1998748458c5e000ea1eca8d5a6f28b76ba74398a7de904706a"} Oct 30 09:22:05.813427 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:05.813062 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6c9de5-895f9fb7-48dpg" event={"ID":"3764eee1-fcd1-4d69-8654-0d71e22845e4","Type":"ContainerDied","Data":"29f417ae9ce00799b1a1b87b1bd54365dacb8f95555d44382762f802c57a911a"} Oct 30 09:22:05.813427 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:05.813092 4562 scope.go:122] "RemoveContainer" containerID="fb959c2c6aabc1998748458c5e000ea1eca8d5a6f28b76ba74398a7de904706a" Oct 30 09:22:05.823505 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:05.823484 4562 scope.go:122] "RemoveContainer" containerID="fb959c2c6aabc1998748458c5e000ea1eca8d5a6f28b76ba74398a7de904706a" Oct 30 09:22:05.823810 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:22:05.823779 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb959c2c6aabc1998748458c5e000ea1eca8d5a6f28b76ba74398a7de904706a\": container with ID starting with fb959c2c6aabc1998748458c5e000ea1eca8d5a6f28b76ba74398a7de904706a not found: ID does not exist" containerID="fb959c2c6aabc1998748458c5e000ea1eca8d5a6f28b76ba74398a7de904706a" Oct 30 09:22:05.823889 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:05.823820 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb959c2c6aabc1998748458c5e000ea1eca8d5a6f28b76ba74398a7de904706a"} err="failed to get container status \"fb959c2c6aabc1998748458c5e000ea1eca8d5a6f28b76ba74398a7de904706a\": rpc error: code = NotFound desc = could not find container \"fb959c2c6aabc1998748458c5e000ea1eca8d5a6f28b76ba74398a7de904706a\": container with ID starting with fb959c2c6aabc1998748458c5e000ea1eca8d5a6f28b76ba74398a7de904706a not found: ID does not exist" Oct 30 09:22:05.874157 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:05.874131 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6c9de5-895f9fb7-48dpg"] Oct 30 09:22:05.881194 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:05.881172 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6c9de5-895f9fb7-48dpg"] Oct 30 09:22:06.286364 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.286327 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8"] Oct 30 09:22:06.287315 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.287291 4562 cpu_manager.go:414] "containerMap: removing container" podUID="57af6030-ca4c-48ff-b271-2917affcede5" containerName="payload-processing" Oct 30 09:22:06.287430 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.287318 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57af6030-ca4c-48ff-b271-2917affcede5" containerName="payload-processing" Oct 30 09:22:06.287430 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.287364 4562 cpu_manager.go:414] "containerMap: removing container" podUID="6091ee8a-f7ce-4ea4-a8eb-44d13b5b1bdc" containerName="payload-processing" Oct 30 09:22:06.287430 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.287372 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6091ee8a-f7ce-4ea4-a8eb-44d13b5b1bdc" containerName="payload-processing" Oct 30 09:22:06.287430 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.287428 4562 cpu_manager.go:414] "containerMap: removing container" podUID="3764eee1-fcd1-4d69-8654-0d71e22845e4" containerName="maas-api" Oct 30 09:22:06.287649 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.287436 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3764eee1-fcd1-4d69-8654-0d71e22845e4" containerName="maas-api" Oct 30 09:22:06.287649 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.287473 4562 cpu_manager.go:414] "containerMap: removing container" podUID="d8b544fe-02d1-4d36-8122-ec15d90f14c7" containerName="payload-pre-processing" Oct 30 09:22:06.287649 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.287482 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8b544fe-02d1-4d36-8122-ec15d90f14c7" containerName="payload-pre-processing" Oct 30 09:22:06.287649 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.287502 4562 cpu_manager.go:414] "containerMap: removing container" podUID="9af7addc-7b8d-4204-892f-8225b90ab5aa" containerName="maas-api" Oct 30 09:22:06.287649 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.287509 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9af7addc-7b8d-4204-892f-8225b90ab5aa" containerName="maas-api" Oct 30 09:22:06.287649 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.287526 4562 cpu_manager.go:414] "containerMap: removing container" podUID="24a44609-ca7b-4ddd-9987-8480ebda4ae0" containerName="test-curl-11378-fb8d68" Oct 30 09:22:06.287649 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.287534 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24a44609-ca7b-4ddd-9987-8480ebda4ae0" containerName="test-curl-11378-fb8d68" Oct 30 09:22:06.287649 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.287550 4562 cpu_manager.go:414] "containerMap: removing container" podUID="096665bc-2bb1-46b5-8561-1371424b699d" containerName="payload-pre-processing" Oct 30 09:22:06.287649 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.287558 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="096665bc-2bb1-46b5-8561-1371424b699d" containerName="payload-pre-processing" Oct 30 09:22:06.287910 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.287700 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="9af7addc-7b8d-4204-892f-8225b90ab5aa" containerName="maas-api" Oct 30 09:22:06.287910 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.287716 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8b544fe-02d1-4d36-8122-ec15d90f14c7" containerName="payload-pre-processing" Oct 30 09:22:06.287910 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.287725 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="096665bc-2bb1-46b5-8561-1371424b699d" containerName="payload-pre-processing" Oct 30 09:22:06.287910 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.287735 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="6091ee8a-f7ce-4ea4-a8eb-44d13b5b1bdc" containerName="payload-processing" Oct 30 09:22:06.287910 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.287744 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="3764eee1-fcd1-4d69-8654-0d71e22845e4" containerName="maas-api" Oct 30 09:22:06.287910 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.287754 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="57af6030-ca4c-48ff-b271-2917affcede5" containerName="payload-processing" Oct 30 09:22:06.287910 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.287762 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="24a44609-ca7b-4ddd-9987-8480ebda4ae0" containerName="test-curl-11378-fb8d68" Oct 30 09:22:06.293330 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.293308 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8" Oct 30 09:22:06.312858 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.312827 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8"] Oct 30 09:22:06.319658 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.319634 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-8de308-openshift-default-dockercfg-9xrhg\"" Oct 30 09:22:06.459408 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.453455 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b433eaf-bc2e-4a08-8e7a-527e88ef4c59-workload-socket\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8\" (UID: \"7b433eaf-bc2e-4a08-8e7a-527e88ef4c59\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8" Oct 30 09:22:06.459408 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.453535 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b433eaf-bc2e-4a08-8e7a-527e88ef4c59-istio-podinfo\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8\" (UID: \"7b433eaf-bc2e-4a08-8e7a-527e88ef4c59\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8" Oct 30 09:22:06.459408 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.453632 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b433eaf-bc2e-4a08-8e7a-527e88ef4c59-istio-envoy\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8\" (UID: \"7b433eaf-bc2e-4a08-8e7a-527e88ef4c59\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8" Oct 30 09:22:06.459408 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.453684 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggwg8\" (UniqueName: \"kubernetes.io/projected/7b433eaf-bc2e-4a08-8e7a-527e88ef4c59-kube-api-access-ggwg8\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8\" (UID: \"7b433eaf-bc2e-4a08-8e7a-527e88ef4c59\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8" Oct 30 09:22:06.459408 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.453712 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b433eaf-bc2e-4a08-8e7a-527e88ef4c59-istio-token\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8\" (UID: \"7b433eaf-bc2e-4a08-8e7a-527e88ef4c59\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8" Oct 30 09:22:06.459408 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.453748 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b433eaf-bc2e-4a08-8e7a-527e88ef4c59-istiod-ca-cert\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8\" (UID: \"7b433eaf-bc2e-4a08-8e7a-527e88ef4c59\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8" Oct 30 09:22:06.459408 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.453834 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b433eaf-bc2e-4a08-8e7a-527e88ef4c59-istio-data\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8\" (UID: \"7b433eaf-bc2e-4a08-8e7a-527e88ef4c59\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8" Oct 30 09:22:06.459408 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.453868 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b433eaf-bc2e-4a08-8e7a-527e88ef4c59-workload-certs\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8\" (UID: \"7b433eaf-bc2e-4a08-8e7a-527e88ef4c59\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8" Oct 30 09:22:06.459408 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.453977 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b433eaf-bc2e-4a08-8e7a-527e88ef4c59-credential-socket\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8\" (UID: \"7b433eaf-bc2e-4a08-8e7a-527e88ef4c59\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8" Oct 30 09:22:06.571452 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.566854 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b433eaf-bc2e-4a08-8e7a-527e88ef4c59-istio-envoy\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8\" (UID: \"7b433eaf-bc2e-4a08-8e7a-527e88ef4c59\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8" Oct 30 09:22:06.571452 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.566931 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggwg8\" (UniqueName: \"kubernetes.io/projected/7b433eaf-bc2e-4a08-8e7a-527e88ef4c59-kube-api-access-ggwg8\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8\" (UID: \"7b433eaf-bc2e-4a08-8e7a-527e88ef4c59\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8" Oct 30 09:22:06.571452 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.566958 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b433eaf-bc2e-4a08-8e7a-527e88ef4c59-istio-token\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8\" (UID: \"7b433eaf-bc2e-4a08-8e7a-527e88ef4c59\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8" Oct 30 09:22:06.571452 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.566987 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b433eaf-bc2e-4a08-8e7a-527e88ef4c59-istiod-ca-cert\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8\" (UID: \"7b433eaf-bc2e-4a08-8e7a-527e88ef4c59\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8" Oct 30 09:22:06.571452 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.567063 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b433eaf-bc2e-4a08-8e7a-527e88ef4c59-istio-data\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8\" (UID: \"7b433eaf-bc2e-4a08-8e7a-527e88ef4c59\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8" Oct 30 09:22:06.571452 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.567093 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b433eaf-bc2e-4a08-8e7a-527e88ef4c59-workload-certs\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8\" (UID: \"7b433eaf-bc2e-4a08-8e7a-527e88ef4c59\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8" Oct 30 09:22:06.571452 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.567151 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b433eaf-bc2e-4a08-8e7a-527e88ef4c59-credential-socket\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8\" (UID: \"7b433eaf-bc2e-4a08-8e7a-527e88ef4c59\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8" Oct 30 09:22:06.571452 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.567264 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b433eaf-bc2e-4a08-8e7a-527e88ef4c59-workload-socket\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8\" (UID: \"7b433eaf-bc2e-4a08-8e7a-527e88ef4c59\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8" Oct 30 09:22:06.571452 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.567317 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b433eaf-bc2e-4a08-8e7a-527e88ef4c59-istio-podinfo\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8\" (UID: \"7b433eaf-bc2e-4a08-8e7a-527e88ef4c59\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8" Oct 30 09:22:06.571452 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.568806 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b433eaf-bc2e-4a08-8e7a-527e88ef4c59-istiod-ca-cert\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8\" (UID: \"7b433eaf-bc2e-4a08-8e7a-527e88ef4c59\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8" Oct 30 09:22:06.571452 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.570715 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b433eaf-bc2e-4a08-8e7a-527e88ef4c59-credential-socket\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8\" (UID: \"7b433eaf-bc2e-4a08-8e7a-527e88ef4c59\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8" Oct 30 09:22:06.571452 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.571264 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b433eaf-bc2e-4a08-8e7a-527e88ef4c59-istio-data\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8\" (UID: \"7b433eaf-bc2e-4a08-8e7a-527e88ef4c59\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8" Oct 30 09:22:06.572259 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.571566 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b433eaf-bc2e-4a08-8e7a-527e88ef4c59-workload-socket\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8\" (UID: \"7b433eaf-bc2e-4a08-8e7a-527e88ef4c59\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8" Oct 30 09:22:06.572259 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.572166 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b433eaf-bc2e-4a08-8e7a-527e88ef4c59-workload-certs\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8\" (UID: \"7b433eaf-bc2e-4a08-8e7a-527e88ef4c59\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8" Oct 30 09:22:06.597437 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.595245 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b433eaf-bc2e-4a08-8e7a-527e88ef4c59-istio-envoy\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8\" (UID: \"7b433eaf-bc2e-4a08-8e7a-527e88ef4c59\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8" Oct 30 09:22:06.619409 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.607714 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b433eaf-bc2e-4a08-8e7a-527e88ef4c59-istio-token\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8\" (UID: \"7b433eaf-bc2e-4a08-8e7a-527e88ef4c59\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8" Oct 30 09:22:06.619409 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.608621 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggwg8\" (UniqueName: \"kubernetes.io/projected/7b433eaf-bc2e-4a08-8e7a-527e88ef4c59-kube-api-access-ggwg8\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8\" (UID: \"7b433eaf-bc2e-4a08-8e7a-527e88ef4c59\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8" Oct 30 09:22:06.619409 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.609659 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b433eaf-bc2e-4a08-8e7a-527e88ef4c59-istio-podinfo\") pod \"e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8\" (UID: \"7b433eaf-bc2e-4a08-8e7a-527e88ef4c59\") " pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8" Oct 30 09:22:06.635443 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.633971 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k7rc9"] Oct 30 09:22:06.648463 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.646247 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k7rc9" Oct 30 09:22:06.648463 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.647246 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8" Oct 30 09:22:06.668127 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.665134 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k7rc9"] Oct 30 09:22:06.772782 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.772361 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/866a5fb8-5939-4f6a-b01f-060e32222484-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-k7rc9\" (UID: \"866a5fb8-5939-4f6a-b01f-060e32222484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k7rc9" Oct 30 09:22:06.772782 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.772526 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/866a5fb8-5939-4f6a-b01f-060e32222484-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-k7rc9\" (UID: \"866a5fb8-5939-4f6a-b01f-060e32222484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k7rc9" Oct 30 09:22:06.772782 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.772634 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/866a5fb8-5939-4f6a-b01f-060e32222484-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-k7rc9\" (UID: \"866a5fb8-5939-4f6a-b01f-060e32222484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k7rc9" Oct 30 09:22:06.772782 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.772659 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/866a5fb8-5939-4f6a-b01f-060e32222484-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-k7rc9\" (UID: \"866a5fb8-5939-4f6a-b01f-060e32222484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k7rc9" Oct 30 09:22:06.772782 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.772688 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/866a5fb8-5939-4f6a-b01f-060e32222484-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-k7rc9\" (UID: \"866a5fb8-5939-4f6a-b01f-060e32222484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k7rc9" Oct 30 09:22:06.772782 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.772704 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79mjt\" (UniqueName: \"kubernetes.io/projected/866a5fb8-5939-4f6a-b01f-060e32222484-kube-api-access-79mjt\") pod \"maas-default-gateway-openshift-default-77f556d647-k7rc9\" (UID: \"866a5fb8-5939-4f6a-b01f-060e32222484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k7rc9" Oct 30 09:22:06.772782 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.772725 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/866a5fb8-5939-4f6a-b01f-060e32222484-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-k7rc9\" (UID: \"866a5fb8-5939-4f6a-b01f-060e32222484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k7rc9" Oct 30 09:22:06.773290 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.772832 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/866a5fb8-5939-4f6a-b01f-060e32222484-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-k7rc9\" (UID: \"866a5fb8-5939-4f6a-b01f-060e32222484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k7rc9" Oct 30 09:22:06.773290 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.772898 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/866a5fb8-5939-4f6a-b01f-060e32222484-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-k7rc9\" (UID: \"866a5fb8-5939-4f6a-b01f-060e32222484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k7rc9" Oct 30 09:22:06.874774 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.874674 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/866a5fb8-5939-4f6a-b01f-060e32222484-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-k7rc9\" (UID: \"866a5fb8-5939-4f6a-b01f-060e32222484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k7rc9" Oct 30 09:22:06.874774 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.874732 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79mjt\" (UniqueName: \"kubernetes.io/projected/866a5fb8-5939-4f6a-b01f-060e32222484-kube-api-access-79mjt\") pod \"maas-default-gateway-openshift-default-77f556d647-k7rc9\" (UID: \"866a5fb8-5939-4f6a-b01f-060e32222484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k7rc9" Oct 30 09:22:06.874774 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.874761 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/866a5fb8-5939-4f6a-b01f-060e32222484-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-k7rc9\" (UID: \"866a5fb8-5939-4f6a-b01f-060e32222484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k7rc9" Oct 30 09:22:06.875465 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.874917 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/866a5fb8-5939-4f6a-b01f-060e32222484-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-k7rc9\" (UID: \"866a5fb8-5939-4f6a-b01f-060e32222484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k7rc9" Oct 30 09:22:06.875465 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.875016 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/866a5fb8-5939-4f6a-b01f-060e32222484-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-k7rc9\" (UID: \"866a5fb8-5939-4f6a-b01f-060e32222484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k7rc9" Oct 30 09:22:06.875465 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.875101 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/866a5fb8-5939-4f6a-b01f-060e32222484-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-k7rc9\" (UID: \"866a5fb8-5939-4f6a-b01f-060e32222484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k7rc9" Oct 30 09:22:06.875465 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.875123 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/866a5fb8-5939-4f6a-b01f-060e32222484-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-k7rc9\" (UID: \"866a5fb8-5939-4f6a-b01f-060e32222484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k7rc9" Oct 30 09:22:06.875465 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.875157 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/866a5fb8-5939-4f6a-b01f-060e32222484-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-k7rc9\" (UID: \"866a5fb8-5939-4f6a-b01f-060e32222484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k7rc9" Oct 30 09:22:06.875465 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.875205 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/866a5fb8-5939-4f6a-b01f-060e32222484-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-k7rc9\" (UID: \"866a5fb8-5939-4f6a-b01f-060e32222484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k7rc9" Oct 30 09:22:06.875465 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.875239 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/866a5fb8-5939-4f6a-b01f-060e32222484-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-k7rc9\" (UID: \"866a5fb8-5939-4f6a-b01f-060e32222484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k7rc9" Oct 30 09:22:06.875465 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.875445 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/866a5fb8-5939-4f6a-b01f-060e32222484-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-k7rc9\" (UID: \"866a5fb8-5939-4f6a-b01f-060e32222484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k7rc9" Oct 30 09:22:06.875882 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.875717 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/866a5fb8-5939-4f6a-b01f-060e32222484-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-k7rc9\" (UID: \"866a5fb8-5939-4f6a-b01f-060e32222484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k7rc9" Oct 30 09:22:06.875882 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.875785 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/866a5fb8-5939-4f6a-b01f-060e32222484-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-k7rc9\" (UID: \"866a5fb8-5939-4f6a-b01f-060e32222484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k7rc9" Oct 30 09:22:06.876015 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.875990 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/866a5fb8-5939-4f6a-b01f-060e32222484-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-k7rc9\" (UID: \"866a5fb8-5939-4f6a-b01f-060e32222484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k7rc9" Oct 30 09:22:06.877573 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.877550 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/866a5fb8-5939-4f6a-b01f-060e32222484-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-k7rc9\" (UID: \"866a5fb8-5939-4f6a-b01f-060e32222484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k7rc9" Oct 30 09:22:06.877813 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.877796 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/866a5fb8-5939-4f6a-b01f-060e32222484-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-k7rc9\" (UID: \"866a5fb8-5939-4f6a-b01f-060e32222484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k7rc9" Oct 30 09:22:06.894638 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.894609 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79mjt\" (UniqueName: \"kubernetes.io/projected/866a5fb8-5939-4f6a-b01f-060e32222484-kube-api-access-79mjt\") pod \"maas-default-gateway-openshift-default-77f556d647-k7rc9\" (UID: \"866a5fb8-5939-4f6a-b01f-060e32222484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k7rc9" Oct 30 09:22:06.894829 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:06.894811 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/866a5fb8-5939-4f6a-b01f-060e32222484-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-k7rc9\" (UID: \"866a5fb8-5939-4f6a-b01f-060e32222484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k7rc9" Oct 30 09:22:07.047991 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:07.047953 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k7rc9" Oct 30 09:22:07.059697 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:07.059670 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8"] Oct 30 09:22:07.060611 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:22:07.060582 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b433eaf_bc2e_4a08_8e7a_527e88ef4c59.slice/crio-99685a22e14fb86a27c52799e40a9da7f7037ebc680de2246e1f0e80c3daab40 WatchSource:0}: Error finding container 99685a22e14fb86a27c52799e40a9da7f7037ebc680de2246e1f0e80c3daab40: Status 404 returned error can't find the container with id 99685a22e14fb86a27c52799e40a9da7f7037ebc680de2246e1f0e80c3daab40 Oct 30 09:22:07.062810 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:07.062777 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:22:07.062903 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:07.062841 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:22:07.062903 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:07.062873 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:22:07.376950 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:22:07.376915 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod866a5fb8_5939_4f6a_b01f_060e32222484.slice/crio-d097f8d4f8f5f5e18b6e48a383a69ba4dc46e19e275caf3a66f28aa83a4e9ff1 WatchSource:0}: Error finding container d097f8d4f8f5f5e18b6e48a383a69ba4dc46e19e275caf3a66f28aa83a4e9ff1: Status 404 returned error can't find the container with id d097f8d4f8f5f5e18b6e48a383a69ba4dc46e19e275caf3a66f28aa83a4e9ff1 Oct 30 09:22:07.378615 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:07.378590 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k7rc9"] Oct 30 09:22:07.379877 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:07.379844 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:22:07.379958 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:07.379920 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:22:07.379995 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:07.379958 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:22:07.879849 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:07.879819 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3764eee1-fcd1-4d69-8654-0d71e22845e4" path="/var/lib/kubelet/pods/3764eee1-fcd1-4d69-8654-0d71e22845e4/volumes" Oct 30 09:22:07.882187 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:07.882153 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8" event={"ID":"7b433eaf-bc2e-4a08-8e7a-527e88ef4c59","Type":"ContainerStarted","Data":"4a49092088ec33406b51283b5dc3b0064f3bbd844e34fe625074b009ee0ca805"} Oct 30 09:22:07.882341 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:07.882218 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8" event={"ID":"7b433eaf-bc2e-4a08-8e7a-527e88ef4c59","Type":"ContainerStarted","Data":"99685a22e14fb86a27c52799e40a9da7f7037ebc680de2246e1f0e80c3daab40"} Oct 30 09:22:07.883861 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:07.883836 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k7rc9" event={"ID":"866a5fb8-5939-4f6a-b01f-060e32222484","Type":"ContainerStarted","Data":"b132c538dc26d7d540925a98019ccb6a34c340d12a8fae92c383925d25de4064"} Oct 30 09:22:07.883979 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:07.883879 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k7rc9" event={"ID":"866a5fb8-5939-4f6a-b01f-060e32222484","Type":"ContainerStarted","Data":"d097f8d4f8f5f5e18b6e48a383a69ba4dc46e19e275caf3a66f28aa83a4e9ff1"} Oct 30 09:22:07.928947 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:07.928905 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8" podStartSLOduration=1.928891852 podStartE2EDuration="1.928891852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:22:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:22:07.925695209 +0000 UTC m=+1688.187256212" watchObservedRunningTime="2026-10-30 09:22:07.928891852 +0000 UTC m=+1688.190452839" Oct 30 09:22:07.974077 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:07.974024 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k7rc9" podStartSLOduration=1.974010109 podStartE2EDuration="1.974010109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:22:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:22:07.971539345 +0000 UTC m=+1688.233100334" watchObservedRunningTime="2026-10-30 09:22:07.974010109 +0000 UTC m=+1688.235571097" Oct 30 09:22:08.054567 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:08.052787 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k7rc9" Oct 30 09:22:08.055006 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:08.054964 4562 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-k7rc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 30 09:22:08.055196 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:08.055170 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k7rc9" podUID="866a5fb8-5939-4f6a-b01f-060e32222484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 30 09:22:08.662355 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:08.661494 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8" Oct 30 09:22:08.667596 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:08.667564 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8" Oct 30 09:22:08.890817 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:08.890785 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8" Oct 30 09:22:08.891658 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:08.891636 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8" Oct 30 09:22:09.095679 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:09.090964 4562 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-k7rc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 30 09:22:09.095679 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:09.091023 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k7rc9" podUID="866a5fb8-5939-4f6a-b01f-060e32222484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 30 09:22:09.503713 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:09.503684 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8eada0cf-5884b8bb8-lp9q6" Oct 30 09:22:09.536057 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:09.536027 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79e02fb4-8a91-48ee-8fce-7fb16274008d-kube-api-access-6pjtn\" (UniqueName: \"kubernetes.io/projected/79e02fb4-8a91-48ee-8fce-7fb16274008d-kube-api-access-6pjtn\") pod \"79e02fb4-8a91-48ee-8fce-7fb16274008d\" (UID: \"79e02fb4-8a91-48ee-8fce-7fb16274008d\") " Oct 30 09:22:09.536224 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:09.536211 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79e02fb4-8a91-48ee-8fce-7fb16274008d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79e02fb4-8a91-48ee-8fce-7fb16274008d-plugins-config-volume\") pod \"79e02fb4-8a91-48ee-8fce-7fb16274008d\" (UID: \"79e02fb4-8a91-48ee-8fce-7fb16274008d\") " Oct 30 09:22:09.536550 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:09.536526 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79e02fb4-8a91-48ee-8fce-7fb16274008d-plugins-config-volume" pod "79e02fb4-8a91-48ee-8fce-7fb16274008d" (UID: "79e02fb4-8a91-48ee-8fce-7fb16274008d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:22:09.536717 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:09.536703 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79e02fb4-8a91-48ee-8fce-7fb16274008d-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:22:09.538140 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:09.538109 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79e02fb4-8a91-48ee-8fce-7fb16274008d-kube-api-access-6pjtn" pod "79e02fb4-8a91-48ee-8fce-7fb16274008d" (UID: "79e02fb4-8a91-48ee-8fce-7fb16274008d"). InnerVolumeSpecName "kube-api-access-6pjtn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:22:09.570129 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:09.570089 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8eada0cf-7c5b9fd4ff-9jzlk" Oct 30 09:22:09.637778 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:09.637696 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58bd2646-2220-4f53-9133-c27c4f8b0c15-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58bd2646-2220-4f53-9133-c27c4f8b0c15-plugins-config-volume\") pod \"58bd2646-2220-4f53-9133-c27c4f8b0c15\" (UID: \"58bd2646-2220-4f53-9133-c27c4f8b0c15\") " Oct 30 09:22:09.637944 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:09.637843 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58bd2646-2220-4f53-9133-c27c4f8b0c15-kube-api-access-4q8j5\" (UniqueName: \"kubernetes.io/projected/58bd2646-2220-4f53-9133-c27c4f8b0c15-kube-api-access-4q8j5\") pod \"58bd2646-2220-4f53-9133-c27c4f8b0c15\" (UID: \"58bd2646-2220-4f53-9133-c27c4f8b0c15\") " Oct 30 09:22:09.638119 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:09.638091 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58bd2646-2220-4f53-9133-c27c4f8b0c15-plugins-config-volume" pod "58bd2646-2220-4f53-9133-c27c4f8b0c15" (UID: "58bd2646-2220-4f53-9133-c27c4f8b0c15"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:22:09.648416 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:09.638503 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58bd2646-2220-4f53-9133-c27c4f8b0c15-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:22:09.648416 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:09.638542 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pjtn\" (UniqueName: \"kubernetes.io/projected/79e02fb4-8a91-48ee-8fce-7fb16274008d-kube-api-access-6pjtn\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:22:09.667406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:09.664133 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58bd2646-2220-4f53-9133-c27c4f8b0c15-kube-api-access-4q8j5" pod "58bd2646-2220-4f53-9133-c27c4f8b0c15" (UID: "58bd2646-2220-4f53-9133-c27c4f8b0c15"). InnerVolumeSpecName "kube-api-access-4q8j5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:22:09.763589 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:09.763548 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4q8j5\" (UniqueName: \"kubernetes.io/projected/58bd2646-2220-4f53-9133-c27c4f8b0c15-kube-api-access-4q8j5\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:22:09.897112 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:09.897037 4562 generic.go:356] "Generic (PLEG): container finished" podID="79e02fb4-8a91-48ee-8fce-7fb16274008d" containerID="7fc14de99612c11ff44a5329298c52a2cd6a0c00530c64ea0ca84993dd95275b" exitCode=0 Oct 30 09:22:09.897112 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:09.897102 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8eada0cf-5884b8bb8-lp9q6" Oct 30 09:22:09.897575 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:09.897136 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8eada0cf-5884b8bb8-lp9q6" event={"ID":"79e02fb4-8a91-48ee-8fce-7fb16274008d","Type":"ContainerDied","Data":"7fc14de99612c11ff44a5329298c52a2cd6a0c00530c64ea0ca84993dd95275b"} Oct 30 09:22:09.897575 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:09.897193 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8eada0cf-5884b8bb8-lp9q6" event={"ID":"79e02fb4-8a91-48ee-8fce-7fb16274008d","Type":"ContainerDied","Data":"2698839250a51127eac65e63f50c14c3bcd5f385cf1236dec0c70823a7247ee2"} Oct 30 09:22:09.897575 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:09.897210 4562 scope.go:122] "RemoveContainer" containerID="7fc14de99612c11ff44a5329298c52a2cd6a0c00530c64ea0ca84993dd95275b" Oct 30 09:22:09.898827 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:09.898800 4562 generic.go:356] "Generic (PLEG): container finished" podID="58bd2646-2220-4f53-9133-c27c4f8b0c15" containerID="bd03c52899d66ab056f94c1cd59c14d1a06f38c7fcf5f68d9a508d0ccb5611d3" exitCode=0 Oct 30 09:22:09.898958 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:09.898862 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8eada0cf-7c5b9fd4ff-9jzlk" Oct 30 09:22:09.898958 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:09.898898 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8eada0cf-7c5b9fd4ff-9jzlk" event={"ID":"58bd2646-2220-4f53-9133-c27c4f8b0c15","Type":"ContainerDied","Data":"bd03c52899d66ab056f94c1cd59c14d1a06f38c7fcf5f68d9a508d0ccb5611d3"} Oct 30 09:22:09.898958 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:09.898938 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8eada0cf-7c5b9fd4ff-9jzlk" event={"ID":"58bd2646-2220-4f53-9133-c27c4f8b0c15","Type":"ContainerDied","Data":"3531443a80f805583d0e7c52b21fe5567973a6cebabeecc435f8f363c0927bf9"} Oct 30 09:22:09.907070 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:09.907054 4562 scope.go:122] "RemoveContainer" containerID="7fc14de99612c11ff44a5329298c52a2cd6a0c00530c64ea0ca84993dd95275b" Oct 30 09:22:09.907340 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:22:09.907319 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fc14de99612c11ff44a5329298c52a2cd6a0c00530c64ea0ca84993dd95275b\": container with ID starting with 7fc14de99612c11ff44a5329298c52a2cd6a0c00530c64ea0ca84993dd95275b not found: ID does not exist" containerID="7fc14de99612c11ff44a5329298c52a2cd6a0c00530c64ea0ca84993dd95275b" Oct 30 09:22:09.907414 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:09.907347 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fc14de99612c11ff44a5329298c52a2cd6a0c00530c64ea0ca84993dd95275b"} err="failed to get container status \"7fc14de99612c11ff44a5329298c52a2cd6a0c00530c64ea0ca84993dd95275b\": rpc error: code = NotFound desc = could not find container \"7fc14de99612c11ff44a5329298c52a2cd6a0c00530c64ea0ca84993dd95275b\": container with ID starting with 7fc14de99612c11ff44a5329298c52a2cd6a0c00530c64ea0ca84993dd95275b not found: ID does not exist" Oct 30 09:22:09.907414 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:09.907362 4562 scope.go:122] "RemoveContainer" containerID="bd03c52899d66ab056f94c1cd59c14d1a06f38c7fcf5f68d9a508d0ccb5611d3" Oct 30 09:22:09.916046 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:09.916024 4562 scope.go:122] "RemoveContainer" containerID="bd03c52899d66ab056f94c1cd59c14d1a06f38c7fcf5f68d9a508d0ccb5611d3" Oct 30 09:22:09.916304 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:22:09.916287 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd03c52899d66ab056f94c1cd59c14d1a06f38c7fcf5f68d9a508d0ccb5611d3\": container with ID starting with bd03c52899d66ab056f94c1cd59c14d1a06f38c7fcf5f68d9a508d0ccb5611d3 not found: ID does not exist" containerID="bd03c52899d66ab056f94c1cd59c14d1a06f38c7fcf5f68d9a508d0ccb5611d3" Oct 30 09:22:09.916356 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:09.916311 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd03c52899d66ab056f94c1cd59c14d1a06f38c7fcf5f68d9a508d0ccb5611d3"} err="failed to get container status \"bd03c52899d66ab056f94c1cd59c14d1a06f38c7fcf5f68d9a508d0ccb5611d3\": rpc error: code = NotFound desc = could not find container \"bd03c52899d66ab056f94c1cd59c14d1a06f38c7fcf5f68d9a508d0ccb5611d3\": container with ID starting with bd03c52899d66ab056f94c1cd59c14d1a06f38c7fcf5f68d9a508d0ccb5611d3 not found: ID does not exist" Oct 30 09:22:09.969116 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:09.963537 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8eada0cf-7c5b9fd4ff-9jzlk"] Oct 30 09:22:09.974108 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:09.974078 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8eada0cf-7c5b9fd4ff-9jzlk"] Oct 30 09:22:10.002735 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:10.002708 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8eada0cf-5884b8bb8-lp9q6"] Oct 30 09:22:10.005870 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:10.005847 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8eada0cf-5884b8bb8-lp9q6"] Oct 30 09:22:10.059756 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:10.059722 4562 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-k7rc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 30 09:22:10.060011 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:10.059979 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k7rc9" podUID="866a5fb8-5939-4f6a-b01f-060e32222484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 30 09:22:11.051876 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:11.051760 4562 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-k7rc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 30 09:22:11.051876 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:11.051832 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k7rc9" podUID="866a5fb8-5939-4f6a-b01f-060e32222484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 30 09:22:11.894984 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:11.894942 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58bd2646-2220-4f53-9133-c27c4f8b0c15" path="/var/lib/kubelet/pods/58bd2646-2220-4f53-9133-c27c4f8b0c15/volumes" Oct 30 09:22:11.898403 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:11.898355 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79e02fb4-8a91-48ee-8fce-7fb16274008d" path="/var/lib/kubelet/pods/79e02fb4-8a91-48ee-8fce-7fb16274008d/volumes" Oct 30 09:22:12.058502 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:12.058463 4562 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-k7rc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 30 09:22:12.059094 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:12.059057 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k7rc9" podUID="866a5fb8-5939-4f6a-b01f-060e32222484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 30 09:22:12.179402 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:12.179302 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn"] Oct 30 09:22:12.181608 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:12.181567 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn" podUID="357b42b1-a9e1-423c-96b4-3bb155fed864" containerName="istio-proxy" containerID="cri-o://0afd5c4c97f378b7d68b9d98a65566cb37d1256b31cc99d51875c95f6ff2dcef" gracePeriod=30 Oct 30 09:22:12.368478 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:12.368438 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8de308-64d7c7bc75-p4xhw"] Oct 30 09:22:12.369448 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:12.369420 4562 cpu_manager.go:414] "containerMap: removing container" podUID="58bd2646-2220-4f53-9133-c27c4f8b0c15" containerName="payload-pre-processing" Oct 30 09:22:12.369448 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:12.369446 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58bd2646-2220-4f53-9133-c27c4f8b0c15" containerName="payload-pre-processing" Oct 30 09:22:12.369644 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:12.369500 4562 cpu_manager.go:414] "containerMap: removing container" podUID="79e02fb4-8a91-48ee-8fce-7fb16274008d" containerName="payload-processing" Oct 30 09:22:12.369644 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:12.369508 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79e02fb4-8a91-48ee-8fce-7fb16274008d" containerName="payload-processing" Oct 30 09:22:12.369746 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:12.369682 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="58bd2646-2220-4f53-9133-c27c4f8b0c15" containerName="payload-pre-processing" Oct 30 09:22:12.369746 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:12.369705 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="79e02fb4-8a91-48ee-8fce-7fb16274008d" containerName="payload-processing" Oct 30 09:22:12.374906 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:12.374875 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8de308-64d7c7bc75-p4xhw" Oct 30 09:22:12.376262 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:12.376238 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8de308-64d7c7bc75-p4xhw"] Oct 30 09:22:12.390924 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:12.390904 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-8de308-dockercfg-5p2cn\"" Oct 30 09:22:12.391068 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:12.391045 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-8de308\"" Oct 30 09:22:12.526407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:12.525208 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a82bec3-3dbf-4c3b-87a7-90043b88d127-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8de308-64d7c7bc75-p4xhw\" (UID: \"0a82bec3-3dbf-4c3b-87a7-90043b88d127\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8de308-64d7c7bc75-p4xhw" Oct 30 09:22:12.526407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:12.525316 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj24l\" (UniqueName: \"kubernetes.io/projected/0a82bec3-3dbf-4c3b-87a7-90043b88d127-kube-api-access-tj24l\") pod \"payload-pre-processing-e2e-ipp-a-8de308-64d7c7bc75-p4xhw\" (UID: \"0a82bec3-3dbf-4c3b-87a7-90043b88d127\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8de308-64d7c7bc75-p4xhw" Oct 30 09:22:12.671412 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:12.601454 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8de308-7699bcfb84-hctsq"] Oct 30 09:22:12.717404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:12.715146 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj24l\" (UniqueName: \"kubernetes.io/projected/0a82bec3-3dbf-4c3b-87a7-90043b88d127-kube-api-access-tj24l\") pod \"payload-pre-processing-e2e-ipp-a-8de308-64d7c7bc75-p4xhw\" (UID: \"0a82bec3-3dbf-4c3b-87a7-90043b88d127\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8de308-64d7c7bc75-p4xhw" Oct 30 09:22:12.717404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:12.716003 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a82bec3-3dbf-4c3b-87a7-90043b88d127-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8de308-64d7c7bc75-p4xhw\" (UID: \"0a82bec3-3dbf-4c3b-87a7-90043b88d127\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8de308-64d7c7bc75-p4xhw" Oct 30 09:22:12.719889 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:12.718042 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a82bec3-3dbf-4c3b-87a7-90043b88d127-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8de308-64d7c7bc75-p4xhw\" (UID: \"0a82bec3-3dbf-4c3b-87a7-90043b88d127\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8de308-64d7c7bc75-p4xhw" Oct 30 09:22:12.835485 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:12.819673 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8de308-7699bcfb84-hctsq"] Oct 30 09:22:12.835485 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:12.819860 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8de308-7699bcfb84-hctsq" Oct 30 09:22:12.839860 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:12.839809 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj24l\" (UniqueName: \"kubernetes.io/projected/0a82bec3-3dbf-4c3b-87a7-90043b88d127-kube-api-access-tj24l\") pod \"payload-pre-processing-e2e-ipp-a-8de308-64d7c7bc75-p4xhw\" (UID: \"0a82bec3-3dbf-4c3b-87a7-90043b88d127\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8de308-64d7c7bc75-p4xhw" Oct 30 09:22:12.949984 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:12.949045 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/156b5de0-6a01-4c16-97a8-09c5640def4d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8de308-7699bcfb84-hctsq\" (UID: \"156b5de0-6a01-4c16-97a8-09c5640def4d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8de308-7699bcfb84-hctsq" Oct 30 09:22:12.949984 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:12.949278 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsh8s\" (UniqueName: \"kubernetes.io/projected/156b5de0-6a01-4c16-97a8-09c5640def4d-kube-api-access-dsh8s\") pod \"payload-processing-e2e-ipp-a-8de308-7699bcfb84-hctsq\" (UID: \"156b5de0-6a01-4c16-97a8-09c5640def4d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8de308-7699bcfb84-hctsq" Oct 30 09:22:12.990814 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:12.990781 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8de308-64d7c7bc75-p4xhw" Oct 30 09:22:13.073570 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:13.073496 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/156b5de0-6a01-4c16-97a8-09c5640def4d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8de308-7699bcfb84-hctsq\" (UID: \"156b5de0-6a01-4c16-97a8-09c5640def4d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8de308-7699bcfb84-hctsq" Oct 30 09:22:13.073967 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:13.073719 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsh8s\" (UniqueName: \"kubernetes.io/projected/156b5de0-6a01-4c16-97a8-09c5640def4d-kube-api-access-dsh8s\") pod \"payload-processing-e2e-ipp-a-8de308-7699bcfb84-hctsq\" (UID: \"156b5de0-6a01-4c16-97a8-09c5640def4d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8de308-7699bcfb84-hctsq" Oct 30 09:22:13.074957 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:13.074928 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/156b5de0-6a01-4c16-97a8-09c5640def4d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8de308-7699bcfb84-hctsq\" (UID: \"156b5de0-6a01-4c16-97a8-09c5640def4d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8de308-7699bcfb84-hctsq" Oct 30 09:22:13.075284 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:13.075200 4562 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-k7rc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 30 09:22:13.075284 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:13.075245 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k7rc9" podUID="866a5fb8-5939-4f6a-b01f-060e32222484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 30 09:22:13.106133 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:13.106048 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsh8s\" (UniqueName: \"kubernetes.io/projected/156b5de0-6a01-4c16-97a8-09c5640def4d-kube-api-access-dsh8s\") pod \"payload-processing-e2e-ipp-a-8de308-7699bcfb84-hctsq\" (UID: \"156b5de0-6a01-4c16-97a8-09c5640def4d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8de308-7699bcfb84-hctsq" Oct 30 09:22:13.223857 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:13.223826 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8de308-7699bcfb84-hctsq" Oct 30 09:22:13.320052 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:13.319181 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8de308-64d7c7bc75-p4xhw"] Oct 30 09:22:13.321572 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:22:13.321481 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a82bec3_3dbf_4c3b_87a7_90043b88d127.slice/crio-f5484c6f56247db5ac9ce91f9426d78dfb0097bb4036a9dddfc921060a60ed3a WatchSource:0}: Error finding container f5484c6f56247db5ac9ce91f9426d78dfb0097bb4036a9dddfc921060a60ed3a: Status 404 returned error can't find the container with id f5484c6f56247db5ac9ce91f9426d78dfb0097bb4036a9dddfc921060a60ed3a Oct 30 09:22:13.549712 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:22:13.549685 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod156b5de0_6a01_4c16_97a8_09c5640def4d.slice/crio-5731555b0ad3427ca19947628057e054947a012fc1d08a472ae74c83d15a12be WatchSource:0}: Error finding container 5731555b0ad3427ca19947628057e054947a012fc1d08a472ae74c83d15a12be: Status 404 returned error can't find the container with id 5731555b0ad3427ca19947628057e054947a012fc1d08a472ae74c83d15a12be Oct 30 09:22:13.549712 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:13.549708 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8de308-7699bcfb84-hctsq"] Oct 30 09:22:13.577436 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:13.577367 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc"] Oct 30 09:22:13.584286 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:13.584261 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc" Oct 30 09:22:13.607665 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:13.607646 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc"] Oct 30 09:22:13.649120 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:13.648868 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-c44644-openshift-default-dockercfg-n92rb\"" Oct 30 09:22:13.686860 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:13.686828 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a104dc46-cd60-483a-b159-4512b460125a-workload-certs\") pod \"e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc\" (UID: \"a104dc46-cd60-483a-b159-4512b460125a\") " pod="openshift-ingress/e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc" Oct 30 09:22:13.687018 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:13.686939 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a104dc46-cd60-483a-b159-4512b460125a-istio-data\") pod \"e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc\" (UID: \"a104dc46-cd60-483a-b159-4512b460125a\") " pod="openshift-ingress/e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc" Oct 30 09:22:13.687077 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:13.687034 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a104dc46-cd60-483a-b159-4512b460125a-credential-socket\") pod \"e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc\" (UID: \"a104dc46-cd60-483a-b159-4512b460125a\") " pod="openshift-ingress/e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc" Oct 30 09:22:13.687153 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:13.687139 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8drg\" (UniqueName: \"kubernetes.io/projected/a104dc46-cd60-483a-b159-4512b460125a-kube-api-access-b8drg\") pod \"e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc\" (UID: \"a104dc46-cd60-483a-b159-4512b460125a\") " pod="openshift-ingress/e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc" Oct 30 09:22:13.687247 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:13.687234 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a104dc46-cd60-483a-b159-4512b460125a-istiod-ca-cert\") pod \"e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc\" (UID: \"a104dc46-cd60-483a-b159-4512b460125a\") " pod="openshift-ingress/e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc" Oct 30 09:22:13.687327 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:13.687277 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a104dc46-cd60-483a-b159-4512b460125a-istio-podinfo\") pod \"e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc\" (UID: \"a104dc46-cd60-483a-b159-4512b460125a\") " pod="openshift-ingress/e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc" Oct 30 09:22:13.687398 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:13.687350 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a104dc46-cd60-483a-b159-4512b460125a-istio-envoy\") pod \"e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc\" (UID: \"a104dc46-cd60-483a-b159-4512b460125a\") " pod="openshift-ingress/e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc" Oct 30 09:22:13.687488 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:13.687466 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a104dc46-cd60-483a-b159-4512b460125a-workload-socket\") pod \"e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc\" (UID: \"a104dc46-cd60-483a-b159-4512b460125a\") " pod="openshift-ingress/e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc" Oct 30 09:22:13.687539 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:13.687506 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a104dc46-cd60-483a-b159-4512b460125a-istio-token\") pod \"e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc\" (UID: \"a104dc46-cd60-483a-b159-4512b460125a\") " pod="openshift-ingress/e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc" Oct 30 09:22:13.799403 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:13.789254 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a104dc46-cd60-483a-b159-4512b460125a-istio-data\") pod \"e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc\" (UID: \"a104dc46-cd60-483a-b159-4512b460125a\") " pod="openshift-ingress/e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc" Oct 30 09:22:13.799403 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:13.789329 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a104dc46-cd60-483a-b159-4512b460125a-credential-socket\") pod \"e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc\" (UID: \"a104dc46-cd60-483a-b159-4512b460125a\") " pod="openshift-ingress/e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc" Oct 30 09:22:13.799403 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:13.789450 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8drg\" (UniqueName: \"kubernetes.io/projected/a104dc46-cd60-483a-b159-4512b460125a-kube-api-access-b8drg\") pod \"e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc\" (UID: \"a104dc46-cd60-483a-b159-4512b460125a\") " pod="openshift-ingress/e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc" Oct 30 09:22:13.799403 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:13.789567 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a104dc46-cd60-483a-b159-4512b460125a-istiod-ca-cert\") pod \"e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc\" (UID: \"a104dc46-cd60-483a-b159-4512b460125a\") " pod="openshift-ingress/e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc" Oct 30 09:22:13.799403 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:13.789629 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a104dc46-cd60-483a-b159-4512b460125a-istio-podinfo\") pod \"e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc\" (UID: \"a104dc46-cd60-483a-b159-4512b460125a\") " pod="openshift-ingress/e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc" Oct 30 09:22:13.799403 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:13.789663 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a104dc46-cd60-483a-b159-4512b460125a-istio-envoy\") pod \"e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc\" (UID: \"a104dc46-cd60-483a-b159-4512b460125a\") " pod="openshift-ingress/e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc" Oct 30 09:22:13.799403 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:13.789719 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a104dc46-cd60-483a-b159-4512b460125a-workload-socket\") pod \"e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc\" (UID: \"a104dc46-cd60-483a-b159-4512b460125a\") " pod="openshift-ingress/e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc" Oct 30 09:22:13.799403 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:13.789753 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a104dc46-cd60-483a-b159-4512b460125a-istio-token\") pod \"e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc\" (UID: \"a104dc46-cd60-483a-b159-4512b460125a\") " pod="openshift-ingress/e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc" Oct 30 09:22:13.799403 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:13.789785 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a104dc46-cd60-483a-b159-4512b460125a-workload-certs\") pod \"e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc\" (UID: \"a104dc46-cd60-483a-b159-4512b460125a\") " pod="openshift-ingress/e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc" Oct 30 09:22:13.799403 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:13.790286 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a104dc46-cd60-483a-b159-4512b460125a-workload-certs\") pod \"e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc\" (UID: \"a104dc46-cd60-483a-b159-4512b460125a\") " pod="openshift-ingress/e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc" Oct 30 09:22:13.799403 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:13.790559 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a104dc46-cd60-483a-b159-4512b460125a-istio-data\") pod \"e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc\" (UID: \"a104dc46-cd60-483a-b159-4512b460125a\") " pod="openshift-ingress/e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc" Oct 30 09:22:13.799403 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:13.790807 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a104dc46-cd60-483a-b159-4512b460125a-credential-socket\") pod \"e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc\" (UID: \"a104dc46-cd60-483a-b159-4512b460125a\") " pod="openshift-ingress/e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc" Oct 30 09:22:13.799403 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:13.791745 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a104dc46-cd60-483a-b159-4512b460125a-istiod-ca-cert\") pod \"e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc\" (UID: \"a104dc46-cd60-483a-b159-4512b460125a\") " pod="openshift-ingress/e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc" Oct 30 09:22:13.799403 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:13.792679 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a104dc46-cd60-483a-b159-4512b460125a-workload-socket\") pod \"e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc\" (UID: \"a104dc46-cd60-483a-b159-4512b460125a\") " pod="openshift-ingress/e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc" Oct 30 09:22:13.817401 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:13.811803 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a104dc46-cd60-483a-b159-4512b460125a-istio-envoy\") pod \"e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc\" (UID: \"a104dc46-cd60-483a-b159-4512b460125a\") " pod="openshift-ingress/e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc" Oct 30 09:22:13.857030 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:13.841164 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8drg\" (UniqueName: \"kubernetes.io/projected/a104dc46-cd60-483a-b159-4512b460125a-kube-api-access-b8drg\") pod \"e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc\" (UID: \"a104dc46-cd60-483a-b159-4512b460125a\") " pod="openshift-ingress/e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc" Oct 30 09:22:13.857030 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:13.850896 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a104dc46-cd60-483a-b159-4512b460125a-istio-podinfo\") pod \"e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc\" (UID: \"a104dc46-cd60-483a-b159-4512b460125a\") " pod="openshift-ingress/e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc" Oct 30 09:22:13.857030 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:13.853477 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a104dc46-cd60-483a-b159-4512b460125a-istio-token\") pod \"e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc\" (UID: \"a104dc46-cd60-483a-b159-4512b460125a\") " pod="openshift-ingress/e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc" Oct 30 09:22:13.921466 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:13.919728 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc" Oct 30 09:22:14.045258 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:14.044686 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8de308-64d7c7bc75-p4xhw" event={"ID":"0a82bec3-3dbf-4c3b-87a7-90043b88d127","Type":"ContainerStarted","Data":"0bf7eb55e4e0a1bddc1f8b9a38bad9794960c47517a59891c27b88cdf9a61449"} Oct 30 09:22:14.045258 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:14.044754 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8de308-64d7c7bc75-p4xhw" event={"ID":"0a82bec3-3dbf-4c3b-87a7-90043b88d127","Type":"ContainerStarted","Data":"f5484c6f56247db5ac9ce91f9426d78dfb0097bb4036a9dddfc921060a60ed3a"} Oct 30 09:22:14.046731 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:14.045825 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8de308-64d7c7bc75-p4xhw" Oct 30 09:22:14.060399 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:14.048226 4562 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-k7rc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 30 09:22:14.060399 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:14.048275 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k7rc9" podUID="866a5fb8-5939-4f6a-b01f-060e32222484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 30 09:22:14.060399 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:14.050434 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8de308-7699bcfb84-hctsq" event={"ID":"156b5de0-6a01-4c16-97a8-09c5640def4d","Type":"ContainerStarted","Data":"536de35bf7c6b7e64d224dc5882d526b69001778cc10c8952a1d3ef55694e9ba"} Oct 30 09:22:14.060399 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:14.050481 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8de308-7699bcfb84-hctsq" event={"ID":"156b5de0-6a01-4c16-97a8-09c5640def4d","Type":"ContainerStarted","Data":"5731555b0ad3427ca19947628057e054947a012fc1d08a472ae74c83d15a12be"} Oct 30 09:22:14.060399 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:14.051224 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-8de308-7699bcfb84-hctsq" Oct 30 09:22:14.093605 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:14.093533 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8de308-64d7c7bc75-p4xhw" podStartSLOduration=2.093514762 podStartE2EDuration="2.093514762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:22:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:22:14.092624519 +0000 UTC m=+1694.354185524" watchObservedRunningTime="2026-10-30 09:22:14.093514762 +0000 UTC m=+1694.355075751" Oct 30 09:22:14.137619 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:14.137541 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-8de308-7699bcfb84-hctsq" podStartSLOduration=2.13752219 podStartE2EDuration="2.13752219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:22:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:22:14.134412232 +0000 UTC m=+1694.395973245" watchObservedRunningTime="2026-10-30 09:22:14.13752219 +0000 UTC m=+1694.399083179" Oct 30 09:22:14.376871 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:22:14.376825 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda104dc46_cd60_483a_b159_4512b460125a.slice/crio-2020d4f08ef7b99348a0e7d8a93669c404b4f2b0c0ef7ec48a9046f1c2fe683f WatchSource:0}: Error finding container 2020d4f08ef7b99348a0e7d8a93669c404b4f2b0c0ef7ec48a9046f1c2fe683f: Status 404 returned error can't find the container with id 2020d4f08ef7b99348a0e7d8a93669c404b4f2b0c0ef7ec48a9046f1c2fe683f Oct 30 09:22:14.378003 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:14.377980 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc"] Oct 30 09:22:14.379496 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:14.379465 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:22:14.379562 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:14.379524 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:22:14.379562 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:14.379553 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:22:15.048416 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:15.048353 4562 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-k7rc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 30 09:22:15.048587 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:15.048440 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k7rc9" podUID="866a5fb8-5939-4f6a-b01f-060e32222484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 30 09:22:15.059628 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:15.059596 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc" event={"ID":"a104dc46-cd60-483a-b159-4512b460125a","Type":"ContainerStarted","Data":"3b6eb58a9768a11f8e18b1b69dc60bc1d2ac09c48dff91ca04ad0667be45f33a"} Oct 30 09:22:15.059786 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:15.059658 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc" event={"ID":"a104dc46-cd60-483a-b159-4512b460125a","Type":"ContainerStarted","Data":"2020d4f08ef7b99348a0e7d8a93669c404b4f2b0c0ef7ec48a9046f1c2fe683f"} Oct 30 09:22:15.107546 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:15.107496 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc" podStartSLOduration=2.107478437 podStartE2EDuration="2.107478437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:22:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:22:15.105910578 +0000 UTC m=+1695.367471585" watchObservedRunningTime="2026-10-30 09:22:15.107478437 +0000 UTC m=+1695.369039428" Oct 30 09:22:15.920754 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:15.920718 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc" Oct 30 09:22:15.926642 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:15.926613 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc" Oct 30 09:22:16.077265 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:16.077212 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc" Oct 30 09:22:16.078634 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:16.078581 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc" Oct 30 09:22:16.375163 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:16.375130 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8eada0cf-7c8545fd75-cfnvf"] Oct 30 09:22:16.379376 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:16.379356 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eada0cf-7c8545fd75-cfnvf" Oct 30 09:22:16.415945 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:16.415918 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8eada0cf-7c8545fd75-cfnvf"] Oct 30 09:22:16.430291 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:16.430247 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-8eada0cf-serving-cert\"" Oct 30 09:22:16.481329 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:16.481286 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73717db6-2d44-4474-93f1-df1a19589a47-maas-api-tls\") pod \"maas-api-e2e-mt-8eada0cf-7c8545fd75-cfnvf\" (UID: \"73717db6-2d44-4474-93f1-df1a19589a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eada0cf-7c8545fd75-cfnvf" Oct 30 09:22:16.481584 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:16.481563 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73717db6-2d44-4474-93f1-df1a19589a47-metrics-certs\") pod \"maas-api-e2e-mt-8eada0cf-7c8545fd75-cfnvf\" (UID: \"73717db6-2d44-4474-93f1-df1a19589a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eada0cf-7c8545fd75-cfnvf" Oct 30 09:22:16.481937 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:16.481916 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb8gp\" (UniqueName: \"kubernetes.io/projected/73717db6-2d44-4474-93f1-df1a19589a47-kube-api-access-kb8gp\") pod \"maas-api-e2e-mt-8eada0cf-7c8545fd75-cfnvf\" (UID: \"73717db6-2d44-4474-93f1-df1a19589a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eada0cf-7c8545fd75-cfnvf" Oct 30 09:22:16.482167 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:16.482127 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73717db6-2d44-4474-93f1-df1a19589a47-openshift-service-ca\") pod \"maas-api-e2e-mt-8eada0cf-7c8545fd75-cfnvf\" (UID: \"73717db6-2d44-4474-93f1-df1a19589a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eada0cf-7c8545fd75-cfnvf" Oct 30 09:22:16.493654 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:16.493632 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k7rc9" Oct 30 09:22:16.495700 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:16.495672 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k7rc9" Oct 30 09:22:16.498531 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:16.498506 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k7rc9" Oct 30 09:22:16.584228 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:16.584185 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb8gp\" (UniqueName: \"kubernetes.io/projected/73717db6-2d44-4474-93f1-df1a19589a47-kube-api-access-kb8gp\") pod \"maas-api-e2e-mt-8eada0cf-7c8545fd75-cfnvf\" (UID: \"73717db6-2d44-4474-93f1-df1a19589a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eada0cf-7c8545fd75-cfnvf" Oct 30 09:22:16.584401 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:16.584248 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73717db6-2d44-4474-93f1-df1a19589a47-openshift-service-ca\") pod \"maas-api-e2e-mt-8eada0cf-7c8545fd75-cfnvf\" (UID: \"73717db6-2d44-4474-93f1-df1a19589a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eada0cf-7c8545fd75-cfnvf" Oct 30 09:22:16.584526 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:16.584504 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73717db6-2d44-4474-93f1-df1a19589a47-maas-api-tls\") pod \"maas-api-e2e-mt-8eada0cf-7c8545fd75-cfnvf\" (UID: \"73717db6-2d44-4474-93f1-df1a19589a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eada0cf-7c8545fd75-cfnvf" Oct 30 09:22:16.584585 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:16.584548 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73717db6-2d44-4474-93f1-df1a19589a47-metrics-certs\") pod \"maas-api-e2e-mt-8eada0cf-7c8545fd75-cfnvf\" (UID: \"73717db6-2d44-4474-93f1-df1a19589a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eada0cf-7c8545fd75-cfnvf" Oct 30 09:22:16.586856 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:16.586812 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73717db6-2d44-4474-93f1-df1a19589a47-openshift-service-ca\") pod \"maas-api-e2e-mt-8eada0cf-7c8545fd75-cfnvf\" (UID: \"73717db6-2d44-4474-93f1-df1a19589a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eada0cf-7c8545fd75-cfnvf" Oct 30 09:22:16.587234 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:22:16.587176 4562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-8eada0cf-serving-cert: secret "maas-api-e2e-mt-8eada0cf-serving-cert" not found Oct 30 09:22:16.587342 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:22:16.587330 4562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73717db6-2d44-4474-93f1-df1a19589a47-maas-api-tls podName:73717db6-2d44-4474-93f1-df1a19589a47 nodeName:}" failed. No retries permitted until 2026-10-30 09:22:17.087310928 +0000 UTC m=+1697.348871900 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/73717db6-2d44-4474-93f1-df1a19589a47-maas-api-tls") pod "maas-api-e2e-mt-8eada0cf-7c8545fd75-cfnvf" (UID: "73717db6-2d44-4474-93f1-df1a19589a47") : secret "maas-api-e2e-mt-8eada0cf-serving-cert" not found Oct 30 09:22:16.599177 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:16.599151 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73717db6-2d44-4474-93f1-df1a19589a47-metrics-certs\") pod \"maas-api-e2e-mt-8eada0cf-7c8545fd75-cfnvf\" (UID: \"73717db6-2d44-4474-93f1-df1a19589a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eada0cf-7c8545fd75-cfnvf" Oct 30 09:22:16.627138 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:16.627063 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb8gp\" (UniqueName: \"kubernetes.io/projected/73717db6-2d44-4474-93f1-df1a19589a47-kube-api-access-kb8gp\") pod \"maas-api-e2e-mt-8eada0cf-7c8545fd75-cfnvf\" (UID: \"73717db6-2d44-4474-93f1-df1a19589a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eada0cf-7c8545fd75-cfnvf" Oct 30 09:22:17.147404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:17.131147 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73717db6-2d44-4474-93f1-df1a19589a47-maas-api-tls\") pod \"maas-api-e2e-mt-8eada0cf-7c8545fd75-cfnvf\" (UID: \"73717db6-2d44-4474-93f1-df1a19589a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eada0cf-7c8545fd75-cfnvf" Oct 30 09:22:17.180953 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:17.180904 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73717db6-2d44-4474-93f1-df1a19589a47-maas-api-tls\") pod \"maas-api-e2e-mt-8eada0cf-7c8545fd75-cfnvf\" (UID: \"73717db6-2d44-4474-93f1-df1a19589a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eada0cf-7c8545fd75-cfnvf" Oct 30 09:22:17.296570 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:17.296510 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eada0cf-7c8545fd75-cfnvf" Oct 30 09:22:17.591123 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:17.591101 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn" Oct 30 09:22:17.670991 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:22:17.669045 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73717db6_2d44_4474_93f1_df1a19589a47.slice/crio-ec55dd15c4178ab7e111089e1354622c79907ae2fac8f91ad9af80e8a2979c0b WatchSource:0}: Error finding container ec55dd15c4178ab7e111089e1354622c79907ae2fac8f91ad9af80e8a2979c0b: Status 404 returned error can't find the container with id ec55dd15c4178ab7e111089e1354622c79907ae2fac8f91ad9af80e8a2979c0b Oct 30 09:22:17.709416 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:17.678933 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8eada0cf-7c8545fd75-cfnvf"] Oct 30 09:22:17.778334 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:17.764838 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/357b42b1-a9e1-423c-96b4-3bb155fed864-istio-token\" (UniqueName: \"kubernetes.io/projected/357b42b1-a9e1-423c-96b4-3bb155fed864-istio-token\") pod \"357b42b1-a9e1-423c-96b4-3bb155fed864\" (UID: \"357b42b1-a9e1-423c-96b4-3bb155fed864\") " Oct 30 09:22:17.778334 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:17.765001 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/357b42b1-a9e1-423c-96b4-3bb155fed864-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/357b42b1-a9e1-423c-96b4-3bb155fed864-workload-certs\") pod \"357b42b1-a9e1-423c-96b4-3bb155fed864\" (UID: \"357b42b1-a9e1-423c-96b4-3bb155fed864\") " Oct 30 09:22:17.778334 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:17.765044 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/357b42b1-a9e1-423c-96b4-3bb155fed864-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/357b42b1-a9e1-423c-96b4-3bb155fed864-istio-envoy\") pod \"357b42b1-a9e1-423c-96b4-3bb155fed864\" (UID: \"357b42b1-a9e1-423c-96b4-3bb155fed864\") " Oct 30 09:22:17.778334 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:17.765076 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/357b42b1-a9e1-423c-96b4-3bb155fed864-kube-api-access-qgbvb\" (UniqueName: \"kubernetes.io/projected/357b42b1-a9e1-423c-96b4-3bb155fed864-kube-api-access-qgbvb\") pod \"357b42b1-a9e1-423c-96b4-3bb155fed864\" (UID: \"357b42b1-a9e1-423c-96b4-3bb155fed864\") " Oct 30 09:22:17.778334 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:17.765139 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/357b42b1-a9e1-423c-96b4-3bb155fed864-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/357b42b1-a9e1-423c-96b4-3bb155fed864-istiod-ca-cert\") pod \"357b42b1-a9e1-423c-96b4-3bb155fed864\" (UID: \"357b42b1-a9e1-423c-96b4-3bb155fed864\") " Oct 30 09:22:17.778334 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:17.765184 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/357b42b1-a9e1-423c-96b4-3bb155fed864-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/357b42b1-a9e1-423c-96b4-3bb155fed864-istio-data\") pod \"357b42b1-a9e1-423c-96b4-3bb155fed864\" (UID: \"357b42b1-a9e1-423c-96b4-3bb155fed864\") " Oct 30 09:22:17.778334 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:17.765285 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/357b42b1-a9e1-423c-96b4-3bb155fed864-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/357b42b1-a9e1-423c-96b4-3bb155fed864-credential-socket\") pod \"357b42b1-a9e1-423c-96b4-3bb155fed864\" (UID: \"357b42b1-a9e1-423c-96b4-3bb155fed864\") " Oct 30 09:22:17.778334 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:17.765462 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/357b42b1-a9e1-423c-96b4-3bb155fed864-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/357b42b1-a9e1-423c-96b4-3bb155fed864-istio-podinfo\") pod \"357b42b1-a9e1-423c-96b4-3bb155fed864\" (UID: \"357b42b1-a9e1-423c-96b4-3bb155fed864\") " Oct 30 09:22:17.778334 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:17.765504 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/357b42b1-a9e1-423c-96b4-3bb155fed864-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/357b42b1-a9e1-423c-96b4-3bb155fed864-workload-socket\") pod \"357b42b1-a9e1-423c-96b4-3bb155fed864\" (UID: \"357b42b1-a9e1-423c-96b4-3bb155fed864\") " Oct 30 09:22:17.803404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:17.779058 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/357b42b1-a9e1-423c-96b4-3bb155fed864-workload-socket" pod "357b42b1-a9e1-423c-96b4-3bb155fed864" (UID: "357b42b1-a9e1-423c-96b4-3bb155fed864"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:22:17.803404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:17.796845 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/357b42b1-a9e1-423c-96b4-3bb155fed864-istio-envoy" pod "357b42b1-a9e1-423c-96b4-3bb155fed864" (UID: "357b42b1-a9e1-423c-96b4-3bb155fed864"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:22:17.838809 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:17.820940 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/357b42b1-a9e1-423c-96b4-3bb155fed864-istiod-ca-cert" pod "357b42b1-a9e1-423c-96b4-3bb155fed864" (UID: "357b42b1-a9e1-423c-96b4-3bb155fed864"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:22:17.838809 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:17.825737 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/357b42b1-a9e1-423c-96b4-3bb155fed864-workload-certs" pod "357b42b1-a9e1-423c-96b4-3bb155fed864" (UID: "357b42b1-a9e1-423c-96b4-3bb155fed864"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:22:17.838809 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:17.831508 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/357b42b1-a9e1-423c-96b4-3bb155fed864-credential-socket" pod "357b42b1-a9e1-423c-96b4-3bb155fed864" (UID: "357b42b1-a9e1-423c-96b4-3bb155fed864"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:22:17.838809 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:17.834523 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/357b42b1-a9e1-423c-96b4-3bb155fed864-istio-data" pod "357b42b1-a9e1-423c-96b4-3bb155fed864" (UID: "357b42b1-a9e1-423c-96b4-3bb155fed864"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:22:17.952416 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:17.950861 4562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/357b42b1-a9e1-423c-96b4-3bb155fed864-workload-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:22:17.952416 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:17.951867 4562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/357b42b1-a9e1-423c-96b4-3bb155fed864-workload-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:22:17.952416 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:17.951888 4562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/357b42b1-a9e1-423c-96b4-3bb155fed864-istio-envoy\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:22:17.952416 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:17.951901 4562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/357b42b1-a9e1-423c-96b4-3bb155fed864-istiod-ca-cert\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:22:17.952416 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:17.951913 4562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/357b42b1-a9e1-423c-96b4-3bb155fed864-istio-data\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:22:17.952416 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:17.951924 4562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/357b42b1-a9e1-423c-96b4-3bb155fed864-credential-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:22:17.986650 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:17.973749 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/357b42b1-a9e1-423c-96b4-3bb155fed864-kube-api-access-qgbvb" pod "357b42b1-a9e1-423c-96b4-3bb155fed864" (UID: "357b42b1-a9e1-423c-96b4-3bb155fed864"). InnerVolumeSpecName "kube-api-access-qgbvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:22:18.043513 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:18.041049 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/357b42b1-a9e1-423c-96b4-3bb155fed864-istio-token" pod "357b42b1-a9e1-423c-96b4-3bb155fed864" (UID: "357b42b1-a9e1-423c-96b4-3bb155fed864"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:22:18.066407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:18.065360 4562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/357b42b1-a9e1-423c-96b4-3bb155fed864-istio-token\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:22:18.066407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:18.065411 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgbvb\" (UniqueName: \"kubernetes.io/projected/357b42b1-a9e1-423c-96b4-3bb155fed864-kube-api-access-qgbvb\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:22:18.120412 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:18.116499 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/357b42b1-a9e1-423c-96b4-3bb155fed864-istio-podinfo" pod "357b42b1-a9e1-423c-96b4-3bb155fed864" (UID: "357b42b1-a9e1-423c-96b4-3bb155fed864"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:22:18.204781 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:18.204157 4562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/357b42b1-a9e1-423c-96b4-3bb155fed864-istio-podinfo\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:22:18.265798 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:18.265761 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eada0cf-7c8545fd75-cfnvf" event={"ID":"73717db6-2d44-4474-93f1-df1a19589a47","Type":"ContainerStarted","Data":"ec55dd15c4178ab7e111089e1354622c79907ae2fac8f91ad9af80e8a2979c0b"} Oct 30 09:22:18.268160 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:18.268129 4562 generic.go:356] "Generic (PLEG): container finished" podID="357b42b1-a9e1-423c-96b4-3bb155fed864" containerID="0afd5c4c97f378b7d68b9d98a65566cb37d1256b31cc99d51875c95f6ff2dcef" exitCode=0 Oct 30 09:22:18.270187 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:18.270163 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn" Oct 30 09:22:18.270609 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:18.270582 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn" event={"ID":"357b42b1-a9e1-423c-96b4-3bb155fed864","Type":"ContainerDied","Data":"0afd5c4c97f378b7d68b9d98a65566cb37d1256b31cc99d51875c95f6ff2dcef"} Oct 30 09:22:18.270825 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:18.270798 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn" event={"ID":"357b42b1-a9e1-423c-96b4-3bb155fed864","Type":"ContainerDied","Data":"a2b66918eb8f378c094a17e2aebf673a0bb91e5a521f9a97511cac5a22419c0c"} Oct 30 09:22:18.270983 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:18.270969 4562 scope.go:122] "RemoveContainer" containerID="0afd5c4c97f378b7d68b9d98a65566cb37d1256b31cc99d51875c95f6ff2dcef" Oct 30 09:22:18.306597 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:18.303605 4562 scope.go:122] "RemoveContainer" containerID="774209cf60805d8c3fed9f3186e0ca6185e67bba1e26e9f85705c968d4c6b23c" Oct 30 09:22:18.320159 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:18.320136 4562 scope.go:122] "RemoveContainer" containerID="0afd5c4c97f378b7d68b9d98a65566cb37d1256b31cc99d51875c95f6ff2dcef" Oct 30 09:22:18.320660 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:22:18.320627 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0afd5c4c97f378b7d68b9d98a65566cb37d1256b31cc99d51875c95f6ff2dcef\": container with ID starting with 0afd5c4c97f378b7d68b9d98a65566cb37d1256b31cc99d51875c95f6ff2dcef not found: ID does not exist" containerID="0afd5c4c97f378b7d68b9d98a65566cb37d1256b31cc99d51875c95f6ff2dcef" Oct 30 09:22:18.320787 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:18.320663 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0afd5c4c97f378b7d68b9d98a65566cb37d1256b31cc99d51875c95f6ff2dcef"} err="failed to get container status \"0afd5c4c97f378b7d68b9d98a65566cb37d1256b31cc99d51875c95f6ff2dcef\": rpc error: code = NotFound desc = could not find container \"0afd5c4c97f378b7d68b9d98a65566cb37d1256b31cc99d51875c95f6ff2dcef\": container with ID starting with 0afd5c4c97f378b7d68b9d98a65566cb37d1256b31cc99d51875c95f6ff2dcef not found: ID does not exist" Oct 30 09:22:18.320787 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:18.320685 4562 scope.go:122] "RemoveContainer" containerID="774209cf60805d8c3fed9f3186e0ca6185e67bba1e26e9f85705c968d4c6b23c" Oct 30 09:22:18.320973 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:22:18.320941 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"774209cf60805d8c3fed9f3186e0ca6185e67bba1e26e9f85705c968d4c6b23c\": container with ID starting with 774209cf60805d8c3fed9f3186e0ca6185e67bba1e26e9f85705c968d4c6b23c not found: ID does not exist" containerID="774209cf60805d8c3fed9f3186e0ca6185e67bba1e26e9f85705c968d4c6b23c" Oct 30 09:22:18.321033 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:18.320978 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"774209cf60805d8c3fed9f3186e0ca6185e67bba1e26e9f85705c968d4c6b23c"} err="failed to get container status \"774209cf60805d8c3fed9f3186e0ca6185e67bba1e26e9f85705c968d4c6b23c\": rpc error: code = NotFound desc = could not find container \"774209cf60805d8c3fed9f3186e0ca6185e67bba1e26e9f85705c968d4c6b23c\": container with ID starting with 774209cf60805d8c3fed9f3186e0ca6185e67bba1e26e9f85705c968d4c6b23c not found: ID does not exist" Oct 30 09:22:18.347252 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:18.347228 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn"] Oct 30 09:22:18.361014 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:18.360707 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-6c9de5-openshift-default-5867c4b699-kb7wn"] Oct 30 09:22:19.317408 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:19.311438 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eada0cf-7c8545fd75-cfnvf" event={"ID":"73717db6-2d44-4474-93f1-df1a19589a47","Type":"ContainerStarted","Data":"916befdf2164830e39cb74928c6a8fcea85a03073cb0331c7d5ad6bdd45ea022"} Oct 30 09:22:19.317408 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:19.312561 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eada0cf-7c8545fd75-cfnvf" Oct 30 09:22:19.365406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:19.365223 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eada0cf-7c8545fd75-cfnvf" podStartSLOduration=2.716605163 podStartE2EDuration="3.365202619s" totalImagesPullingTime="648.597456ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:22:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:22:19.360671447 +0000 UTC m=+1699.622232460" watchObservedRunningTime="2026-10-30 09:22:19.365202619 +0000 UTC m=+1699.626763614" Oct 30 09:22:19.909283 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:19.909251 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="357b42b1-a9e1-423c-96b4-3bb155fed864" path="/var/lib/kubelet/pods/357b42b1-a9e1-423c-96b4-3bb155fed864/volumes" Oct 30 09:22:20.893321 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:20.893274 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c44644-7db5575f5d-qzlhc"] Oct 30 09:22:20.894004 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:20.893983 4562 cpu_manager.go:414] "containerMap: removing container" podUID="357b42b1-a9e1-423c-96b4-3bb155fed864" containerName="istio-proxy" Oct 30 09:22:20.894004 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:20.894003 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="357b42b1-a9e1-423c-96b4-3bb155fed864" containerName="istio-proxy" Oct 30 09:22:20.894235 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:20.894045 4562 cpu_manager.go:414] "containerMap: removing container" podUID="357b42b1-a9e1-423c-96b4-3bb155fed864" containerName="istio-proxy" Oct 30 09:22:20.894235 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:20.894053 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="357b42b1-a9e1-423c-96b4-3bb155fed864" containerName="istio-proxy" Oct 30 09:22:20.894235 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:20.894190 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="357b42b1-a9e1-423c-96b4-3bb155fed864" containerName="istio-proxy" Oct 30 09:22:20.894235 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:20.894215 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="357b42b1-a9e1-423c-96b4-3bb155fed864" containerName="istio-proxy" Oct 30 09:22:20.899785 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:20.899763 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c44644-7db5575f5d-qzlhc" Oct 30 09:22:20.915983 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:20.915949 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psvd5\" (UniqueName: \"kubernetes.io/projected/b8c2c2f2-d8ce-450b-9b00-bb8d96ac68bd-kube-api-access-psvd5\") pod \"payload-processing-e2e-ipp-a-c44644-7db5575f5d-qzlhc\" (UID: \"b8c2c2f2-d8ce-450b-9b00-bb8d96ac68bd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c44644-7db5575f5d-qzlhc" Oct 30 09:22:20.916181 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:20.916158 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8c2c2f2-d8ce-450b-9b00-bb8d96ac68bd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c44644-7db5575f5d-qzlhc\" (UID: \"b8c2c2f2-d8ce-450b-9b00-bb8d96ac68bd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c44644-7db5575f5d-qzlhc" Oct 30 09:22:20.916744 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:20.916717 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c44644-7db5575f5d-qzlhc"] Oct 30 09:22:20.921542 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:20.921521 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-c44644\"" Oct 30 09:22:20.921652 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:20.921599 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-c44644-dockercfg-fjrn7\"" Oct 30 09:22:20.971818 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:20.971784 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c44644-58d55657ff-tghrj"] Oct 30 09:22:20.977878 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:20.977847 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c44644-58d55657ff-tghrj" Oct 30 09:22:20.979795 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:20.979769 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c44644-58d55657ff-tghrj"] Oct 30 09:22:21.034235 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:21.034190 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psvd5\" (UniqueName: \"kubernetes.io/projected/b8c2c2f2-d8ce-450b-9b00-bb8d96ac68bd-kube-api-access-psvd5\") pod \"payload-processing-e2e-ipp-a-c44644-7db5575f5d-qzlhc\" (UID: \"b8c2c2f2-d8ce-450b-9b00-bb8d96ac68bd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c44644-7db5575f5d-qzlhc" Oct 30 09:22:21.034701 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:21.034649 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8c2c2f2-d8ce-450b-9b00-bb8d96ac68bd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c44644-7db5575f5d-qzlhc\" (UID: \"b8c2c2f2-d8ce-450b-9b00-bb8d96ac68bd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c44644-7db5575f5d-qzlhc" Oct 30 09:22:21.035966 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:21.035941 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8c2c2f2-d8ce-450b-9b00-bb8d96ac68bd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c44644-7db5575f5d-qzlhc\" (UID: \"b8c2c2f2-d8ce-450b-9b00-bb8d96ac68bd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c44644-7db5575f5d-qzlhc" Oct 30 09:22:21.179454 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:21.155795 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psvd5\" (UniqueName: \"kubernetes.io/projected/b8c2c2f2-d8ce-450b-9b00-bb8d96ac68bd-kube-api-access-psvd5\") pod \"payload-processing-e2e-ipp-a-c44644-7db5575f5d-qzlhc\" (UID: \"b8c2c2f2-d8ce-450b-9b00-bb8d96ac68bd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c44644-7db5575f5d-qzlhc" Oct 30 09:22:21.179454 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:21.155850 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/987a60a0-3b09-4fc8-aff6-2c5b47c5f225-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c44644-58d55657ff-tghrj\" (UID: \"987a60a0-3b09-4fc8-aff6-2c5b47c5f225\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c44644-58d55657ff-tghrj" Oct 30 09:22:21.179454 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:21.155891 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpf56\" (UniqueName: \"kubernetes.io/projected/987a60a0-3b09-4fc8-aff6-2c5b47c5f225-kube-api-access-gpf56\") pod \"payload-pre-processing-e2e-ipp-a-c44644-58d55657ff-tghrj\" (UID: \"987a60a0-3b09-4fc8-aff6-2c5b47c5f225\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c44644-58d55657ff-tghrj" Oct 30 09:22:21.228857 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:21.228600 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c44644-7db5575f5d-qzlhc" Oct 30 09:22:21.306124 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:21.306068 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/987a60a0-3b09-4fc8-aff6-2c5b47c5f225-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c44644-58d55657ff-tghrj\" (UID: \"987a60a0-3b09-4fc8-aff6-2c5b47c5f225\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c44644-58d55657ff-tghrj" Oct 30 09:22:21.312078 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:21.312043 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpf56\" (UniqueName: \"kubernetes.io/projected/987a60a0-3b09-4fc8-aff6-2c5b47c5f225-kube-api-access-gpf56\") pod \"payload-pre-processing-e2e-ipp-a-c44644-58d55657ff-tghrj\" (UID: \"987a60a0-3b09-4fc8-aff6-2c5b47c5f225\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c44644-58d55657ff-tghrj" Oct 30 09:22:21.314483 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:21.309605 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/987a60a0-3b09-4fc8-aff6-2c5b47c5f225-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c44644-58d55657ff-tghrj\" (UID: \"987a60a0-3b09-4fc8-aff6-2c5b47c5f225\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c44644-58d55657ff-tghrj" Oct 30 09:22:21.336028 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:21.335991 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpf56\" (UniqueName: \"kubernetes.io/projected/987a60a0-3b09-4fc8-aff6-2c5b47c5f225-kube-api-access-gpf56\") pod \"payload-pre-processing-e2e-ipp-a-c44644-58d55657ff-tghrj\" (UID: \"987a60a0-3b09-4fc8-aff6-2c5b47c5f225\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c44644-58d55657ff-tghrj" Oct 30 09:22:21.606875 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:21.606821 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c44644-58d55657ff-tghrj" Oct 30 09:22:21.635706 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:22:21.635673 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8c2c2f2_d8ce_450b_9b00_bb8d96ac68bd.slice/crio-569076cf170ee6e4261d12c252b79c261d0abf8d1799398ec34974c60ca90e07 WatchSource:0}: Error finding container 569076cf170ee6e4261d12c252b79c261d0abf8d1799398ec34974c60ca90e07: Status 404 returned error can't find the container with id 569076cf170ee6e4261d12c252b79c261d0abf8d1799398ec34974c60ca90e07 Oct 30 09:22:21.637977 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:21.637953 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c44644-7db5575f5d-qzlhc"] Oct 30 09:22:21.965026 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:22:21.964996 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod987a60a0_3b09_4fc8_aff6_2c5b47c5f225.slice/crio-9487962bcfc579a66fe9398168b1734fb6d8ddb4ae3e77b0e1ad14c7c1836234 WatchSource:0}: Error finding container 9487962bcfc579a66fe9398168b1734fb6d8ddb4ae3e77b0e1ad14c7c1836234: Status 404 returned error can't find the container with id 9487962bcfc579a66fe9398168b1734fb6d8ddb4ae3e77b0e1ad14c7c1836234 Oct 30 09:22:21.967754 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:21.967731 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c44644-58d55657ff-tghrj"] Oct 30 09:22:22.342612 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:22.342515 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c44644-58d55657ff-tghrj" event={"ID":"987a60a0-3b09-4fc8-aff6-2c5b47c5f225","Type":"ContainerStarted","Data":"5335ad5ccf9291944df03ba3e75b8e6b8a78f75e6b1b89f51e7e41908d251b42"} Oct 30 09:22:22.342778 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:22.342666 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c44644-58d55657ff-tghrj" Oct 30 09:22:22.342778 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:22.342704 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c44644-58d55657ff-tghrj" event={"ID":"987a60a0-3b09-4fc8-aff6-2c5b47c5f225","Type":"ContainerStarted","Data":"9487962bcfc579a66fe9398168b1734fb6d8ddb4ae3e77b0e1ad14c7c1836234"} Oct 30 09:22:22.344252 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:22.344227 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c44644-7db5575f5d-qzlhc" event={"ID":"b8c2c2f2-d8ce-450b-9b00-bb8d96ac68bd","Type":"ContainerStarted","Data":"964fb44b0e8e47ca32cf980d9695b2ab2f6803445f2d5a16dc982788496e4cfb"} Oct 30 09:22:22.344352 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:22.344280 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c44644-7db5575f5d-qzlhc" Oct 30 09:22:22.344352 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:22.344297 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c44644-7db5575f5d-qzlhc" event={"ID":"b8c2c2f2-d8ce-450b-9b00-bb8d96ac68bd","Type":"ContainerStarted","Data":"569076cf170ee6e4261d12c252b79c261d0abf8d1799398ec34974c60ca90e07"} Oct 30 09:22:22.383279 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:22.383237 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c44644-58d55657ff-tghrj" podStartSLOduration=2.383224357 podStartE2EDuration="2.383224357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:22:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:22:22.382448177 +0000 UTC m=+1702.644009165" watchObservedRunningTime="2026-10-30 09:22:22.383224357 +0000 UTC m=+1702.644785345" Oct 30 09:22:22.425339 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:22.425297 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-c44644-7db5575f5d-qzlhc" podStartSLOduration=2.4252855970000002 podStartE2EDuration="2.425285597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:22:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:22:22.423302577 +0000 UTC m=+1702.684863577" watchObservedRunningTime="2026-10-30 09:22:22.425285597 +0000 UTC m=+1702.686846585" Oct 30 09:22:24.752406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:24.749357 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8de308-64c66d9c69-7p6mx"] Oct 30 09:22:24.847437 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:24.837926 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8de308-64c66d9c69-7p6mx"] Oct 30 09:22:24.847437 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:24.838055 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8de308-64c66d9c69-7p6mx" Oct 30 09:22:24.862404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:24.854686 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-8de308-serving-cert\"" Oct 30 09:22:24.921248 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:24.921212 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0b5325c-a526-4cbe-9d70-2dd858185eb5-metrics-certs\") pod \"maas-api-e2e-ipp-a-8de308-64c66d9c69-7p6mx\" (UID: \"f0b5325c-a526-4cbe-9d70-2dd858185eb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8de308-64c66d9c69-7p6mx" Oct 30 09:22:24.922941 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:24.921761 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0b5325c-a526-4cbe-9d70-2dd858185eb5-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8de308-64c66d9c69-7p6mx\" (UID: \"f0b5325c-a526-4cbe-9d70-2dd858185eb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8de308-64c66d9c69-7p6mx" Oct 30 09:22:24.922941 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:24.921830 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0b5325c-a526-4cbe-9d70-2dd858185eb5-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-8de308-64c66d9c69-7p6mx\" (UID: \"f0b5325c-a526-4cbe-9d70-2dd858185eb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8de308-64c66d9c69-7p6mx" Oct 30 09:22:24.922941 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:24.921890 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d29jd\" (UniqueName: \"kubernetes.io/projected/f0b5325c-a526-4cbe-9d70-2dd858185eb5-kube-api-access-d29jd\") pod \"maas-api-e2e-ipp-a-8de308-64c66d9c69-7p6mx\" (UID: \"f0b5325c-a526-4cbe-9d70-2dd858185eb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8de308-64c66d9c69-7p6mx" Oct 30 09:22:25.030469 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:25.029478 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0b5325c-a526-4cbe-9d70-2dd858185eb5-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-8de308-64c66d9c69-7p6mx\" (UID: \"f0b5325c-a526-4cbe-9d70-2dd858185eb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8de308-64c66d9c69-7p6mx" Oct 30 09:22:25.030469 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:25.029576 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d29jd\" (UniqueName: \"kubernetes.io/projected/f0b5325c-a526-4cbe-9d70-2dd858185eb5-kube-api-access-d29jd\") pod \"maas-api-e2e-ipp-a-8de308-64c66d9c69-7p6mx\" (UID: \"f0b5325c-a526-4cbe-9d70-2dd858185eb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8de308-64c66d9c69-7p6mx" Oct 30 09:22:25.030469 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:25.029710 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0b5325c-a526-4cbe-9d70-2dd858185eb5-metrics-certs\") pod \"maas-api-e2e-ipp-a-8de308-64c66d9c69-7p6mx\" (UID: \"f0b5325c-a526-4cbe-9d70-2dd858185eb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8de308-64c66d9c69-7p6mx" Oct 30 09:22:25.030469 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:25.029900 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0b5325c-a526-4cbe-9d70-2dd858185eb5-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8de308-64c66d9c69-7p6mx\" (UID: \"f0b5325c-a526-4cbe-9d70-2dd858185eb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8de308-64c66d9c69-7p6mx" Oct 30 09:22:25.030469 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:22:25.030080 4562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-8de308-serving-cert: secret "maas-api-e2e-ipp-a-8de308-serving-cert" not found Oct 30 09:22:25.030469 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:22:25.030146 4562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0b5325c-a526-4cbe-9d70-2dd858185eb5-maas-api-tls podName:f0b5325c-a526-4cbe-9d70-2dd858185eb5 nodeName:}" failed. No retries permitted until 2026-10-30 09:22:25.530125011 +0000 UTC m=+1705.791685982 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f0b5325c-a526-4cbe-9d70-2dd858185eb5-maas-api-tls") pod "maas-api-e2e-ipp-a-8de308-64c66d9c69-7p6mx" (UID: "f0b5325c-a526-4cbe-9d70-2dd858185eb5") : secret "maas-api-e2e-ipp-a-8de308-serving-cert" not found Oct 30 09:22:25.046495 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:25.031217 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0b5325c-a526-4cbe-9d70-2dd858185eb5-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-8de308-64c66d9c69-7p6mx\" (UID: \"f0b5325c-a526-4cbe-9d70-2dd858185eb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8de308-64c66d9c69-7p6mx" Oct 30 09:22:25.061403 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:25.060482 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d29jd\" (UniqueName: \"kubernetes.io/projected/f0b5325c-a526-4cbe-9d70-2dd858185eb5-kube-api-access-d29jd\") pod \"maas-api-e2e-ipp-a-8de308-64c66d9c69-7p6mx\" (UID: \"f0b5325c-a526-4cbe-9d70-2dd858185eb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8de308-64c66d9c69-7p6mx" Oct 30 09:22:25.082642 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:25.076575 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0b5325c-a526-4cbe-9d70-2dd858185eb5-metrics-certs\") pod \"maas-api-e2e-ipp-a-8de308-64c66d9c69-7p6mx\" (UID: \"f0b5325c-a526-4cbe-9d70-2dd858185eb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8de308-64c66d9c69-7p6mx" Oct 30 09:22:25.538966 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:25.538935 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0b5325c-a526-4cbe-9d70-2dd858185eb5-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8de308-64c66d9c69-7p6mx\" (UID: \"f0b5325c-a526-4cbe-9d70-2dd858185eb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8de308-64c66d9c69-7p6mx" Oct 30 09:22:25.541259 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:25.541234 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0b5325c-a526-4cbe-9d70-2dd858185eb5-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8de308-64c66d9c69-7p6mx\" (UID: \"f0b5325c-a526-4cbe-9d70-2dd858185eb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8de308-64c66d9c69-7p6mx" Oct 30 09:22:25.823736 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:25.823659 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8de308-64c66d9c69-7p6mx" Oct 30 09:22:26.139532 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:26.139450 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8de308-64d7c7bc75-p4xhw" Oct 30 09:22:26.141843 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:26.141580 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-8de308-7699bcfb84-hctsq" Oct 30 09:22:26.181302 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:26.181267 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8de308-64c66d9c69-7p6mx"] Oct 30 09:22:26.215566 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:22:26.215528 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0b5325c_a526_4cbe_9d70_2dd858185eb5.slice/crio-1f5dec07e176805f4d3bfb60263075f46d2f76638f6b03299ab4bfbcdd03f414 WatchSource:0}: Error finding container 1f5dec07e176805f4d3bfb60263075f46d2f76638f6b03299ab4bfbcdd03f414: Status 404 returned error can't find the container with id 1f5dec07e176805f4d3bfb60263075f46d2f76638f6b03299ab4bfbcdd03f414 Oct 30 09:22:26.340712 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:26.340677 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eada0cf-7c8545fd75-cfnvf" Oct 30 09:22:26.376132 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:26.376098 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8de308-64c66d9c69-7p6mx" event={"ID":"f0b5325c-a526-4cbe-9d70-2dd858185eb5","Type":"ContainerStarted","Data":"1f5dec07e176805f4d3bfb60263075f46d2f76638f6b03299ab4bfbcdd03f414"} Oct 30 09:22:27.384606 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:27.384573 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8de308-64c66d9c69-7p6mx" event={"ID":"f0b5325c-a526-4cbe-9d70-2dd858185eb5","Type":"ContainerStarted","Data":"16e065b53259ca85c8c40d128148896e4c17c534b0f2c1909426d6e51e3342de"} Oct 30 09:22:27.385030 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:27.384758 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8de308-64c66d9c69-7p6mx" Oct 30 09:22:27.427115 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:27.427068 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8de308-64c66d9c69-7p6mx" podStartSLOduration=3.003673877 podStartE2EDuration="3.427053245s" totalImagesPullingTime="423.379368ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:22:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:22:27.424427195 +0000 UTC m=+1707.685988179" watchObservedRunningTime="2026-10-30 09:22:27.427053245 +0000 UTC m=+1707.688614279" Oct 30 09:22:28.664903 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:22:28.664857 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:22:33.357490 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:33.357455 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c44644-7db5575f5d-qzlhc" Oct 30 09:22:33.358068 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:33.357575 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c44644-58d55657ff-tghrj" Oct 30 09:22:33.419263 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:33.419227 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8de308-64c66d9c69-7p6mx" Oct 30 09:22:33.467616 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:33.467571 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c44644-8555bd6cd4-pmg85"] Oct 30 09:22:33.473047 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:33.473022 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c44644-8555bd6cd4-pmg85" Oct 30 09:22:33.492440 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:33.492347 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c44644-8555bd6cd4-pmg85"] Oct 30 09:22:33.492583 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:33.492474 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-c44644-serving-cert\"" Oct 30 09:22:33.548018 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:33.547982 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39375e4b-ce51-48bc-b63f-fee3fd03f3b7-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c44644-8555bd6cd4-pmg85\" (UID: \"39375e4b-ce51-48bc-b63f-fee3fd03f3b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c44644-8555bd6cd4-pmg85" Oct 30 09:22:33.548226 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:33.548146 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv9dj\" (UniqueName: \"kubernetes.io/projected/39375e4b-ce51-48bc-b63f-fee3fd03f3b7-kube-api-access-lv9dj\") pod \"maas-api-e2e-ipp-a-c44644-8555bd6cd4-pmg85\" (UID: \"39375e4b-ce51-48bc-b63f-fee3fd03f3b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c44644-8555bd6cd4-pmg85" Oct 30 09:22:33.548226 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:33.548218 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39375e4b-ce51-48bc-b63f-fee3fd03f3b7-metrics-certs\") pod \"maas-api-e2e-ipp-a-c44644-8555bd6cd4-pmg85\" (UID: \"39375e4b-ce51-48bc-b63f-fee3fd03f3b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c44644-8555bd6cd4-pmg85" Oct 30 09:22:33.548368 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:33.548245 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39375e4b-ce51-48bc-b63f-fee3fd03f3b7-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c44644-8555bd6cd4-pmg85\" (UID: \"39375e4b-ce51-48bc-b63f-fee3fd03f3b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c44644-8555bd6cd4-pmg85" Oct 30 09:22:33.649982 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:33.649875 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv9dj\" (UniqueName: \"kubernetes.io/projected/39375e4b-ce51-48bc-b63f-fee3fd03f3b7-kube-api-access-lv9dj\") pod \"maas-api-e2e-ipp-a-c44644-8555bd6cd4-pmg85\" (UID: \"39375e4b-ce51-48bc-b63f-fee3fd03f3b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c44644-8555bd6cd4-pmg85" Oct 30 09:22:33.650291 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:33.650253 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39375e4b-ce51-48bc-b63f-fee3fd03f3b7-metrics-certs\") pod \"maas-api-e2e-ipp-a-c44644-8555bd6cd4-pmg85\" (UID: \"39375e4b-ce51-48bc-b63f-fee3fd03f3b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c44644-8555bd6cd4-pmg85" Oct 30 09:22:33.650550 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:33.650516 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39375e4b-ce51-48bc-b63f-fee3fd03f3b7-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c44644-8555bd6cd4-pmg85\" (UID: \"39375e4b-ce51-48bc-b63f-fee3fd03f3b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c44644-8555bd6cd4-pmg85" Oct 30 09:22:33.650714 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:33.650693 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39375e4b-ce51-48bc-b63f-fee3fd03f3b7-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c44644-8555bd6cd4-pmg85\" (UID: \"39375e4b-ce51-48bc-b63f-fee3fd03f3b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c44644-8555bd6cd4-pmg85" Oct 30 09:22:33.651464 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:33.651443 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39375e4b-ce51-48bc-b63f-fee3fd03f3b7-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c44644-8555bd6cd4-pmg85\" (UID: \"39375e4b-ce51-48bc-b63f-fee3fd03f3b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c44644-8555bd6cd4-pmg85" Oct 30 09:22:33.653480 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:33.653456 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39375e4b-ce51-48bc-b63f-fee3fd03f3b7-metrics-certs\") pod \"maas-api-e2e-ipp-a-c44644-8555bd6cd4-pmg85\" (UID: \"39375e4b-ce51-48bc-b63f-fee3fd03f3b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c44644-8555bd6cd4-pmg85" Oct 30 09:22:33.653711 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:33.653688 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39375e4b-ce51-48bc-b63f-fee3fd03f3b7-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c44644-8555bd6cd4-pmg85\" (UID: \"39375e4b-ce51-48bc-b63f-fee3fd03f3b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c44644-8555bd6cd4-pmg85" Oct 30 09:22:33.671686 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:33.671628 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv9dj\" (UniqueName: \"kubernetes.io/projected/39375e4b-ce51-48bc-b63f-fee3fd03f3b7-kube-api-access-lv9dj\") pod \"maas-api-e2e-ipp-a-c44644-8555bd6cd4-pmg85\" (UID: \"39375e4b-ce51-48bc-b63f-fee3fd03f3b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c44644-8555bd6cd4-pmg85" Oct 30 09:22:33.802326 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:33.802287 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c44644-8555bd6cd4-pmg85" Oct 30 09:22:33.885454 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:33.881682 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8f40a3-68bb7574dc-mdb4m"] Oct 30 09:22:33.889083 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:33.889051 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f40a3-68bb7574dc-mdb4m" Oct 30 09:22:33.901281 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:33.901091 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8f40a3-68bb7574dc-mdb4m"] Oct 30 09:22:33.905608 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:33.905285 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-8f40a3\"" Oct 30 09:22:33.905608 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:33.905329 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-8f40a3-dockercfg-pxg7m\"" Oct 30 09:22:34.122760 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:34.106225 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f182c94b-511e-4b0b-b236-236001563a92-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8f40a3-68bb7574dc-mdb4m\" (UID: \"f182c94b-511e-4b0b-b236-236001563a92\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f40a3-68bb7574dc-mdb4m" Oct 30 09:22:34.122760 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:34.106688 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frc7n\" (UniqueName: \"kubernetes.io/projected/f182c94b-511e-4b0b-b236-236001563a92-kube-api-access-frc7n\") pod \"payload-pre-processing-e2e-ipp-a-8f40a3-68bb7574dc-mdb4m\" (UID: \"f182c94b-511e-4b0b-b236-236001563a92\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f40a3-68bb7574dc-mdb4m" Oct 30 09:22:34.241564 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:34.241235 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f182c94b-511e-4b0b-b236-236001563a92-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8f40a3-68bb7574dc-mdb4m\" (UID: \"f182c94b-511e-4b0b-b236-236001563a92\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f40a3-68bb7574dc-mdb4m" Oct 30 09:22:34.245727 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:34.245682 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f182c94b-511e-4b0b-b236-236001563a92-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8f40a3-68bb7574dc-mdb4m\" (UID: \"f182c94b-511e-4b0b-b236-236001563a92\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f40a3-68bb7574dc-mdb4m" Oct 30 09:22:34.256265 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:34.256222 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c44644-8555bd6cd4-pmg85"] Oct 30 09:22:34.259828 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:34.259451 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frc7n\" (UniqueName: \"kubernetes.io/projected/f182c94b-511e-4b0b-b236-236001563a92-kube-api-access-frc7n\") pod \"payload-pre-processing-e2e-ipp-a-8f40a3-68bb7574dc-mdb4m\" (UID: \"f182c94b-511e-4b0b-b236-236001563a92\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f40a3-68bb7574dc-mdb4m" Oct 30 09:22:34.286418 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:34.286143 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frc7n\" (UniqueName: \"kubernetes.io/projected/f182c94b-511e-4b0b-b236-236001563a92-kube-api-access-frc7n\") pod \"payload-pre-processing-e2e-ipp-a-8f40a3-68bb7574dc-mdb4m\" (UID: \"f182c94b-511e-4b0b-b236-236001563a92\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f40a3-68bb7574dc-mdb4m" Oct 30 09:22:34.453444 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:34.453406 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c44644-8555bd6cd4-pmg85" event={"ID":"39375e4b-ce51-48bc-b63f-fee3fd03f3b7","Type":"ContainerStarted","Data":"81b1fdcef2c7b58b876651b5740d8609829a96aca9b8f5d7cfbf44304df0bd39"} Oct 30 09:22:34.511344 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:34.511265 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f40a3-68bb7574dc-mdb4m" Oct 30 09:22:35.050164 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:35.050139 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8f40a3-68bb7574dc-mdb4m"] Oct 30 09:22:35.462799 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:35.462752 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f40a3-68bb7574dc-mdb4m" event={"ID":"f182c94b-511e-4b0b-b236-236001563a92","Type":"ContainerStarted","Data":"94adc0cf3b23d8cee1a4bbea30c361033954e7c831e937c8917fdb8301d06e83"} Oct 30 09:22:35.463295 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:35.462822 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f40a3-68bb7574dc-mdb4m" event={"ID":"f182c94b-511e-4b0b-b236-236001563a92","Type":"ContainerStarted","Data":"5e83e2a85dc67c5fdf4d324a841947b6bb88018b3ca881fb6454a4174f7c56a6"} Oct 30 09:22:35.463295 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:35.462854 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f40a3-68bb7574dc-mdb4m" Oct 30 09:22:35.465087 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:35.465060 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c44644-8555bd6cd4-pmg85" event={"ID":"39375e4b-ce51-48bc-b63f-fee3fd03f3b7","Type":"ContainerStarted","Data":"0ec7aa7aeeedb4dcfba5df11bd88d52b77a4c8596b78f92e942abbb2904f6229"} Oct 30 09:22:35.465228 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:35.465174 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c44644-8555bd6cd4-pmg85" Oct 30 09:22:35.504121 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:35.504078 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f40a3-68bb7574dc-mdb4m" podStartSLOduration=2.504066226 podStartE2EDuration="2.504066226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:22:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:22:35.502845942 +0000 UTC m=+1715.764406932" watchObservedRunningTime="2026-10-30 09:22:35.504066226 +0000 UTC m=+1715.765627250" Oct 30 09:22:35.545313 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:35.545268 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c44644-8555bd6cd4-pmg85" podStartSLOduration=2.208292191 podStartE2EDuration="2.545255986s" totalImagesPullingTime="336.963795ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:22:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:22:35.54313576 +0000 UTC m=+1715.804696747" watchObservedRunningTime="2026-10-30 09:22:35.545255986 +0000 UTC m=+1715.806816974" Oct 30 09:22:38.525099 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:38.525068 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11378-ed61b7"] Oct 30 09:22:38.529507 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:38.529483 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11378-ed61b7" Oct 30 09:22:38.543369 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:38.543342 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11378-ed61b7"] Oct 30 09:22:38.550834 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:38.550813 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-jjt9f\"" Oct 30 09:22:38.582108 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:38.582083 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2r8v\" (UniqueName: \"kubernetes.io/projected/214f47d1-6f1a-4200-8973-f965173ee0c6-kube-api-access-h2r8v\") pod \"test-curl-11378-ed61b7\" (UID: \"214f47d1-6f1a-4200-8973-f965173ee0c6\") " pod="opendatahub/test-curl-11378-ed61b7" Oct 30 09:22:38.683757 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:38.683705 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2r8v\" (UniqueName: \"kubernetes.io/projected/214f47d1-6f1a-4200-8973-f965173ee0c6-kube-api-access-h2r8v\") pod \"test-curl-11378-ed61b7\" (UID: \"214f47d1-6f1a-4200-8973-f965173ee0c6\") " pod="opendatahub/test-curl-11378-ed61b7" Oct 30 09:22:38.703272 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:38.703233 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2r8v\" (UniqueName: \"kubernetes.io/projected/214f47d1-6f1a-4200-8973-f965173ee0c6-kube-api-access-h2r8v\") pod \"test-curl-11378-ed61b7\" (UID: \"214f47d1-6f1a-4200-8973-f965173ee0c6\") " pod="opendatahub/test-curl-11378-ed61b7" Oct 30 09:22:38.868471 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:38.857502 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11378-ed61b7" Oct 30 09:22:39.365212 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:22:39.365174 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod214f47d1_6f1a_4200_8973_f965173ee0c6.slice/crio-859c93cf72a82511784e5316277f05c74bb8d59c27a517bacb783a29299dd098 WatchSource:0}: Error finding container 859c93cf72a82511784e5316277f05c74bb8d59c27a517bacb783a29299dd098: Status 404 returned error can't find the container with id 859c93cf72a82511784e5316277f05c74bb8d59c27a517bacb783a29299dd098 Oct 30 09:22:39.367967 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:39.367934 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11378-ed61b7"] Oct 30 09:22:39.494258 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:39.494215 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11378-ed61b7" event={"ID":"214f47d1-6f1a-4200-8973-f965173ee0c6","Type":"ContainerStarted","Data":"909e1383f675b30e4b90b6b10838c083692c6aa55a77b3f59eca60b6753f0ab9"} Oct 30 09:22:39.494258 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:39.494261 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11378-ed61b7" event={"ID":"214f47d1-6f1a-4200-8973-f965173ee0c6","Type":"ContainerStarted","Data":"859c93cf72a82511784e5316277f05c74bb8d59c27a517bacb783a29299dd098"} Oct 30 09:22:39.532831 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:39.532765 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11378-ed61b7" podStartSLOduration=1.532747267 podStartE2EDuration="1.532747267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:22:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:22:39.531779805 +0000 UTC m=+1719.793340797" watchObservedRunningTime="2026-10-30 09:22:39.532747267 +0000 UTC m=+1719.794308256" Oct 30 09:22:40.496510 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:40.496436 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11378-ed61b7"] Oct 30 09:22:40.501223 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:40.501193 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11378-ed61b7"] Oct 30 09:22:41.477295 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:41.477267 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c44644-8555bd6cd4-pmg85" Oct 30 09:22:41.509580 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:41.509525 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11378-ed61b7" podUID="214f47d1-6f1a-4200-8973-f965173ee0c6" containerName="test-curl-11378-ed61b7" containerID="cri-o://909e1383f675b30e4b90b6b10838c083692c6aa55a77b3f59eca60b6753f0ab9" gracePeriod=2 Oct 30 09:22:41.535189 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:22:41.531984 4562 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11378-ed61b7\" is forbidden: User \"system:node:ip-10-0-143-244.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-244.ec2.internal' and this object" podUID="214f47d1-6f1a-4200-8973-f965173ee0c6" pod="opendatahub/test-curl-11378-ed61b7" Oct 30 09:22:43.582036 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:43.581996 4562 generic.go:356] "Generic (PLEG): container finished" podID="214f47d1-6f1a-4200-8973-f965173ee0c6" containerID="909e1383f675b30e4b90b6b10838c083692c6aa55a77b3f59eca60b6753f0ab9" exitCode=137 Oct 30 09:22:43.736241 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:43.736214 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11378-ed61b7" Oct 30 09:22:43.908704 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:43.908633 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/214f47d1-6f1a-4200-8973-f965173ee0c6-kube-api-access-h2r8v\" (UniqueName: \"kubernetes.io/projected/214f47d1-6f1a-4200-8973-f965173ee0c6-kube-api-access-h2r8v\") pod \"214f47d1-6f1a-4200-8973-f965173ee0c6\" (UID: \"214f47d1-6f1a-4200-8973-f965173ee0c6\") " Oct 30 09:22:43.910776 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:43.910748 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/214f47d1-6f1a-4200-8973-f965173ee0c6-kube-api-access-h2r8v" pod "214f47d1-6f1a-4200-8973-f965173ee0c6" (UID: "214f47d1-6f1a-4200-8973-f965173ee0c6"). InnerVolumeSpecName "kube-api-access-h2r8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:22:44.010887 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:44.010846 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2r8v\" (UniqueName: \"kubernetes.io/projected/214f47d1-6f1a-4200-8973-f965173ee0c6-kube-api-access-h2r8v\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:22:44.591173 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:44.591135 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11378-ed61b7" Oct 30 09:22:44.591617 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:44.591138 4562 scope.go:122] "RemoveContainer" containerID="909e1383f675b30e4b90b6b10838c083692c6aa55a77b3f59eca60b6753f0ab9" Oct 30 09:22:45.820736 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:45.820701 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8de308-7699bcfb84-hctsq"] Oct 30 09:22:45.875468 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:45.875436 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="214f47d1-6f1a-4200-8973-f965173ee0c6" path="/var/lib/kubelet/pods/214f47d1-6f1a-4200-8973-f965173ee0c6/volumes" Oct 30 09:22:45.875775 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:45.875761 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8de308-64d7c7bc75-p4xhw"] Oct 30 09:22:46.474480 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:46.474451 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f40a3-68bb7574dc-mdb4m" Oct 30 09:22:47.548472 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:47.548430 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-58fadf-5968d7d495-gprm9"] Oct 30 09:22:47.549317 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:47.549293 4562 cpu_manager.go:414] "containerMap: removing container" podUID="214f47d1-6f1a-4200-8973-f965173ee0c6" containerName="test-curl-11378-ed61b7" Oct 30 09:22:47.549454 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:47.549318 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="214f47d1-6f1a-4200-8973-f965173ee0c6" containerName="test-curl-11378-ed61b7" Oct 30 09:22:47.549540 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:47.549526 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="214f47d1-6f1a-4200-8973-f965173ee0c6" containerName="test-curl-11378-ed61b7" Oct 30 09:22:47.554539 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:47.554519 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-58fadf-5968d7d495-gprm9" Oct 30 09:22:47.569434 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:47.569410 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-58fadf-5968d7d495-gprm9"] Oct 30 09:22:47.569777 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:47.569755 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-58fadf\"" Oct 30 09:22:47.578081 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:47.578060 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-58fadf-dockercfg-pzdlm\"" Oct 30 09:22:47.803495 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:47.786034 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bc980b8-abbe-4e78-bf92-cd6654479cbe-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-58fadf-5968d7d495-gprm9\" (UID: \"5bc980b8-abbe-4e78-bf92-cd6654479cbe\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-58fadf-5968d7d495-gprm9" Oct 30 09:22:47.803495 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:47.786359 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc9hj\" (UniqueName: \"kubernetes.io/projected/5bc980b8-abbe-4e78-bf92-cd6654479cbe-kube-api-access-bc9hj\") pod \"payload-pre-processing-e2e-api-a-58fadf-5968d7d495-gprm9\" (UID: \"5bc980b8-abbe-4e78-bf92-cd6654479cbe\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-58fadf-5968d7d495-gprm9" Oct 30 09:22:47.812212 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:47.808453 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-58fadf-7748bcb6d6-9pwn4"] Oct 30 09:22:47.817360 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:47.815169 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-58fadf-7748bcb6d6-9pwn4" Oct 30 09:22:47.844758 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:47.844681 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-58fadf-7748bcb6d6-9pwn4"] Oct 30 09:22:47.928413 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:47.927927 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bc980b8-abbe-4e78-bf92-cd6654479cbe-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-58fadf-5968d7d495-gprm9\" (UID: \"5bc980b8-abbe-4e78-bf92-cd6654479cbe\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-58fadf-5968d7d495-gprm9" Oct 30 09:22:47.928413 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:47.928039 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh2np\" (UniqueName: \"kubernetes.io/projected/2fd05ff0-d658-452d-8b4a-399aca8f7876-kube-api-access-kh2np\") pod \"payload-processing-e2e-api-a-58fadf-7748bcb6d6-9pwn4\" (UID: \"2fd05ff0-d658-452d-8b4a-399aca8f7876\") " pod="openshift-ingress/payload-processing-e2e-api-a-58fadf-7748bcb6d6-9pwn4" Oct 30 09:22:47.928413 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:47.928237 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fd05ff0-d658-452d-8b4a-399aca8f7876-plugins-config-volume\") pod \"payload-processing-e2e-api-a-58fadf-7748bcb6d6-9pwn4\" (UID: \"2fd05ff0-d658-452d-8b4a-399aca8f7876\") " pod="openshift-ingress/payload-processing-e2e-api-a-58fadf-7748bcb6d6-9pwn4" Oct 30 09:22:47.928413 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:47.928286 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc9hj\" (UniqueName: \"kubernetes.io/projected/5bc980b8-abbe-4e78-bf92-cd6654479cbe-kube-api-access-bc9hj\") pod \"payload-pre-processing-e2e-api-a-58fadf-5968d7d495-gprm9\" (UID: \"5bc980b8-abbe-4e78-bf92-cd6654479cbe\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-58fadf-5968d7d495-gprm9" Oct 30 09:22:47.931329 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:47.931304 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bc980b8-abbe-4e78-bf92-cd6654479cbe-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-58fadf-5968d7d495-gprm9\" (UID: \"5bc980b8-abbe-4e78-bf92-cd6654479cbe\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-58fadf-5968d7d495-gprm9" Oct 30 09:22:47.950107 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:47.950077 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc9hj\" (UniqueName: \"kubernetes.io/projected/5bc980b8-abbe-4e78-bf92-cd6654479cbe-kube-api-access-bc9hj\") pod \"payload-pre-processing-e2e-api-a-58fadf-5968d7d495-gprm9\" (UID: \"5bc980b8-abbe-4e78-bf92-cd6654479cbe\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-58fadf-5968d7d495-gprm9" Oct 30 09:22:48.031003 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:48.030966 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh2np\" (UniqueName: \"kubernetes.io/projected/2fd05ff0-d658-452d-8b4a-399aca8f7876-kube-api-access-kh2np\") pod \"payload-processing-e2e-api-a-58fadf-7748bcb6d6-9pwn4\" (UID: \"2fd05ff0-d658-452d-8b4a-399aca8f7876\") " pod="openshift-ingress/payload-processing-e2e-api-a-58fadf-7748bcb6d6-9pwn4" Oct 30 09:22:48.031181 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:48.031128 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fd05ff0-d658-452d-8b4a-399aca8f7876-plugins-config-volume\") pod \"payload-processing-e2e-api-a-58fadf-7748bcb6d6-9pwn4\" (UID: \"2fd05ff0-d658-452d-8b4a-399aca8f7876\") " pod="openshift-ingress/payload-processing-e2e-api-a-58fadf-7748bcb6d6-9pwn4" Oct 30 09:22:48.032127 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:48.032097 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fd05ff0-d658-452d-8b4a-399aca8f7876-plugins-config-volume\") pod \"payload-processing-e2e-api-a-58fadf-7748bcb6d6-9pwn4\" (UID: \"2fd05ff0-d658-452d-8b4a-399aca8f7876\") " pod="openshift-ingress/payload-processing-e2e-api-a-58fadf-7748bcb6d6-9pwn4" Oct 30 09:22:48.051963 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:48.051928 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh2np\" (UniqueName: \"kubernetes.io/projected/2fd05ff0-d658-452d-8b4a-399aca8f7876-kube-api-access-kh2np\") pod \"payload-processing-e2e-api-a-58fadf-7748bcb6d6-9pwn4\" (UID: \"2fd05ff0-d658-452d-8b4a-399aca8f7876\") " pod="openshift-ingress/payload-processing-e2e-api-a-58fadf-7748bcb6d6-9pwn4" Oct 30 09:22:48.158824 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:48.158722 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-58fadf-7748bcb6d6-9pwn4" Oct 30 09:22:48.171058 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:48.170985 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-58fadf-5968d7d495-gprm9" Oct 30 09:22:48.175670 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:48.175643 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8f40a3-784d84f5c5-dtwh7"] Oct 30 09:22:48.185965 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:48.185752 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8f40a3-784d84f5c5-dtwh7" Oct 30 09:22:48.215400 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:48.211452 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8f40a3-784d84f5c5-dtwh7"] Oct 30 09:22:48.215400 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:48.212328 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-8f40a3-serving-cert\"" Oct 30 09:22:48.336769 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:48.336736 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eada6a8f-af88-4279-ae23-1bab2a23be56-metrics-certs\") pod \"maas-api-e2e-ipp-a-8f40a3-784d84f5c5-dtwh7\" (UID: \"eada6a8f-af88-4279-ae23-1bab2a23be56\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8f40a3-784d84f5c5-dtwh7" Oct 30 09:22:48.336940 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:48.336881 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xswvz\" (UniqueName: \"kubernetes.io/projected/eada6a8f-af88-4279-ae23-1bab2a23be56-kube-api-access-xswvz\") pod \"maas-api-e2e-ipp-a-8f40a3-784d84f5c5-dtwh7\" (UID: \"eada6a8f-af88-4279-ae23-1bab2a23be56\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8f40a3-784d84f5c5-dtwh7" Oct 30 09:22:48.337006 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:48.336940 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eada6a8f-af88-4279-ae23-1bab2a23be56-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-8f40a3-784d84f5c5-dtwh7\" (UID: \"eada6a8f-af88-4279-ae23-1bab2a23be56\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8f40a3-784d84f5c5-dtwh7" Oct 30 09:22:48.337098 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:48.337075 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eada6a8f-af88-4279-ae23-1bab2a23be56-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8f40a3-784d84f5c5-dtwh7\" (UID: \"eada6a8f-af88-4279-ae23-1bab2a23be56\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8f40a3-784d84f5c5-dtwh7" Oct 30 09:22:48.438918 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:48.438881 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eada6a8f-af88-4279-ae23-1bab2a23be56-metrics-certs\") pod \"maas-api-e2e-ipp-a-8f40a3-784d84f5c5-dtwh7\" (UID: \"eada6a8f-af88-4279-ae23-1bab2a23be56\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8f40a3-784d84f5c5-dtwh7" Oct 30 09:22:48.439092 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:48.439044 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xswvz\" (UniqueName: \"kubernetes.io/projected/eada6a8f-af88-4279-ae23-1bab2a23be56-kube-api-access-xswvz\") pod \"maas-api-e2e-ipp-a-8f40a3-784d84f5c5-dtwh7\" (UID: \"eada6a8f-af88-4279-ae23-1bab2a23be56\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8f40a3-784d84f5c5-dtwh7" Oct 30 09:22:48.439092 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:48.439077 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eada6a8f-af88-4279-ae23-1bab2a23be56-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-8f40a3-784d84f5c5-dtwh7\" (UID: \"eada6a8f-af88-4279-ae23-1bab2a23be56\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8f40a3-784d84f5c5-dtwh7" Oct 30 09:22:48.439202 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:48.439150 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eada6a8f-af88-4279-ae23-1bab2a23be56-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8f40a3-784d84f5c5-dtwh7\" (UID: \"eada6a8f-af88-4279-ae23-1bab2a23be56\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8f40a3-784d84f5c5-dtwh7" Oct 30 09:22:48.439984 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:48.439957 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eada6a8f-af88-4279-ae23-1bab2a23be56-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-8f40a3-784d84f5c5-dtwh7\" (UID: \"eada6a8f-af88-4279-ae23-1bab2a23be56\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8f40a3-784d84f5c5-dtwh7" Oct 30 09:22:48.441399 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:48.441364 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eada6a8f-af88-4279-ae23-1bab2a23be56-metrics-certs\") pod \"maas-api-e2e-ipp-a-8f40a3-784d84f5c5-dtwh7\" (UID: \"eada6a8f-af88-4279-ae23-1bab2a23be56\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8f40a3-784d84f5c5-dtwh7" Oct 30 09:22:48.441675 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:48.441655 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eada6a8f-af88-4279-ae23-1bab2a23be56-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8f40a3-784d84f5c5-dtwh7\" (UID: \"eada6a8f-af88-4279-ae23-1bab2a23be56\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8f40a3-784d84f5c5-dtwh7" Oct 30 09:22:48.522325 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:48.522213 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xswvz\" (UniqueName: \"kubernetes.io/projected/eada6a8f-af88-4279-ae23-1bab2a23be56-kube-api-access-xswvz\") pod \"maas-api-e2e-ipp-a-8f40a3-784d84f5c5-dtwh7\" (UID: \"eada6a8f-af88-4279-ae23-1bab2a23be56\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8f40a3-784d84f5c5-dtwh7" Oct 30 09:22:48.555220 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:48.554912 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-58fadf-7748bcb6d6-9pwn4"] Oct 30 09:22:48.559710 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:48.558642 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8f40a3-784d84f5c5-dtwh7" Oct 30 09:22:48.586041 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:48.586010 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-58fadf-5968d7d495-gprm9"] Oct 30 09:22:48.632062 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:48.632031 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-58fadf-5968d7d495-gprm9" event={"ID":"5bc980b8-abbe-4e78-bf92-cd6654479cbe","Type":"ContainerStarted","Data":"574dc4d247bbad456d68fbe6c0a5e2a22a8652ef7c2ed3ff4696796623903089"} Oct 30 09:22:48.633806 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:48.633577 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-58fadf-7748bcb6d6-9pwn4" event={"ID":"2fd05ff0-d658-452d-8b4a-399aca8f7876","Type":"ContainerStarted","Data":"1cc797d36fe6eed0a8629527e2684279dfb5c27e1bcc4cc63cc044bbbe45daa8"} Oct 30 09:22:48.909330 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:22:48.909299 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeada6a8f_af88_4279_ae23_1bab2a23be56.slice/crio-e7b4a692828ca7598ba11d98554cf0ef4039e6f40e6cba9acd2e4b50c30983f1 WatchSource:0}: Error finding container e7b4a692828ca7598ba11d98554cf0ef4039e6f40e6cba9acd2e4b50c30983f1: Status 404 returned error can't find the container with id e7b4a692828ca7598ba11d98554cf0ef4039e6f40e6cba9acd2e4b50c30983f1 Oct 30 09:22:48.911798 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:48.911780 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8f40a3-784d84f5c5-dtwh7"] Oct 30 09:22:49.642151 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:49.642066 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-58fadf-5968d7d495-gprm9" event={"ID":"5bc980b8-abbe-4e78-bf92-cd6654479cbe","Type":"ContainerStarted","Data":"414f71432cf30512f53cd0b55bc7e3c0167c846de4fca35d86d68ad57dd93a6c"} Oct 30 09:22:49.642680 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:49.642279 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-58fadf-5968d7d495-gprm9" Oct 30 09:22:49.643983 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:49.643959 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-58fadf-7748bcb6d6-9pwn4" event={"ID":"2fd05ff0-d658-452d-8b4a-399aca8f7876","Type":"ContainerStarted","Data":"5c60c0e665f6f406eff6c8026dc02a810f4fdcceb784688edafa9d5a7e6c14b6"} Oct 30 09:22:49.644093 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:49.644011 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-58fadf-7748bcb6d6-9pwn4" Oct 30 09:22:49.645840 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:49.645814 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8f40a3-784d84f5c5-dtwh7" event={"ID":"eada6a8f-af88-4279-ae23-1bab2a23be56","Type":"ContainerStarted","Data":"7eae696ae7f676e5db6bb1239b6b1616057437bcf4062ebd4417ce548cc14a81"} Oct 30 09:22:49.645933 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:49.645854 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8f40a3-784d84f5c5-dtwh7" event={"ID":"eada6a8f-af88-4279-ae23-1bab2a23be56","Type":"ContainerStarted","Data":"e7b4a692828ca7598ba11d98554cf0ef4039e6f40e6cba9acd2e4b50c30983f1"} Oct 30 09:22:49.645933 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:49.645880 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8f40a3-784d84f5c5-dtwh7" Oct 30 09:22:49.697744 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:49.697672 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-58fadf-5968d7d495-gprm9" podStartSLOduration=2.69764923 podStartE2EDuration="2.69764923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:22:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:22:49.686472913 +0000 UTC m=+1729.948033928" watchObservedRunningTime="2026-10-30 09:22:49.69764923 +0000 UTC m=+1729.959210259" Oct 30 09:22:49.731947 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:49.731879 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8f40a3-784d84f5c5-dtwh7" podStartSLOduration=1.404399465 podStartE2EDuration="1.731857523s" totalImagesPullingTime="327.458058ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:22:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:22:49.726015938 +0000 UTC m=+1729.987576927" watchObservedRunningTime="2026-10-30 09:22:49.731857523 +0000 UTC m=+1729.993418514" Oct 30 09:22:49.766043 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:49.765992 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-58fadf-7748bcb6d6-9pwn4" podStartSLOduration=2.765978869 podStartE2EDuration="2.765978869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:22:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:22:49.765090177 +0000 UTC m=+1730.026651166" watchObservedRunningTime="2026-10-30 09:22:49.765978869 +0000 UTC m=+1730.027539856" Oct 30 09:22:50.821534 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:50.821476 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-8de308-7699bcfb84-hctsq" podUID="156b5de0-6a01-4c16-97a8-09c5640def4d" containerName="payload-processing" containerID="cri-o://536de35bf7c6b7e64d224dc5882d526b69001778cc10c8952a1d3ef55694e9ba" gracePeriod=25 Oct 30 09:22:50.876661 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:50.876572 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8de308-64d7c7bc75-p4xhw" podUID="0a82bec3-3dbf-4c3b-87a7-90043b88d127" containerName="payload-pre-processing" containerID="cri-o://0bf7eb55e4e0a1bddc1f8b9a38bad9794960c47517a59891c27b88cdf9a61449" gracePeriod=25 Oct 30 09:22:51.195171 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:51.195144 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8de308-7699bcfb84-hctsq" Oct 30 09:22:51.213492 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:51.213468 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8de308-64d7c7bc75-p4xhw" Oct 30 09:22:51.237571 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:51.236536 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a82bec3-3dbf-4c3b-87a7-90043b88d127-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a82bec3-3dbf-4c3b-87a7-90043b88d127-plugins-config-volume\") pod \"0a82bec3-3dbf-4c3b-87a7-90043b88d127\" (UID: \"0a82bec3-3dbf-4c3b-87a7-90043b88d127\") " Oct 30 09:22:51.237571 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:51.236648 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/156b5de0-6a01-4c16-97a8-09c5640def4d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/156b5de0-6a01-4c16-97a8-09c5640def4d-plugins-config-volume\") pod \"156b5de0-6a01-4c16-97a8-09c5640def4d\" (UID: \"156b5de0-6a01-4c16-97a8-09c5640def4d\") " Oct 30 09:22:51.237571 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:51.236691 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a82bec3-3dbf-4c3b-87a7-90043b88d127-kube-api-access-tj24l\" (UniqueName: \"kubernetes.io/projected/0a82bec3-3dbf-4c3b-87a7-90043b88d127-kube-api-access-tj24l\") pod \"0a82bec3-3dbf-4c3b-87a7-90043b88d127\" (UID: \"0a82bec3-3dbf-4c3b-87a7-90043b88d127\") " Oct 30 09:22:51.237571 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:51.236843 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/156b5de0-6a01-4c16-97a8-09c5640def4d-kube-api-access-dsh8s\" (UniqueName: \"kubernetes.io/projected/156b5de0-6a01-4c16-97a8-09c5640def4d-kube-api-access-dsh8s\") pod \"156b5de0-6a01-4c16-97a8-09c5640def4d\" (UID: \"156b5de0-6a01-4c16-97a8-09c5640def4d\") " Oct 30 09:22:51.237948 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:51.237782 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a82bec3-3dbf-4c3b-87a7-90043b88d127-plugins-config-volume" pod "0a82bec3-3dbf-4c3b-87a7-90043b88d127" (UID: "0a82bec3-3dbf-4c3b-87a7-90043b88d127"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:22:51.238281 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:51.238252 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/156b5de0-6a01-4c16-97a8-09c5640def4d-plugins-config-volume" pod "156b5de0-6a01-4c16-97a8-09c5640def4d" (UID: "156b5de0-6a01-4c16-97a8-09c5640def4d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:22:51.239977 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:51.239935 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/156b5de0-6a01-4c16-97a8-09c5640def4d-kube-api-access-dsh8s" pod "156b5de0-6a01-4c16-97a8-09c5640def4d" (UID: "156b5de0-6a01-4c16-97a8-09c5640def4d"). InnerVolumeSpecName "kube-api-access-dsh8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:22:51.244238 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:51.244211 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a82bec3-3dbf-4c3b-87a7-90043b88d127-kube-api-access-tj24l" pod "0a82bec3-3dbf-4c3b-87a7-90043b88d127" (UID: "0a82bec3-3dbf-4c3b-87a7-90043b88d127"). InnerVolumeSpecName "kube-api-access-tj24l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:22:51.341438 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:51.340163 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsh8s\" (UniqueName: \"kubernetes.io/projected/156b5de0-6a01-4c16-97a8-09c5640def4d-kube-api-access-dsh8s\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:22:51.341438 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:51.340212 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a82bec3-3dbf-4c3b-87a7-90043b88d127-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:22:51.341438 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:51.340228 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/156b5de0-6a01-4c16-97a8-09c5640def4d-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:22:51.341438 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:51.340242 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tj24l\" (UniqueName: \"kubernetes.io/projected/0a82bec3-3dbf-4c3b-87a7-90043b88d127-kube-api-access-tj24l\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:22:51.659859 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:51.659827 4562 generic.go:356] "Generic (PLEG): container finished" podID="0a82bec3-3dbf-4c3b-87a7-90043b88d127" containerID="0bf7eb55e4e0a1bddc1f8b9a38bad9794960c47517a59891c27b88cdf9a61449" exitCode=0 Oct 30 09:22:51.660061 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:51.659872 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8de308-64d7c7bc75-p4xhw" Oct 30 09:22:51.660061 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:51.659920 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8de308-64d7c7bc75-p4xhw" event={"ID":"0a82bec3-3dbf-4c3b-87a7-90043b88d127","Type":"ContainerDied","Data":"0bf7eb55e4e0a1bddc1f8b9a38bad9794960c47517a59891c27b88cdf9a61449"} Oct 30 09:22:51.660061 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:51.659962 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8de308-64d7c7bc75-p4xhw" event={"ID":"0a82bec3-3dbf-4c3b-87a7-90043b88d127","Type":"ContainerDied","Data":"f5484c6f56247db5ac9ce91f9426d78dfb0097bb4036a9dddfc921060a60ed3a"} Oct 30 09:22:51.660061 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:51.659987 4562 scope.go:122] "RemoveContainer" containerID="0bf7eb55e4e0a1bddc1f8b9a38bad9794960c47517a59891c27b88cdf9a61449" Oct 30 09:22:51.661701 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:51.661588 4562 generic.go:356] "Generic (PLEG): container finished" podID="156b5de0-6a01-4c16-97a8-09c5640def4d" containerID="536de35bf7c6b7e64d224dc5882d526b69001778cc10c8952a1d3ef55694e9ba" exitCode=0 Oct 30 09:22:51.661701 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:51.661618 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8de308-7699bcfb84-hctsq" event={"ID":"156b5de0-6a01-4c16-97a8-09c5640def4d","Type":"ContainerDied","Data":"536de35bf7c6b7e64d224dc5882d526b69001778cc10c8952a1d3ef55694e9ba"} Oct 30 09:22:51.661701 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:51.661639 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8de308-7699bcfb84-hctsq" Oct 30 09:22:51.661701 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:51.661667 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8de308-7699bcfb84-hctsq" event={"ID":"156b5de0-6a01-4c16-97a8-09c5640def4d","Type":"ContainerDied","Data":"5731555b0ad3427ca19947628057e054947a012fc1d08a472ae74c83d15a12be"} Oct 30 09:22:51.674255 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:51.674235 4562 scope.go:122] "RemoveContainer" containerID="0bf7eb55e4e0a1bddc1f8b9a38bad9794960c47517a59891c27b88cdf9a61449" Oct 30 09:22:51.674533 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:22:51.674514 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bf7eb55e4e0a1bddc1f8b9a38bad9794960c47517a59891c27b88cdf9a61449\": container with ID starting with 0bf7eb55e4e0a1bddc1f8b9a38bad9794960c47517a59891c27b88cdf9a61449 not found: ID does not exist" containerID="0bf7eb55e4e0a1bddc1f8b9a38bad9794960c47517a59891c27b88cdf9a61449" Oct 30 09:22:51.674594 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:51.674542 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bf7eb55e4e0a1bddc1f8b9a38bad9794960c47517a59891c27b88cdf9a61449"} err="failed to get container status \"0bf7eb55e4e0a1bddc1f8b9a38bad9794960c47517a59891c27b88cdf9a61449\": rpc error: code = NotFound desc = could not find container \"0bf7eb55e4e0a1bddc1f8b9a38bad9794960c47517a59891c27b88cdf9a61449\": container with ID starting with 0bf7eb55e4e0a1bddc1f8b9a38bad9794960c47517a59891c27b88cdf9a61449 not found: ID does not exist" Oct 30 09:22:51.674594 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:51.674558 4562 scope.go:122] "RemoveContainer" containerID="536de35bf7c6b7e64d224dc5882d526b69001778cc10c8952a1d3ef55694e9ba" Oct 30 09:22:51.685505 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:51.685483 4562 scope.go:122] "RemoveContainer" containerID="536de35bf7c6b7e64d224dc5882d526b69001778cc10c8952a1d3ef55694e9ba" Oct 30 09:22:51.685758 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:22:51.685740 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"536de35bf7c6b7e64d224dc5882d526b69001778cc10c8952a1d3ef55694e9ba\": container with ID starting with 536de35bf7c6b7e64d224dc5882d526b69001778cc10c8952a1d3ef55694e9ba not found: ID does not exist" containerID="536de35bf7c6b7e64d224dc5882d526b69001778cc10c8952a1d3ef55694e9ba" Oct 30 09:22:51.685833 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:51.685770 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"536de35bf7c6b7e64d224dc5882d526b69001778cc10c8952a1d3ef55694e9ba"} err="failed to get container status \"536de35bf7c6b7e64d224dc5882d526b69001778cc10c8952a1d3ef55694e9ba\": rpc error: code = NotFound desc = could not find container \"536de35bf7c6b7e64d224dc5882d526b69001778cc10c8952a1d3ef55694e9ba\": container with ID starting with 536de35bf7c6b7e64d224dc5882d526b69001778cc10c8952a1d3ef55694e9ba not found: ID does not exist" Oct 30 09:22:51.715702 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:51.715680 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8de308-7699bcfb84-hctsq"] Oct 30 09:22:51.727363 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:51.727333 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8de308-7699bcfb84-hctsq"] Oct 30 09:22:51.770554 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:51.770525 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8de308-64d7c7bc75-p4xhw"] Oct 30 09:22:51.776218 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:51.776187 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8de308-64d7c7bc75-p4xhw"] Oct 30 09:22:51.905411 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:51.899819 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a82bec3-3dbf-4c3b-87a7-90043b88d127" path="/var/lib/kubelet/pods/0a82bec3-3dbf-4c3b-87a7-90043b88d127/volumes" Oct 30 09:22:51.905411 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:51.900650 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="156b5de0-6a01-4c16-97a8-09c5640def4d" path="/var/lib/kubelet/pods/156b5de0-6a01-4c16-97a8-09c5640def4d/volumes" Oct 30 09:22:52.671569 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:52.671535 4562 generic.go:356] "Generic (PLEG): container finished" podID="7b433eaf-bc2e-4a08-8e7a-527e88ef4c59" containerID="4a49092088ec33406b51283b5dc3b0064f3bbd844e34fe625074b009ee0ca805" exitCode=0 Oct 30 09:22:52.671741 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:52.671603 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8" event={"ID":"7b433eaf-bc2e-4a08-8e7a-527e88ef4c59","Type":"ContainerDied","Data":"4a49092088ec33406b51283b5dc3b0064f3bbd844e34fe625074b009ee0ca805"} Oct 30 09:22:52.672021 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:52.672006 4562 scope.go:122] "RemoveContainer" containerID="4a49092088ec33406b51283b5dc3b0064f3bbd844e34fe625074b009ee0ca805" Oct 30 09:22:52.673492 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:52.673466 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:22:52.673569 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:52.673536 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:22:52.673607 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:52.673568 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:22:53.697404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:53.694575 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8" event={"ID":"7b433eaf-bc2e-4a08-8e7a-527e88ef4c59","Type":"ContainerStarted","Data":"1e903368f7a3186b97d6e289bd97e71795cb7f690d1565926e5368ca4ef759e7"} Oct 30 09:22:53.890743 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:53.890707 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8" Oct 30 09:22:54.663445 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:54.662750 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8" Oct 30 09:22:54.665170 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:54.665140 4562 patch_prober.go:29] interesting pod/e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 30 09:22:54.665321 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:54.665193 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8" podUID="7b433eaf-bc2e-4a08-8e7a-527e88ef4c59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 30 09:22:55.662405 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:55.661487 4562 patch_prober.go:29] interesting pod/e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 30 09:22:55.662405 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:55.661563 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8" podUID="7b433eaf-bc2e-4a08-8e7a-527e88ef4c59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 30 09:22:55.667879 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:55.667854 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8f40a3-784d84f5c5-dtwh7" Oct 30 09:22:56.659407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:56.657761 4562 patch_prober.go:29] interesting pod/e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 30 09:22:56.659407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:56.657856 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8" podUID="7b433eaf-bc2e-4a08-8e7a-527e88ef4c59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 30 09:22:57.663876 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:57.663824 4562 patch_prober.go:29] interesting pod/e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 30 09:22:57.664291 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:57.663942 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8" podUID="7b433eaf-bc2e-4a08-8e7a-527e88ef4c59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 30 09:22:58.131777 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:58.131740 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c44644-7db5575f5d-qzlhc"] Oct 30 09:22:58.169067 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:58.169031 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c44644-58d55657ff-tghrj"] Oct 30 09:22:58.660418 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:58.657791 4562 patch_prober.go:29] interesting pod/e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 30 09:22:58.660418 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:58.657868 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8" podUID="7b433eaf-bc2e-4a08-8e7a-527e88ef4c59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 30 09:22:58.694795 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:22:58.694759 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:22:59.660290 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:59.660261 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8" Oct 30 09:22:59.661247 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:22:59.661226 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8" Oct 30 09:23:00.699688 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:00.698726 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-58fadf-7748bcb6d6-9pwn4" Oct 30 09:23:00.706573 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:00.705569 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-58fadf-5968d7d495-gprm9" Oct 30 09:23:01.386274 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:01.386243 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8de308-7b79454f5d-w8kbm"] Oct 30 09:23:01.386979 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:01.386962 4562 cpu_manager.go:414] "containerMap: removing container" podUID="0a82bec3-3dbf-4c3b-87a7-90043b88d127" containerName="payload-pre-processing" Oct 30 09:23:01.387037 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:01.386980 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a82bec3-3dbf-4c3b-87a7-90043b88d127" containerName="payload-pre-processing" Oct 30 09:23:01.387037 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:01.386995 4562 cpu_manager.go:414] "containerMap: removing container" podUID="156b5de0-6a01-4c16-97a8-09c5640def4d" containerName="payload-processing" Oct 30 09:23:01.387037 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:01.387000 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="156b5de0-6a01-4c16-97a8-09c5640def4d" containerName="payload-processing" Oct 30 09:23:01.387139 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:01.387126 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a82bec3-3dbf-4c3b-87a7-90043b88d127" containerName="payload-pre-processing" Oct 30 09:23:01.387139 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:01.387135 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="156b5de0-6a01-4c16-97a8-09c5640def4d" containerName="payload-processing" Oct 30 09:23:01.390999 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:01.390973 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8de308-7b79454f5d-w8kbm" Oct 30 09:23:01.408525 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:01.408501 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8de308-7b79454f5d-w8kbm"] Oct 30 09:23:01.417540 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:01.417509 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-8de308\"" Oct 30 09:23:01.417676 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:01.417564 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-8de308-dockercfg-gdh57\"" Oct 30 09:23:01.565406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:01.563535 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dacc132b-fcda-49a6-a569-4696ea206638-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8de308-7b79454f5d-w8kbm\" (UID: \"dacc132b-fcda-49a6-a569-4696ea206638\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8de308-7b79454f5d-w8kbm" Oct 30 09:23:01.565406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:01.563799 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8nt5\" (UniqueName: \"kubernetes.io/projected/dacc132b-fcda-49a6-a569-4696ea206638-kube-api-access-j8nt5\") pod \"payload-processing-e2e-ipp-a-8de308-7b79454f5d-w8kbm\" (UID: \"dacc132b-fcda-49a6-a569-4696ea206638\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8de308-7b79454f5d-w8kbm" Oct 30 09:23:01.665442 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:01.665345 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dacc132b-fcda-49a6-a569-4696ea206638-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8de308-7b79454f5d-w8kbm\" (UID: \"dacc132b-fcda-49a6-a569-4696ea206638\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8de308-7b79454f5d-w8kbm" Oct 30 09:23:01.665615 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:01.665484 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8nt5\" (UniqueName: \"kubernetes.io/projected/dacc132b-fcda-49a6-a569-4696ea206638-kube-api-access-j8nt5\") pod \"payload-processing-e2e-ipp-a-8de308-7b79454f5d-w8kbm\" (UID: \"dacc132b-fcda-49a6-a569-4696ea206638\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8de308-7b79454f5d-w8kbm" Oct 30 09:23:01.666130 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:01.666102 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dacc132b-fcda-49a6-a569-4696ea206638-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8de308-7b79454f5d-w8kbm\" (UID: \"dacc132b-fcda-49a6-a569-4696ea206638\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8de308-7b79454f5d-w8kbm" Oct 30 09:23:01.685363 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:01.685339 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8nt5\" (UniqueName: \"kubernetes.io/projected/dacc132b-fcda-49a6-a569-4696ea206638-kube-api-access-j8nt5\") pod \"payload-processing-e2e-ipp-a-8de308-7b79454f5d-w8kbm\" (UID: \"dacc132b-fcda-49a6-a569-4696ea206638\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8de308-7b79454f5d-w8kbm" Oct 30 09:23:01.704231 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:01.704211 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8de308-7b79454f5d-w8kbm" Oct 30 09:23:02.002016 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:23:02.001987 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddacc132b_fcda_49a6_a569_4696ea206638.slice/crio-0d81575f9ef37e05968636a6a9f4c76972345f1c9a47bb411a7b2e69e169fd93 WatchSource:0}: Error finding container 0d81575f9ef37e05968636a6a9f4c76972345f1c9a47bb411a7b2e69e169fd93: Status 404 returned error can't find the container with id 0d81575f9ef37e05968636a6a9f4c76972345f1c9a47bb411a7b2e69e169fd93 Oct 30 09:23:02.003342 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:02.003319 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8de308-7b79454f5d-w8kbm"] Oct 30 09:23:03.081462 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:03.066652 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8de308-7b79454f5d-w8kbm" event={"ID":"dacc132b-fcda-49a6-a569-4696ea206638","Type":"ContainerStarted","Data":"0d81575f9ef37e05968636a6a9f4c76972345f1c9a47bb411a7b2e69e169fd93"} Oct 30 09:23:03.133481 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:03.133413 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-c44644-7db5575f5d-qzlhc" podUID="b8c2c2f2-d8ce-450b-9b00-bb8d96ac68bd" containerName="payload-processing" containerID="cri-o://964fb44b0e8e47ca32cf980d9695b2ab2f6803445f2d5a16dc982788496e4cfb" gracePeriod=25 Oct 30 09:23:03.169347 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:03.169305 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c44644-58d55657ff-tghrj" podUID="987a60a0-3b09-4fc8-aff6-2c5b47c5f225" containerName="payload-pre-processing" containerID="cri-o://5335ad5ccf9291944df03ba3e75b8e6b8a78f75e6b1b89f51e7e41908d251b42" gracePeriod=25 Oct 30 09:23:03.171347 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:03.171318 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-58fadf-667c9dd778-swcxh"] Oct 30 09:23:03.175649 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:03.175633 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58fadf-667c9dd778-swcxh" Oct 30 09:23:03.193017 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:03.192989 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-58fadf-667c9dd778-swcxh"] Oct 30 09:23:03.218438 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:03.212651 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-58fadf-serving-cert\"" Oct 30 09:23:03.228939 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:03.228444 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5aa403a-1de7-40d7-93ef-238dac140203-maas-api-tls\") pod \"maas-api-e2e-api-a-58fadf-667c9dd778-swcxh\" (UID: \"c5aa403a-1de7-40d7-93ef-238dac140203\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58fadf-667c9dd778-swcxh" Oct 30 09:23:03.228939 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:03.228604 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp9gl\" (UniqueName: \"kubernetes.io/projected/c5aa403a-1de7-40d7-93ef-238dac140203-kube-api-access-cp9gl\") pod \"maas-api-e2e-api-a-58fadf-667c9dd778-swcxh\" (UID: \"c5aa403a-1de7-40d7-93ef-238dac140203\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58fadf-667c9dd778-swcxh" Oct 30 09:23:03.228939 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:03.228709 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5aa403a-1de7-40d7-93ef-238dac140203-metrics-certs\") pod \"maas-api-e2e-api-a-58fadf-667c9dd778-swcxh\" (UID: \"c5aa403a-1de7-40d7-93ef-238dac140203\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58fadf-667c9dd778-swcxh" Oct 30 09:23:03.228939 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:03.228794 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5aa403a-1de7-40d7-93ef-238dac140203-openshift-service-ca\") pod \"maas-api-e2e-api-a-58fadf-667c9dd778-swcxh\" (UID: \"c5aa403a-1de7-40d7-93ef-238dac140203\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58fadf-667c9dd778-swcxh" Oct 30 09:23:03.330005 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:03.329963 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5aa403a-1de7-40d7-93ef-238dac140203-metrics-certs\") pod \"maas-api-e2e-api-a-58fadf-667c9dd778-swcxh\" (UID: \"c5aa403a-1de7-40d7-93ef-238dac140203\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58fadf-667c9dd778-swcxh" Oct 30 09:23:03.330202 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:03.330080 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5aa403a-1de7-40d7-93ef-238dac140203-openshift-service-ca\") pod \"maas-api-e2e-api-a-58fadf-667c9dd778-swcxh\" (UID: \"c5aa403a-1de7-40d7-93ef-238dac140203\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58fadf-667c9dd778-swcxh" Oct 30 09:23:03.330288 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:03.330240 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5aa403a-1de7-40d7-93ef-238dac140203-maas-api-tls\") pod \"maas-api-e2e-api-a-58fadf-667c9dd778-swcxh\" (UID: \"c5aa403a-1de7-40d7-93ef-238dac140203\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58fadf-667c9dd778-swcxh" Oct 30 09:23:03.330375 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:03.330357 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp9gl\" (UniqueName: \"kubernetes.io/projected/c5aa403a-1de7-40d7-93ef-238dac140203-kube-api-access-cp9gl\") pod \"maas-api-e2e-api-a-58fadf-667c9dd778-swcxh\" (UID: \"c5aa403a-1de7-40d7-93ef-238dac140203\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58fadf-667c9dd778-swcxh" Oct 30 09:23:03.331424 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:03.331398 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5aa403a-1de7-40d7-93ef-238dac140203-openshift-service-ca\") pod \"maas-api-e2e-api-a-58fadf-667c9dd778-swcxh\" (UID: \"c5aa403a-1de7-40d7-93ef-238dac140203\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58fadf-667c9dd778-swcxh" Oct 30 09:23:03.343480 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:03.335346 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5aa403a-1de7-40d7-93ef-238dac140203-maas-api-tls\") pod \"maas-api-e2e-api-a-58fadf-667c9dd778-swcxh\" (UID: \"c5aa403a-1de7-40d7-93ef-238dac140203\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58fadf-667c9dd778-swcxh" Oct 30 09:23:03.345293 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:03.345268 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5aa403a-1de7-40d7-93ef-238dac140203-metrics-certs\") pod \"maas-api-e2e-api-a-58fadf-667c9dd778-swcxh\" (UID: \"c5aa403a-1de7-40d7-93ef-238dac140203\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58fadf-667c9dd778-swcxh" Oct 30 09:23:03.355245 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:23:03.354544 4562 logging.go:55] [core] [Channel #1770 SubChannel #1772]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.108:50052", ServerName: "10.133.0.108:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.108:50052: connect: connection refused" Oct 30 09:23:03.355245 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:23:03.354645 4562 logging.go:55] [core] [Channel #1769 SubChannel #1771]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.109:50052", ServerName: "10.133.0.109:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.109:50052: connect: connection refused" Oct 30 09:23:03.355476 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:03.355299 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp9gl\" (UniqueName: \"kubernetes.io/projected/c5aa403a-1de7-40d7-93ef-238dac140203-kube-api-access-cp9gl\") pod \"maas-api-e2e-api-a-58fadf-667c9dd778-swcxh\" (UID: \"c5aa403a-1de7-40d7-93ef-238dac140203\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58fadf-667c9dd778-swcxh" Oct 30 09:23:03.633459 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:03.631785 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58fadf-667c9dd778-swcxh" Oct 30 09:23:03.815775 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:03.815730 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c44644-7db5575f5d-qzlhc" Oct 30 09:23:03.816789 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:03.816761 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c44644-58d55657ff-tghrj" Oct 30 09:23:03.891883 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:03.891819 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8c2c2f2-d8ce-450b-9b00-bb8d96ac68bd-kube-api-access-psvd5\" (UniqueName: \"kubernetes.io/projected/b8c2c2f2-d8ce-450b-9b00-bb8d96ac68bd-kube-api-access-psvd5\") pod \"b8c2c2f2-d8ce-450b-9b00-bb8d96ac68bd\" (UID: \"b8c2c2f2-d8ce-450b-9b00-bb8d96ac68bd\") " Oct 30 09:23:03.892206 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:03.892193 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/987a60a0-3b09-4fc8-aff6-2c5b47c5f225-kube-api-access-gpf56\" (UniqueName: \"kubernetes.io/projected/987a60a0-3b09-4fc8-aff6-2c5b47c5f225-kube-api-access-gpf56\") pod \"987a60a0-3b09-4fc8-aff6-2c5b47c5f225\" (UID: \"987a60a0-3b09-4fc8-aff6-2c5b47c5f225\") " Oct 30 09:23:03.892329 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:03.892317 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8c2c2f2-d8ce-450b-9b00-bb8d96ac68bd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8c2c2f2-d8ce-450b-9b00-bb8d96ac68bd-plugins-config-volume\") pod \"b8c2c2f2-d8ce-450b-9b00-bb8d96ac68bd\" (UID: \"b8c2c2f2-d8ce-450b-9b00-bb8d96ac68bd\") " Oct 30 09:23:03.892422 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:03.892409 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/987a60a0-3b09-4fc8-aff6-2c5b47c5f225-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/987a60a0-3b09-4fc8-aff6-2c5b47c5f225-plugins-config-volume\") pod \"987a60a0-3b09-4fc8-aff6-2c5b47c5f225\" (UID: \"987a60a0-3b09-4fc8-aff6-2c5b47c5f225\") " Oct 30 09:23:03.893164 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:03.893146 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/987a60a0-3b09-4fc8-aff6-2c5b47c5f225-plugins-config-volume" pod "987a60a0-3b09-4fc8-aff6-2c5b47c5f225" (UID: "987a60a0-3b09-4fc8-aff6-2c5b47c5f225"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:23:03.893264 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:03.893159 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8c2c2f2-d8ce-450b-9b00-bb8d96ac68bd-plugins-config-volume" pod "b8c2c2f2-d8ce-450b-9b00-bb8d96ac68bd" (UID: "b8c2c2f2-d8ce-450b-9b00-bb8d96ac68bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:23:03.903520 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:03.903492 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8c2c2f2-d8ce-450b-9b00-bb8d96ac68bd-kube-api-access-psvd5" pod "b8c2c2f2-d8ce-450b-9b00-bb8d96ac68bd" (UID: "b8c2c2f2-d8ce-450b-9b00-bb8d96ac68bd"). InnerVolumeSpecName "kube-api-access-psvd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:23:03.905837 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:03.905692 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/987a60a0-3b09-4fc8-aff6-2c5b47c5f225-kube-api-access-gpf56" pod "987a60a0-3b09-4fc8-aff6-2c5b47c5f225" (UID: "987a60a0-3b09-4fc8-aff6-2c5b47c5f225"). InnerVolumeSpecName "kube-api-access-gpf56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:23:03.994666 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:03.994624 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psvd5\" (UniqueName: \"kubernetes.io/projected/b8c2c2f2-d8ce-450b-9b00-bb8d96ac68bd-kube-api-access-psvd5\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:23:03.994666 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:03.994662 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpf56\" (UniqueName: \"kubernetes.io/projected/987a60a0-3b09-4fc8-aff6-2c5b47c5f225-kube-api-access-gpf56\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:23:03.994918 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:03.994678 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8c2c2f2-d8ce-450b-9b00-bb8d96ac68bd-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:23:03.994918 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:03.994693 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/987a60a0-3b09-4fc8-aff6-2c5b47c5f225-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:23:04.091430 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:04.089588 4562 generic.go:356] "Generic (PLEG): container finished" podID="b8c2c2f2-d8ce-450b-9b00-bb8d96ac68bd" containerID="964fb44b0e8e47ca32cf980d9695b2ab2f6803445f2d5a16dc982788496e4cfb" exitCode=0 Oct 30 09:23:04.091430 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:04.089852 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c44644-7db5575f5d-qzlhc" Oct 30 09:23:04.116194 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:04.094508 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c44644-7db5575f5d-qzlhc" event={"ID":"b8c2c2f2-d8ce-450b-9b00-bb8d96ac68bd","Type":"ContainerDied","Data":"964fb44b0e8e47ca32cf980d9695b2ab2f6803445f2d5a16dc982788496e4cfb"} Oct 30 09:23:04.116194 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:04.098024 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c44644-7db5575f5d-qzlhc" event={"ID":"b8c2c2f2-d8ce-450b-9b00-bb8d96ac68bd","Type":"ContainerDied","Data":"569076cf170ee6e4261d12c252b79c261d0abf8d1799398ec34974c60ca90e07"} Oct 30 09:23:04.116194 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:04.098079 4562 scope.go:122] "RemoveContainer" containerID="964fb44b0e8e47ca32cf980d9695b2ab2f6803445f2d5a16dc982788496e4cfb" Oct 30 09:23:04.116194 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:04.101316 4562 generic.go:356] "Generic (PLEG): container finished" podID="987a60a0-3b09-4fc8-aff6-2c5b47c5f225" containerID="5335ad5ccf9291944df03ba3e75b8e6b8a78f75e6b1b89f51e7e41908d251b42" exitCode=0 Oct 30 09:23:04.116194 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:04.101428 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c44644-58d55657ff-tghrj" event={"ID":"987a60a0-3b09-4fc8-aff6-2c5b47c5f225","Type":"ContainerDied","Data":"5335ad5ccf9291944df03ba3e75b8e6b8a78f75e6b1b89f51e7e41908d251b42"} Oct 30 09:23:04.116194 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:04.101476 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c44644-58d55657ff-tghrj" event={"ID":"987a60a0-3b09-4fc8-aff6-2c5b47c5f225","Type":"ContainerDied","Data":"9487962bcfc579a66fe9398168b1734fb6d8ddb4ae3e77b0e1ad14c7c1836234"} Oct 30 09:23:04.116194 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:04.101548 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c44644-58d55657ff-tghrj" Oct 30 09:23:04.161617 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:04.161313 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-58fadf-667c9dd778-swcxh"] Oct 30 09:23:04.208647 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:04.204491 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c44644-7db5575f5d-qzlhc"] Oct 30 09:23:04.217922 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:04.217854 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c44644-7db5575f5d-qzlhc"] Oct 30 09:23:04.219620 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:04.219593 4562 scope.go:122] "RemoveContainer" containerID="964fb44b0e8e47ca32cf980d9695b2ab2f6803445f2d5a16dc982788496e4cfb" Oct 30 09:23:04.225858 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:23:04.225826 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"964fb44b0e8e47ca32cf980d9695b2ab2f6803445f2d5a16dc982788496e4cfb\": container with ID starting with 964fb44b0e8e47ca32cf980d9695b2ab2f6803445f2d5a16dc982788496e4cfb not found: ID does not exist" containerID="964fb44b0e8e47ca32cf980d9695b2ab2f6803445f2d5a16dc982788496e4cfb" Oct 30 09:23:04.225973 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:04.225868 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"964fb44b0e8e47ca32cf980d9695b2ab2f6803445f2d5a16dc982788496e4cfb"} err="failed to get container status \"964fb44b0e8e47ca32cf980d9695b2ab2f6803445f2d5a16dc982788496e4cfb\": rpc error: code = NotFound desc = could not find container \"964fb44b0e8e47ca32cf980d9695b2ab2f6803445f2d5a16dc982788496e4cfb\": container with ID starting with 964fb44b0e8e47ca32cf980d9695b2ab2f6803445f2d5a16dc982788496e4cfb not found: ID does not exist" Oct 30 09:23:04.225973 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:04.225893 4562 scope.go:122] "RemoveContainer" containerID="5335ad5ccf9291944df03ba3e75b8e6b8a78f75e6b1b89f51e7e41908d251b42" Oct 30 09:23:04.263514 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:04.263474 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c44644-58d55657ff-tghrj"] Oct 30 09:23:04.274995 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:04.274968 4562 scope.go:122] "RemoveContainer" containerID="5335ad5ccf9291944df03ba3e75b8e6b8a78f75e6b1b89f51e7e41908d251b42" Oct 30 09:23:04.276329 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:23:04.276299 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5335ad5ccf9291944df03ba3e75b8e6b8a78f75e6b1b89f51e7e41908d251b42\": container with ID starting with 5335ad5ccf9291944df03ba3e75b8e6b8a78f75e6b1b89f51e7e41908d251b42 not found: ID does not exist" containerID="5335ad5ccf9291944df03ba3e75b8e6b8a78f75e6b1b89f51e7e41908d251b42" Oct 30 09:23:04.276517 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:04.276489 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5335ad5ccf9291944df03ba3e75b8e6b8a78f75e6b1b89f51e7e41908d251b42"} err="failed to get container status \"5335ad5ccf9291944df03ba3e75b8e6b8a78f75e6b1b89f51e7e41908d251b42\": rpc error: code = NotFound desc = could not find container \"5335ad5ccf9291944df03ba3e75b8e6b8a78f75e6b1b89f51e7e41908d251b42\": container with ID starting with 5335ad5ccf9291944df03ba3e75b8e6b8a78f75e6b1b89f51e7e41908d251b42 not found: ID does not exist" Oct 30 09:23:04.277129 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:04.277110 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c44644-58d55657ff-tghrj"] Oct 30 09:23:04.362951 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:04.361942 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c44644-58d55657ff-tghrj" podUID="987a60a0-3b09-4fc8-aff6-2c5b47c5f225" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.109:50052\" within 1s: context deadline exceeded" Oct 30 09:23:04.362951 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:23:04.362308 4562 logging.go:55] [core] [Channel #1769 SubChannel #1771]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.109:50052", ServerName: "10.133.0.109:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.109:50052: operation was canceled" Oct 30 09:23:04.362951 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:04.362412 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-a-c44644-7db5575f5d-qzlhc" podUID="b8c2c2f2-d8ce-450b-9b00-bb8d96ac68bd" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.108:50052\" within 1s: context deadline exceeded" Oct 30 09:23:04.362951 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:23:04.362514 4562 logging.go:55] [core] [Channel #1770 SubChannel #1772]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.108:50052", ServerName: "10.133.0.108:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.108:50052: operation was canceled" Oct 30 09:23:05.113617 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:05.113566 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58fadf-667c9dd778-swcxh" event={"ID":"c5aa403a-1de7-40d7-93ef-238dac140203","Type":"ContainerStarted","Data":"aa138d0fd4c7087e32a534c3562253df92bd692f9a7d445dc04f7a72cb3a184a"} Oct 30 09:23:05.114084 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:05.113637 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58fadf-667c9dd778-swcxh" event={"ID":"c5aa403a-1de7-40d7-93ef-238dac140203","Type":"ContainerStarted","Data":"9a4e0f181691f9237d842270fb5b58487077abbff76a05dfdbf14438aa7ef3ff"} Oct 30 09:23:05.114084 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:05.113692 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58fadf-667c9dd778-swcxh" Oct 30 09:23:05.156716 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:05.156653 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58fadf-667c9dd778-swcxh" podStartSLOduration=1.68141667 podStartE2EDuration="2.156639218s" totalImagesPullingTime="475.222548ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:23:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:23:05.156512964 +0000 UTC m=+1745.418073944" watchObservedRunningTime="2026-10-30 09:23:05.156639218 +0000 UTC m=+1745.418200216" Oct 30 09:23:05.879013 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:05.878973 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="987a60a0-3b09-4fc8-aff6-2c5b47c5f225" path="/var/lib/kubelet/pods/987a60a0-3b09-4fc8-aff6-2c5b47c5f225/volumes" Oct 30 09:23:05.879621 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:05.879597 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8c2c2f2-d8ce-450b-9b00-bb8d96ac68bd" path="/var/lib/kubelet/pods/b8c2c2f2-d8ce-450b-9b00-bb8d96ac68bd/volumes" Oct 30 09:23:06.124729 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:06.124687 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8de308-7b79454f5d-w8kbm" event={"ID":"dacc132b-fcda-49a6-a569-4696ea206638","Type":"ContainerStarted","Data":"65134e9f0a92a5edee1ccf07bc5d5397f1d8fc12b6cd48efffadc3db52ad2805"} Oct 30 09:23:06.125184 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:06.124972 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-8de308-7b79454f5d-w8kbm" Oct 30 09:23:06.173780 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:06.173662 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-8de308-7b79454f5d-w8kbm" podStartSLOduration=1.917214175 podStartE2EDuration="5.173648726s" totalImagesPullingTime="3.256434551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:23:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:23:06.17147181 +0000 UTC m=+1746.433032821" watchObservedRunningTime="2026-10-30 09:23:06.173648726 +0000 UTC m=+1746.435209714" Oct 30 09:23:10.045512 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:10.043891 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8eada0cf-7c8545fd75-cfnvf"] Oct 30 09:23:10.045512 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:10.044168 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eada0cf-7c8545fd75-cfnvf" podUID="73717db6-2d44-4474-93f1-df1a19589a47" containerName="maas-api" containerID="cri-o://916befdf2164830e39cb74928c6a8fcea85a03073cb0331c7d5ad6bdd45ea022" gracePeriod=30 Oct 30 09:23:10.190832 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:10.190794 4562 generic.go:356] "Generic (PLEG): container finished" podID="73717db6-2d44-4474-93f1-df1a19589a47" containerID="916befdf2164830e39cb74928c6a8fcea85a03073cb0331c7d5ad6bdd45ea022" exitCode=0 Oct 30 09:23:10.190991 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:10.190872 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eada0cf-7c8545fd75-cfnvf" event={"ID":"73717db6-2d44-4474-93f1-df1a19589a47","Type":"ContainerDied","Data":"916befdf2164830e39cb74928c6a8fcea85a03073cb0331c7d5ad6bdd45ea022"} Oct 30 09:23:10.776584 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:10.776552 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eada0cf-7c8545fd75-cfnvf" Oct 30 09:23:10.848282 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:10.844739 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73717db6-2d44-4474-93f1-df1a19589a47-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73717db6-2d44-4474-93f1-df1a19589a47-openshift-service-ca\") pod \"73717db6-2d44-4474-93f1-df1a19589a47\" (UID: \"73717db6-2d44-4474-93f1-df1a19589a47\") " Oct 30 09:23:10.848282 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:10.844825 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73717db6-2d44-4474-93f1-df1a19589a47-metrics-certs\" (UniqueName: \"kubernetes.io/secret/73717db6-2d44-4474-93f1-df1a19589a47-metrics-certs\") pod \"73717db6-2d44-4474-93f1-df1a19589a47\" (UID: \"73717db6-2d44-4474-93f1-df1a19589a47\") " Oct 30 09:23:10.848282 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:10.844866 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73717db6-2d44-4474-93f1-df1a19589a47-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73717db6-2d44-4474-93f1-df1a19589a47-maas-api-tls\") pod \"73717db6-2d44-4474-93f1-df1a19589a47\" (UID: \"73717db6-2d44-4474-93f1-df1a19589a47\") " Oct 30 09:23:10.848282 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:10.844982 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73717db6-2d44-4474-93f1-df1a19589a47-kube-api-access-kb8gp\" (UniqueName: \"kubernetes.io/projected/73717db6-2d44-4474-93f1-df1a19589a47-kube-api-access-kb8gp\") pod \"73717db6-2d44-4474-93f1-df1a19589a47\" (UID: \"73717db6-2d44-4474-93f1-df1a19589a47\") " Oct 30 09:23:10.848282 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:10.845111 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73717db6-2d44-4474-93f1-df1a19589a47-openshift-service-ca" pod "73717db6-2d44-4474-93f1-df1a19589a47" (UID: "73717db6-2d44-4474-93f1-df1a19589a47"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:23:10.848282 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:10.845772 4562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73717db6-2d44-4474-93f1-df1a19589a47-openshift-service-ca\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:23:10.849600 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:10.849567 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73717db6-2d44-4474-93f1-df1a19589a47-maas-api-tls" pod "73717db6-2d44-4474-93f1-df1a19589a47" (UID: "73717db6-2d44-4474-93f1-df1a19589a47"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:23:10.852882 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:10.852849 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73717db6-2d44-4474-93f1-df1a19589a47-metrics-certs" pod "73717db6-2d44-4474-93f1-df1a19589a47" (UID: "73717db6-2d44-4474-93f1-df1a19589a47"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:23:10.855950 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:10.855923 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73717db6-2d44-4474-93f1-df1a19589a47-kube-api-access-kb8gp" pod "73717db6-2d44-4474-93f1-df1a19589a47" (UID: "73717db6-2d44-4474-93f1-df1a19589a47"). InnerVolumeSpecName "kube-api-access-kb8gp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:23:10.960571 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:10.960477 4562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73717db6-2d44-4474-93f1-df1a19589a47-metrics-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:23:10.960571 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:10.960511 4562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73717db6-2d44-4474-93f1-df1a19589a47-maas-api-tls\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:23:10.960571 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:10.960526 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kb8gp\" (UniqueName: \"kubernetes.io/projected/73717db6-2d44-4474-93f1-df1a19589a47-kube-api-access-kb8gp\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:23:11.130232 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:11.130200 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58fadf-667c9dd778-swcxh" Oct 30 09:23:11.248256 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:11.248172 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eada0cf-7c8545fd75-cfnvf" event={"ID":"73717db6-2d44-4474-93f1-df1a19589a47","Type":"ContainerDied","Data":"ec55dd15c4178ab7e111089e1354622c79907ae2fac8f91ad9af80e8a2979c0b"} Oct 30 09:23:11.248256 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:11.248236 4562 scope.go:122] "RemoveContainer" containerID="916befdf2164830e39cb74928c6a8fcea85a03073cb0331c7d5ad6bdd45ea022" Oct 30 09:23:11.248529 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:11.248491 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eada0cf-7c8545fd75-cfnvf" Oct 30 09:23:11.309273 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:11.309241 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8eada0cf-7c8545fd75-cfnvf"] Oct 30 09:23:11.312531 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:11.312509 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8eada0cf-7c8545fd75-cfnvf"] Oct 30 09:23:11.876208 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:11.876178 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73717db6-2d44-4474-93f1-df1a19589a47" path="/var/lib/kubelet/pods/73717db6-2d44-4474-93f1-df1a19589a47/volumes" Oct 30 09:23:12.532724 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:12.532691 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8f40a3-68bb7574dc-mdb4m"] Oct 30 09:23:12.894635 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:12.894565 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c44644-76499c9b99-ld6jd"] Oct 30 09:23:12.895208 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:12.895194 4562 cpu_manager.go:414] "containerMap: removing container" podUID="73717db6-2d44-4474-93f1-df1a19589a47" containerName="maas-api" Oct 30 09:23:12.895252 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:12.895210 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73717db6-2d44-4474-93f1-df1a19589a47" containerName="maas-api" Oct 30 09:23:12.895252 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:12.895230 4562 cpu_manager.go:414] "containerMap: removing container" podUID="b8c2c2f2-d8ce-450b-9b00-bb8d96ac68bd" containerName="payload-processing" Oct 30 09:23:12.895252 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:12.895236 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8c2c2f2-d8ce-450b-9b00-bb8d96ac68bd" containerName="payload-processing" Oct 30 09:23:12.895252 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:12.895245 4562 cpu_manager.go:414] "containerMap: removing container" podUID="987a60a0-3b09-4fc8-aff6-2c5b47c5f225" containerName="payload-pre-processing" Oct 30 09:23:12.895252 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:12.895251 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="987a60a0-3b09-4fc8-aff6-2c5b47c5f225" containerName="payload-pre-processing" Oct 30 09:23:12.895434 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:12.895363 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="73717db6-2d44-4474-93f1-df1a19589a47" containerName="maas-api" Oct 30 09:23:12.895434 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:12.895372 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8c2c2f2-d8ce-450b-9b00-bb8d96ac68bd" containerName="payload-processing" Oct 30 09:23:12.895434 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:12.895394 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="987a60a0-3b09-4fc8-aff6-2c5b47c5f225" containerName="payload-pre-processing" Oct 30 09:23:12.899505 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:12.899484 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c44644-76499c9b99-ld6jd" Oct 30 09:23:12.917156 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:12.917121 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c44644-76499c9b99-ld6jd"] Oct 30 09:23:12.917432 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:12.917393 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-c44644\"" Oct 30 09:23:12.986716 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:12.986682 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-c44644-dockercfg-kmx54\"" Oct 30 09:23:13.106881 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:13.073673 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9dfb6a8-2d84-49d2-b5b3-3289259cc7bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c44644-76499c9b99-ld6jd\" (UID: \"f9dfb6a8-2d84-49d2-b5b3-3289259cc7bf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c44644-76499c9b99-ld6jd" Oct 30 09:23:13.114404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:13.108706 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzswb\" (UniqueName: \"kubernetes.io/projected/f9dfb6a8-2d84-49d2-b5b3-3289259cc7bf-kube-api-access-tzswb\") pod \"payload-pre-processing-e2e-ipp-a-c44644-76499c9b99-ld6jd\" (UID: \"f9dfb6a8-2d84-49d2-b5b3-3289259cc7bf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c44644-76499c9b99-ld6jd" Oct 30 09:23:13.210448 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:13.210334 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9dfb6a8-2d84-49d2-b5b3-3289259cc7bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c44644-76499c9b99-ld6jd\" (UID: \"f9dfb6a8-2d84-49d2-b5b3-3289259cc7bf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c44644-76499c9b99-ld6jd" Oct 30 09:23:13.221049 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:13.210832 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzswb\" (UniqueName: \"kubernetes.io/projected/f9dfb6a8-2d84-49d2-b5b3-3289259cc7bf-kube-api-access-tzswb\") pod \"payload-pre-processing-e2e-ipp-a-c44644-76499c9b99-ld6jd\" (UID: \"f9dfb6a8-2d84-49d2-b5b3-3289259cc7bf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c44644-76499c9b99-ld6jd" Oct 30 09:23:13.222941 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:13.222909 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9dfb6a8-2d84-49d2-b5b3-3289259cc7bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c44644-76499c9b99-ld6jd\" (UID: \"f9dfb6a8-2d84-49d2-b5b3-3289259cc7bf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c44644-76499c9b99-ld6jd" Oct 30 09:23:13.393441 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:13.349183 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzswb\" (UniqueName: \"kubernetes.io/projected/f9dfb6a8-2d84-49d2-b5b3-3289259cc7bf-kube-api-access-tzswb\") pod \"payload-pre-processing-e2e-ipp-a-c44644-76499c9b99-ld6jd\" (UID: \"f9dfb6a8-2d84-49d2-b5b3-3289259cc7bf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c44644-76499c9b99-ld6jd" Oct 30 09:23:13.519921 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:13.519057 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c44644-76499c9b99-ld6jd" Oct 30 09:23:14.051557 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:14.050880 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c44644-76499c9b99-ld6jd"] Oct 30 09:23:14.530746 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:14.530705 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c44644-76499c9b99-ld6jd" event={"ID":"f9dfb6a8-2d84-49d2-b5b3-3289259cc7bf","Type":"ContainerStarted","Data":"8211361f376d5ac49eb8df59cd69d9aeb51ccecf3928cea1b2b6a47ec4bb2d21"} Oct 30 09:23:14.530939 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:14.530773 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c44644-76499c9b99-ld6jd" event={"ID":"f9dfb6a8-2d84-49d2-b5b3-3289259cc7bf","Type":"ContainerStarted","Data":"02be1e00c9eaa742160b25ee8ee3b5205b6c05c2fc0110937ebaf3779c9b6660"} Oct 30 09:23:14.530939 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:14.530807 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c44644-76499c9b99-ld6jd" Oct 30 09:23:15.539164 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:15.539075 4562 generic.go:356] "Generic (PLEG): container finished" podID="a104dc46-cd60-483a-b159-4512b460125a" containerID="3b6eb58a9768a11f8e18b1b69dc60bc1d2ac09c48dff91ca04ad0667be45f33a" exitCode=0 Oct 30 09:23:15.539708 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:15.539156 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc" event={"ID":"a104dc46-cd60-483a-b159-4512b460125a","Type":"ContainerDied","Data":"3b6eb58a9768a11f8e18b1b69dc60bc1d2ac09c48dff91ca04ad0667be45f33a"} Oct 30 09:23:15.539781 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:15.539762 4562 scope.go:122] "RemoveContainer" containerID="3b6eb58a9768a11f8e18b1b69dc60bc1d2ac09c48dff91ca04ad0667be45f33a" Oct 30 09:23:15.541432 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:15.541401 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:23:15.541509 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:15.541480 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:23:15.541545 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:15.541521 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:23:15.634673 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:15.634594 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c44644-76499c9b99-ld6jd" podStartSLOduration=3.634575264 podStartE2EDuration="3.634575264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:23:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:23:14.602581786 +0000 UTC m=+1754.864142775" watchObservedRunningTime="2026-10-30 09:23:15.634575264 +0000 UTC m=+1755.896136254" Oct 30 09:23:16.076834 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:16.076800 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc" Oct 30 09:23:16.575991 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:16.575958 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc" event={"ID":"a104dc46-cd60-483a-b159-4512b460125a","Type":"ContainerStarted","Data":"21cd33cb88caab9445664f94ddb97d7a765690eecc3b37ee1b4dd2401ab1a0ef"} Oct 30 09:23:16.920869 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:16.920790 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc" Oct 30 09:23:16.922139 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:16.922116 4562 patch_prober.go:29] interesting pod/e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 30 09:23:16.922255 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:16.922167 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc" podUID="a104dc46-cd60-483a-b159-4512b460125a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 30 09:23:17.168971 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:17.168590 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-8de308-7b79454f5d-w8kbm" Oct 30 09:23:17.533346 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:17.533306 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f40a3-68bb7574dc-mdb4m" podUID="f182c94b-511e-4b0b-b236-236001563a92" containerName="payload-pre-processing" containerID="cri-o://94adc0cf3b23d8cee1a4bbea30c361033954e7c831e937c8917fdb8301d06e83" gracePeriod=25 Oct 30 09:23:17.882127 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:17.882101 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f40a3-68bb7574dc-mdb4m" Oct 30 09:23:17.894840 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:17.894795 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f182c94b-511e-4b0b-b236-236001563a92-kube-api-access-frc7n\" (UniqueName: \"kubernetes.io/projected/f182c94b-511e-4b0b-b236-236001563a92-kube-api-access-frc7n\") pod \"f182c94b-511e-4b0b-b236-236001563a92\" (UID: \"f182c94b-511e-4b0b-b236-236001563a92\") " Oct 30 09:23:17.894996 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:17.894974 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f182c94b-511e-4b0b-b236-236001563a92-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f182c94b-511e-4b0b-b236-236001563a92-plugins-config-volume\") pod \"f182c94b-511e-4b0b-b236-236001563a92\" (UID: \"f182c94b-511e-4b0b-b236-236001563a92\") " Oct 30 09:23:17.895437 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:17.895410 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f182c94b-511e-4b0b-b236-236001563a92-plugins-config-volume" pod "f182c94b-511e-4b0b-b236-236001563a92" (UID: "f182c94b-511e-4b0b-b236-236001563a92"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:23:17.895917 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:17.895901 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f182c94b-511e-4b0b-b236-236001563a92-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:23:17.900561 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:17.900537 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f182c94b-511e-4b0b-b236-236001563a92-kube-api-access-frc7n" pod "f182c94b-511e-4b0b-b236-236001563a92" (UID: "f182c94b-511e-4b0b-b236-236001563a92"). InnerVolumeSpecName "kube-api-access-frc7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:23:17.926372 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:17.920700 4562 patch_prober.go:29] interesting pod/e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 30 09:23:17.926372 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:17.920749 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc" podUID="a104dc46-cd60-483a-b159-4512b460125a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 30 09:23:17.997207 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:17.997163 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frc7n\" (UniqueName: \"kubernetes.io/projected/f182c94b-511e-4b0b-b236-236001563a92-kube-api-access-frc7n\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:23:18.590981 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:18.590944 4562 generic.go:356] "Generic (PLEG): container finished" podID="f182c94b-511e-4b0b-b236-236001563a92" containerID="94adc0cf3b23d8cee1a4bbea30c361033954e7c831e937c8917fdb8301d06e83" exitCode=0 Oct 30 09:23:18.591284 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:18.591009 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f40a3-68bb7574dc-mdb4m" Oct 30 09:23:18.591492 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:18.591026 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f40a3-68bb7574dc-mdb4m" event={"ID":"f182c94b-511e-4b0b-b236-236001563a92","Type":"ContainerDied","Data":"94adc0cf3b23d8cee1a4bbea30c361033954e7c831e937c8917fdb8301d06e83"} Oct 30 09:23:18.591492 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:18.591455 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f40a3-68bb7574dc-mdb4m" event={"ID":"f182c94b-511e-4b0b-b236-236001563a92","Type":"ContainerDied","Data":"5e83e2a85dc67c5fdf4d324a841947b6bb88018b3ca881fb6454a4174f7c56a6"} Oct 30 09:23:18.591609 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:18.591493 4562 scope.go:122] "RemoveContainer" containerID="94adc0cf3b23d8cee1a4bbea30c361033954e7c831e937c8917fdb8301d06e83" Oct 30 09:23:18.602355 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:18.602339 4562 scope.go:122] "RemoveContainer" containerID="94adc0cf3b23d8cee1a4bbea30c361033954e7c831e937c8917fdb8301d06e83" Oct 30 09:23:18.602654 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:23:18.602637 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94adc0cf3b23d8cee1a4bbea30c361033954e7c831e937c8917fdb8301d06e83\": container with ID starting with 94adc0cf3b23d8cee1a4bbea30c361033954e7c831e937c8917fdb8301d06e83 not found: ID does not exist" containerID="94adc0cf3b23d8cee1a4bbea30c361033954e7c831e937c8917fdb8301d06e83" Oct 30 09:23:18.602699 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:18.602662 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94adc0cf3b23d8cee1a4bbea30c361033954e7c831e937c8917fdb8301d06e83"} err="failed to get container status \"94adc0cf3b23d8cee1a4bbea30c361033954e7c831e937c8917fdb8301d06e83\": rpc error: code = NotFound desc = could not find container \"94adc0cf3b23d8cee1a4bbea30c361033954e7c831e937c8917fdb8301d06e83\": container with ID starting with 94adc0cf3b23d8cee1a4bbea30c361033954e7c831e937c8917fdb8301d06e83 not found: ID does not exist" Oct 30 09:23:18.665649 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:18.665621 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8f40a3-68bb7574dc-mdb4m"] Oct 30 09:23:18.673687 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:18.673647 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8f40a3-68bb7574dc-mdb4m"] Oct 30 09:23:18.937639 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:18.937547 4562 patch_prober.go:29] interesting pod/e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 30 09:23:18.938297 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:18.938264 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc" podUID="a104dc46-cd60-483a-b159-4512b460125a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 30 09:23:19.940429 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:19.939014 4562 patch_prober.go:29] interesting pod/e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 30 09:23:19.940429 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:19.939072 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc" podUID="a104dc46-cd60-483a-b159-4512b460125a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 30 09:23:19.956418 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:19.951091 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f182c94b-511e-4b0b-b236-236001563a92" path="/var/lib/kubelet/pods/f182c94b-511e-4b0b-b236-236001563a92/volumes" Oct 30 09:23:20.921375 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:20.921340 4562 patch_prober.go:29] interesting pod/e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 30 09:23:20.921596 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:20.921425 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc" podUID="a104dc46-cd60-483a-b159-4512b460125a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 30 09:23:21.964502 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:21.964472 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc" Oct 30 09:23:21.964990 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:21.964746 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc" Oct 30 09:23:21.966412 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:21.966346 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc" Oct 30 09:23:24.078518 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:24.078480 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cbe974-d65fcff9-6hvv7"] Oct 30 09:23:24.079499 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:24.079481 4562 cpu_manager.go:414] "containerMap: removing container" podUID="f182c94b-511e-4b0b-b236-236001563a92" containerName="payload-pre-processing" Oct 30 09:23:24.079584 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:24.079500 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f182c94b-511e-4b0b-b236-236001563a92" containerName="payload-pre-processing" Oct 30 09:23:24.079693 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:24.079681 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="f182c94b-511e-4b0b-b236-236001563a92" containerName="payload-pre-processing" Oct 30 09:23:24.084245 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:24.084224 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbe974-d65fcff9-6hvv7" Oct 30 09:23:24.100835 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:24.100813 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cbe974-d65fcff9-6hvv7"] Oct 30 09:23:24.107631 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:24.107598 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-cbe974-dockercfg-45xln\"" Oct 30 09:23:24.107744 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:24.107685 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-cbe974\"" Oct 30 09:23:24.155492 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:24.155464 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cbe974-b85c966d6-6fqhj"] Oct 30 09:23:24.159746 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:24.159726 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbe974-b85c966d6-6fqhj" Oct 30 09:23:24.167426 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:24.167402 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2754d28e-7bd7-457c-b087-6074d969e0d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-cbe974-d65fcff9-6hvv7\" (UID: \"2754d28e-7bd7-457c-b087-6074d969e0d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbe974-d65fcff9-6hvv7" Oct 30 09:23:24.167563 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:24.167539 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4d5n\" (UniqueName: \"kubernetes.io/projected/2754d28e-7bd7-457c-b087-6074d969e0d7-kube-api-access-r4d5n\") pod \"payload-pre-processing-e2e-ipp-b-cbe974-d65fcff9-6hvv7\" (UID: \"2754d28e-7bd7-457c-b087-6074d969e0d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbe974-d65fcff9-6hvv7" Oct 30 09:23:24.177349 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:24.177322 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cbe974-b85c966d6-6fqhj"] Oct 30 09:23:24.280438 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:24.280396 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sdcz\" (UniqueName: \"kubernetes.io/projected/12a110c3-50f9-4443-96cc-fbe6245adc61-kube-api-access-5sdcz\") pod \"payload-processing-e2e-ipp-b-cbe974-b85c966d6-6fqhj\" (UID: \"12a110c3-50f9-4443-96cc-fbe6245adc61\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cbe974-b85c966d6-6fqhj" Oct 30 09:23:24.280887 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:24.280863 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12a110c3-50f9-4443-96cc-fbe6245adc61-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-cbe974-b85c966d6-6fqhj\" (UID: \"12a110c3-50f9-4443-96cc-fbe6245adc61\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cbe974-b85c966d6-6fqhj" Oct 30 09:23:24.281176 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:24.281154 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2754d28e-7bd7-457c-b087-6074d969e0d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-cbe974-d65fcff9-6hvv7\" (UID: \"2754d28e-7bd7-457c-b087-6074d969e0d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbe974-d65fcff9-6hvv7" Oct 30 09:23:24.315578 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:24.315542 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4d5n\" (UniqueName: \"kubernetes.io/projected/2754d28e-7bd7-457c-b087-6074d969e0d7-kube-api-access-r4d5n\") pod \"payload-pre-processing-e2e-ipp-b-cbe974-d65fcff9-6hvv7\" (UID: \"2754d28e-7bd7-457c-b087-6074d969e0d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbe974-d65fcff9-6hvv7" Oct 30 09:23:24.317478 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:24.282151 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2754d28e-7bd7-457c-b087-6074d969e0d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-cbe974-d65fcff9-6hvv7\" (UID: \"2754d28e-7bd7-457c-b087-6074d969e0d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbe974-d65fcff9-6hvv7" Oct 30 09:23:24.376938 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:24.376854 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4d5n\" (UniqueName: \"kubernetes.io/projected/2754d28e-7bd7-457c-b087-6074d969e0d7-kube-api-access-r4d5n\") pod \"payload-pre-processing-e2e-ipp-b-cbe974-d65fcff9-6hvv7\" (UID: \"2754d28e-7bd7-457c-b087-6074d969e0d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbe974-d65fcff9-6hvv7" Oct 30 09:23:24.403153 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:24.403124 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbe974-d65fcff9-6hvv7" Oct 30 09:23:24.449425 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:24.449367 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sdcz\" (UniqueName: \"kubernetes.io/projected/12a110c3-50f9-4443-96cc-fbe6245adc61-kube-api-access-5sdcz\") pod \"payload-processing-e2e-ipp-b-cbe974-b85c966d6-6fqhj\" (UID: \"12a110c3-50f9-4443-96cc-fbe6245adc61\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cbe974-b85c966d6-6fqhj" Oct 30 09:23:24.466787 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:24.453237 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12a110c3-50f9-4443-96cc-fbe6245adc61-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-cbe974-b85c966d6-6fqhj\" (UID: \"12a110c3-50f9-4443-96cc-fbe6245adc61\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cbe974-b85c966d6-6fqhj" Oct 30 09:23:24.477985 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:24.466428 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12a110c3-50f9-4443-96cc-fbe6245adc61-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-cbe974-b85c966d6-6fqhj\" (UID: \"12a110c3-50f9-4443-96cc-fbe6245adc61\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cbe974-b85c966d6-6fqhj" Oct 30 09:23:24.510076 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:24.510036 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sdcz\" (UniqueName: \"kubernetes.io/projected/12a110c3-50f9-4443-96cc-fbe6245adc61-kube-api-access-5sdcz\") pod \"payload-processing-e2e-ipp-b-cbe974-b85c966d6-6fqhj\" (UID: \"12a110c3-50f9-4443-96cc-fbe6245adc61\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cbe974-b85c966d6-6fqhj" Oct 30 09:23:24.775786 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:24.775758 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbe974-b85c966d6-6fqhj" Oct 30 09:23:24.810777 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:23:24.810745 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2754d28e_7bd7_457c_b087_6074d969e0d7.slice/crio-171c54dc78061553f87dc0a0702fcb876228c07d1709da91f61399ebede50146 WatchSource:0}: Error finding container 171c54dc78061553f87dc0a0702fcb876228c07d1709da91f61399ebede50146: Status 404 returned error can't find the container with id 171c54dc78061553f87dc0a0702fcb876228c07d1709da91f61399ebede50146 Oct 30 09:23:24.811713 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:24.811689 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cbe974-d65fcff9-6hvv7"] Oct 30 09:23:25.085584 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:23:25.085557 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12a110c3_50f9_4443_96cc_fbe6245adc61.slice/crio-c682103b7b659c95635a5ac4fff23747be2289f8cad8bd922684c661555d0ff4 WatchSource:0}: Error finding container c682103b7b659c95635a5ac4fff23747be2289f8cad8bd922684c661555d0ff4: Status 404 returned error can't find the container with id c682103b7b659c95635a5ac4fff23747be2289f8cad8bd922684c661555d0ff4 Oct 30 09:23:25.087182 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:25.087152 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cbe974-b85c966d6-6fqhj"] Oct 30 09:23:25.542079 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:25.542050 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c44644-76499c9b99-ld6jd" Oct 30 09:23:25.659222 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:25.659185 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbe974-b85c966d6-6fqhj" event={"ID":"12a110c3-50f9-4443-96cc-fbe6245adc61","Type":"ContainerStarted","Data":"50449a1654ae1bda53eb3b470bec3507f1ca898643a1ba851d51719d40da21e8"} Oct 30 09:23:25.659434 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:25.659255 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbe974-b85c966d6-6fqhj" Oct 30 09:23:25.659434 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:25.659279 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbe974-b85c966d6-6fqhj" event={"ID":"12a110c3-50f9-4443-96cc-fbe6245adc61","Type":"ContainerStarted","Data":"c682103b7b659c95635a5ac4fff23747be2289f8cad8bd922684c661555d0ff4"} Oct 30 09:23:25.660993 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:25.660964 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbe974-d65fcff9-6hvv7" event={"ID":"2754d28e-7bd7-457c-b087-6074d969e0d7","Type":"ContainerStarted","Data":"6e75ebee0e55aecfa0154ecf834b1b8d66cbdc2a1d52ede50ccd5ff901c11b53"} Oct 30 09:23:25.661112 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:25.661007 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbe974-d65fcff9-6hvv7" event={"ID":"2754d28e-7bd7-457c-b087-6074d969e0d7","Type":"ContainerStarted","Data":"171c54dc78061553f87dc0a0702fcb876228c07d1709da91f61399ebede50146"} Oct 30 09:23:25.661112 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:25.661029 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbe974-d65fcff9-6hvv7" Oct 30 09:23:25.746971 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:25.744607 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbe974-b85c966d6-6fqhj" podStartSLOduration=1.744585642 podStartE2EDuration="1.744585642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:23:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:23:25.715725229 +0000 UTC m=+1765.977286218" watchObservedRunningTime="2026-10-30 09:23:25.744585642 +0000 UTC m=+1766.006146633" Oct 30 09:23:25.793357 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:25.793236 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbe974-d65fcff9-6hvv7" podStartSLOduration=1.793211261 podStartE2EDuration="1.793211261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:23:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:23:25.771783712 +0000 UTC m=+1766.033344701" watchObservedRunningTime="2026-10-30 09:23:25.793211261 +0000 UTC m=+1766.054772250" Oct 30 09:23:28.603390 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:23:28.603339 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:23:30.340404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:30.338724 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8f40a3-84f8df4f47-6p5c8"] Oct 30 09:23:30.400509 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:30.396534 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f40a3-84f8df4f47-6p5c8" Oct 30 09:23:30.424592 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:30.412585 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8f40a3-84f8df4f47-6p5c8"] Oct 30 09:23:30.492301 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:30.492271 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-8f40a3\"" Oct 30 09:23:30.497594 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:30.497568 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-8f40a3-dockercfg-nn5qf\"" Oct 30 09:23:30.546564 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:30.542275 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45bmd\" (UniqueName: \"kubernetes.io/projected/bdfbe6ee-d5c2-48e6-a7d8-a05ea3bc8e01-kube-api-access-45bmd\") pod \"payload-pre-processing-e2e-ipp-a-8f40a3-84f8df4f47-6p5c8\" (UID: \"bdfbe6ee-d5c2-48e6-a7d8-a05ea3bc8e01\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f40a3-84f8df4f47-6p5c8" Oct 30 09:23:30.587070 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:30.578884 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdfbe6ee-d5c2-48e6-a7d8-a05ea3bc8e01-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8f40a3-84f8df4f47-6p5c8\" (UID: \"bdfbe6ee-d5c2-48e6-a7d8-a05ea3bc8e01\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f40a3-84f8df4f47-6p5c8" Oct 30 09:23:30.754841 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:30.754794 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45bmd\" (UniqueName: \"kubernetes.io/projected/bdfbe6ee-d5c2-48e6-a7d8-a05ea3bc8e01-kube-api-access-45bmd\") pod \"payload-pre-processing-e2e-ipp-a-8f40a3-84f8df4f47-6p5c8\" (UID: \"bdfbe6ee-d5c2-48e6-a7d8-a05ea3bc8e01\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f40a3-84f8df4f47-6p5c8" Oct 30 09:23:30.755056 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:30.754994 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdfbe6ee-d5c2-48e6-a7d8-a05ea3bc8e01-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8f40a3-84f8df4f47-6p5c8\" (UID: \"bdfbe6ee-d5c2-48e6-a7d8-a05ea3bc8e01\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f40a3-84f8df4f47-6p5c8" Oct 30 09:23:30.755756 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:30.755733 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdfbe6ee-d5c2-48e6-a7d8-a05ea3bc8e01-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8f40a3-84f8df4f47-6p5c8\" (UID: \"bdfbe6ee-d5c2-48e6-a7d8-a05ea3bc8e01\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f40a3-84f8df4f47-6p5c8" Oct 30 09:23:30.780334 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:30.778645 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45bmd\" (UniqueName: \"kubernetes.io/projected/bdfbe6ee-d5c2-48e6-a7d8-a05ea3bc8e01-kube-api-access-45bmd\") pod \"payload-pre-processing-e2e-ipp-a-8f40a3-84f8df4f47-6p5c8\" (UID: \"bdfbe6ee-d5c2-48e6-a7d8-a05ea3bc8e01\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f40a3-84f8df4f47-6p5c8" Oct 30 09:23:30.786070 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:30.786043 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f"] Oct 30 09:23:30.806399 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:30.804883 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f" Oct 30 09:23:30.816834 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:30.816807 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f"] Oct 30 09:23:30.828589 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:30.824648 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-962e2e-openshift-default-dockercfg-krhx5\"" Oct 30 09:23:30.856681 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:30.856640 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/286eb740-46f4-44ba-80f1-0de69bd89861-istio-envoy\") pod \"e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f\" (UID: \"286eb740-46f4-44ba-80f1-0de69bd89861\") " pod="openshift-ingress/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f" Oct 30 09:23:30.856883 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:30.856714 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/286eb740-46f4-44ba-80f1-0de69bd89861-istiod-ca-cert\") pod \"e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f\" (UID: \"286eb740-46f4-44ba-80f1-0de69bd89861\") " pod="openshift-ingress/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f" Oct 30 09:23:30.856883 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:30.856794 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/286eb740-46f4-44ba-80f1-0de69bd89861-credential-socket\") pod \"e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f\" (UID: \"286eb740-46f4-44ba-80f1-0de69bd89861\") " pod="openshift-ingress/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f" Oct 30 09:23:30.856883 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:30.856838 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/286eb740-46f4-44ba-80f1-0de69bd89861-istio-token\") pod \"e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f\" (UID: \"286eb740-46f4-44ba-80f1-0de69bd89861\") " pod="openshift-ingress/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f" Oct 30 09:23:30.857073 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:30.856907 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/286eb740-46f4-44ba-80f1-0de69bd89861-workload-socket\") pod \"e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f\" (UID: \"286eb740-46f4-44ba-80f1-0de69bd89861\") " pod="openshift-ingress/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f" Oct 30 09:23:30.857073 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:30.856943 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/286eb740-46f4-44ba-80f1-0de69bd89861-workload-certs\") pod \"e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f\" (UID: \"286eb740-46f4-44ba-80f1-0de69bd89861\") " pod="openshift-ingress/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f" Oct 30 09:23:30.857073 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:30.856975 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/286eb740-46f4-44ba-80f1-0de69bd89861-istio-data\") pod \"e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f\" (UID: \"286eb740-46f4-44ba-80f1-0de69bd89861\") " pod="openshift-ingress/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f" Oct 30 09:23:30.857238 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:30.857105 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/286eb740-46f4-44ba-80f1-0de69bd89861-istio-podinfo\") pod \"e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f\" (UID: \"286eb740-46f4-44ba-80f1-0de69bd89861\") " pod="openshift-ingress/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f" Oct 30 09:23:30.857238 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:30.857155 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xnwh\" (UniqueName: \"kubernetes.io/projected/286eb740-46f4-44ba-80f1-0de69bd89861-kube-api-access-6xnwh\") pod \"e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f\" (UID: \"286eb740-46f4-44ba-80f1-0de69bd89861\") " pod="openshift-ingress/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f" Oct 30 09:23:30.958797 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:30.958748 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/286eb740-46f4-44ba-80f1-0de69bd89861-istio-envoy\") pod \"e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f\" (UID: \"286eb740-46f4-44ba-80f1-0de69bd89861\") " pod="openshift-ingress/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f" Oct 30 09:23:30.959004 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:30.958908 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/286eb740-46f4-44ba-80f1-0de69bd89861-istiod-ca-cert\") pod \"e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f\" (UID: \"286eb740-46f4-44ba-80f1-0de69bd89861\") " pod="openshift-ingress/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f" Oct 30 09:23:30.959084 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:30.959001 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/286eb740-46f4-44ba-80f1-0de69bd89861-credential-socket\") pod \"e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f\" (UID: \"286eb740-46f4-44ba-80f1-0de69bd89861\") " pod="openshift-ingress/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f" Oct 30 09:23:30.959084 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:30.959049 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/286eb740-46f4-44ba-80f1-0de69bd89861-istio-token\") pod \"e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f\" (UID: \"286eb740-46f4-44ba-80f1-0de69bd89861\") " pod="openshift-ingress/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f" Oct 30 09:23:30.959188 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:30.959130 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/286eb740-46f4-44ba-80f1-0de69bd89861-workload-socket\") pod \"e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f\" (UID: \"286eb740-46f4-44ba-80f1-0de69bd89861\") " pod="openshift-ingress/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f" Oct 30 09:23:30.959256 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:30.959186 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/286eb740-46f4-44ba-80f1-0de69bd89861-workload-certs\") pod \"e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f\" (UID: \"286eb740-46f4-44ba-80f1-0de69bd89861\") " pod="openshift-ingress/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f" Oct 30 09:23:30.959256 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:30.959218 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/286eb740-46f4-44ba-80f1-0de69bd89861-istio-data\") pod \"e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f\" (UID: \"286eb740-46f4-44ba-80f1-0de69bd89861\") " pod="openshift-ingress/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f" Oct 30 09:23:30.959361 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:30.959340 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/286eb740-46f4-44ba-80f1-0de69bd89861-istio-podinfo\") pod \"e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f\" (UID: \"286eb740-46f4-44ba-80f1-0de69bd89861\") " pod="openshift-ingress/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f" Oct 30 09:23:30.959443 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:30.959361 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/286eb740-46f4-44ba-80f1-0de69bd89861-credential-socket\") pod \"e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f\" (UID: \"286eb740-46f4-44ba-80f1-0de69bd89861\") " pod="openshift-ingress/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f" Oct 30 09:23:30.959443 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:30.959419 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xnwh\" (UniqueName: \"kubernetes.io/projected/286eb740-46f4-44ba-80f1-0de69bd89861-kube-api-access-6xnwh\") pod \"e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f\" (UID: \"286eb740-46f4-44ba-80f1-0de69bd89861\") " pod="openshift-ingress/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f" Oct 30 09:23:30.959578 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:30.959553 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/286eb740-46f4-44ba-80f1-0de69bd89861-workload-socket\") pod \"e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f\" (UID: \"286eb740-46f4-44ba-80f1-0de69bd89861\") " pod="openshift-ingress/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f" Oct 30 09:23:30.959696 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:30.959671 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/286eb740-46f4-44ba-80f1-0de69bd89861-workload-certs\") pod \"e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f\" (UID: \"286eb740-46f4-44ba-80f1-0de69bd89861\") " pod="openshift-ingress/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f" Oct 30 09:23:30.959758 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:30.959714 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/286eb740-46f4-44ba-80f1-0de69bd89861-istiod-ca-cert\") pod \"e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f\" (UID: \"286eb740-46f4-44ba-80f1-0de69bd89861\") " pod="openshift-ingress/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f" Oct 30 09:23:30.960190 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:30.959997 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/286eb740-46f4-44ba-80f1-0de69bd89861-istio-data\") pod \"e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f\" (UID: \"286eb740-46f4-44ba-80f1-0de69bd89861\") " pod="openshift-ingress/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f" Oct 30 09:23:30.961605 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:30.961582 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/286eb740-46f4-44ba-80f1-0de69bd89861-istio-envoy\") pod \"e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f\" (UID: \"286eb740-46f4-44ba-80f1-0de69bd89861\") " pod="openshift-ingress/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f" Oct 30 09:23:30.961940 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:30.961920 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/286eb740-46f4-44ba-80f1-0de69bd89861-istio-podinfo\") pod \"e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f\" (UID: \"286eb740-46f4-44ba-80f1-0de69bd89861\") " pod="openshift-ingress/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f" Oct 30 09:23:30.969566 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:30.969545 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f40a3-84f8df4f47-6p5c8" Oct 30 09:23:30.980003 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:30.979975 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/286eb740-46f4-44ba-80f1-0de69bd89861-istio-token\") pod \"e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f\" (UID: \"286eb740-46f4-44ba-80f1-0de69bd89861\") " pod="openshift-ingress/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f" Oct 30 09:23:30.980989 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:30.980961 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xnwh\" (UniqueName: \"kubernetes.io/projected/286eb740-46f4-44ba-80f1-0de69bd89861-kube-api-access-6xnwh\") pod \"e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f\" (UID: \"286eb740-46f4-44ba-80f1-0de69bd89861\") " pod="openshift-ingress/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f" Oct 30 09:23:31.144583 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:31.144258 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f" Oct 30 09:23:31.568498 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:23:31.568455 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdfbe6ee_d5c2_48e6_a7d8_a05ea3bc8e01.slice/crio-1286f69aa59c208b1816264b0a8f4e0b1b8e05bef659cd4aff2f1b14a12cdd9d WatchSource:0}: Error finding container 1286f69aa59c208b1816264b0a8f4e0b1b8e05bef659cd4aff2f1b14a12cdd9d: Status 404 returned error can't find the container with id 1286f69aa59c208b1816264b0a8f4e0b1b8e05bef659cd4aff2f1b14a12cdd9d Oct 30 09:23:31.571051 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:31.571024 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8f40a3-84f8df4f47-6p5c8"] Oct 30 09:23:31.674972 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:23:31.674942 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod286eb740_46f4_44ba_80f1_0de69bd89861.slice/crio-4a1f386c91ba3924fddbc6318f32313ee055b30bccd5270f5d13c1ec1232ce1a WatchSource:0}: Error finding container 4a1f386c91ba3924fddbc6318f32313ee055b30bccd5270f5d13c1ec1232ce1a: Status 404 returned error can't find the container with id 4a1f386c91ba3924fddbc6318f32313ee055b30bccd5270f5d13c1ec1232ce1a Oct 30 09:23:31.677348 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:31.677324 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f"] Oct 30 09:23:31.677664 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:31.677633 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:23:31.677738 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:31.677715 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:23:31.677793 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:31.677761 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:23:31.707948 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:31.707924 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f" event={"ID":"286eb740-46f4-44ba-80f1-0de69bd89861","Type":"ContainerStarted","Data":"4a1f386c91ba3924fddbc6318f32313ee055b30bccd5270f5d13c1ec1232ce1a"} Oct 30 09:23:31.709886 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:31.709859 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f40a3-84f8df4f47-6p5c8" event={"ID":"bdfbe6ee-d5c2-48e6-a7d8-a05ea3bc8e01","Type":"ContainerStarted","Data":"d74cb5b1825d8a598692fd0b330440b6c825515046469df8dff77ba53f5bbb4b"} Oct 30 09:23:31.709989 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:31.709909 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f40a3-84f8df4f47-6p5c8" event={"ID":"bdfbe6ee-d5c2-48e6-a7d8-a05ea3bc8e01","Type":"ContainerStarted","Data":"1286f69aa59c208b1816264b0a8f4e0b1b8e05bef659cd4aff2f1b14a12cdd9d"} Oct 30 09:23:31.709989 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:31.709970 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f40a3-84f8df4f47-6p5c8" Oct 30 09:23:31.765508 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:31.765462 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f40a3-84f8df4f47-6p5c8" podStartSLOduration=1.765448857 podStartE2EDuration="1.765448857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:23:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:23:31.764527157 +0000 UTC m=+1772.026088147" watchObservedRunningTime="2026-10-30 09:23:31.765448857 +0000 UTC m=+1772.027009845" Oct 30 09:23:32.717950 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:32.717918 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f" event={"ID":"286eb740-46f4-44ba-80f1-0de69bd89861","Type":"ContainerStarted","Data":"bce982b30d40131ba088ef0c2518e3b3d42db0314ab6426dde722029d794bcd8"} Oct 30 09:23:32.776683 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:32.776627 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f" podStartSLOduration=2.77661432 podStartE2EDuration="2.77661432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:23:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:23:32.774610719 +0000 UTC m=+1773.036171709" watchObservedRunningTime="2026-10-30 09:23:32.77661432 +0000 UTC m=+1773.038175308" Oct 30 09:23:33.146113 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:33.146034 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f" Oct 30 09:23:33.152513 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:33.152490 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f" Oct 30 09:23:33.797686 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:33.795458 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f" Oct 30 09:23:33.807991 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:33.807957 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f" Oct 30 09:23:36.674505 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:36.674473 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbe974-d65fcff9-6hvv7" Oct 30 09:23:36.674914 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:36.674550 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbe974-b85c966d6-6fqhj" Oct 30 09:23:37.198066 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:37.198028 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-962e2e-76469968d7-bx7fz"] Oct 30 09:23:37.204009 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:37.203981 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-962e2e-76469968d7-bx7fz" Oct 30 09:23:37.228923 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:37.227756 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-962e2e-dockercfg-ls7mt\"" Oct 30 09:23:37.247197 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:37.242776 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-962e2e\"" Oct 30 09:23:37.311516 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:37.311474 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-962e2e-76469968d7-bx7fz"] Oct 30 09:23:37.389516 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:37.381968 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-962e2e-756c88587b-dnvhs"] Oct 30 09:23:37.497473 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:37.496854 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4df6984-60f1-4ed6-910c-2608e94ab9db-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-962e2e-76469968d7-bx7fz\" (UID: \"e4df6984-60f1-4ed6-910c-2608e94ab9db\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-962e2e-76469968d7-bx7fz" Oct 30 09:23:37.497473 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:37.497206 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhjzp\" (UniqueName: \"kubernetes.io/projected/e4df6984-60f1-4ed6-910c-2608e94ab9db-kube-api-access-lhjzp\") pod \"payload-pre-processing-e2e-ipp-b-962e2e-76469968d7-bx7fz\" (UID: \"e4df6984-60f1-4ed6-910c-2608e94ab9db\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-962e2e-76469968d7-bx7fz" Oct 30 09:23:37.500951 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:37.500914 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-962e2e-756c88587b-dnvhs" Oct 30 09:23:37.561635 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:37.561215 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-962e2e-756c88587b-dnvhs"] Oct 30 09:23:37.685402 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:37.652670 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhjzp\" (UniqueName: \"kubernetes.io/projected/e4df6984-60f1-4ed6-910c-2608e94ab9db-kube-api-access-lhjzp\") pod \"payload-pre-processing-e2e-ipp-b-962e2e-76469968d7-bx7fz\" (UID: \"e4df6984-60f1-4ed6-910c-2608e94ab9db\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-962e2e-76469968d7-bx7fz" Oct 30 09:23:37.685402 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:37.652791 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xqtv\" (UniqueName: \"kubernetes.io/projected/e70dc648-2cba-4c25-af40-e83ee85a58a1-kube-api-access-4xqtv\") pod \"payload-processing-e2e-ipp-b-962e2e-756c88587b-dnvhs\" (UID: \"e70dc648-2cba-4c25-af40-e83ee85a58a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-962e2e-756c88587b-dnvhs" Oct 30 09:23:37.685402 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:37.652999 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4df6984-60f1-4ed6-910c-2608e94ab9db-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-962e2e-76469968d7-bx7fz\" (UID: \"e4df6984-60f1-4ed6-910c-2608e94ab9db\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-962e2e-76469968d7-bx7fz" Oct 30 09:23:37.685402 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:37.653157 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e70dc648-2cba-4c25-af40-e83ee85a58a1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-962e2e-756c88587b-dnvhs\" (UID: \"e70dc648-2cba-4c25-af40-e83ee85a58a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-962e2e-756c88587b-dnvhs" Oct 30 09:23:37.685402 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:37.655238 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4df6984-60f1-4ed6-910c-2608e94ab9db-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-962e2e-76469968d7-bx7fz\" (UID: \"e4df6984-60f1-4ed6-910c-2608e94ab9db\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-962e2e-76469968d7-bx7fz" Oct 30 09:23:37.706437 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:37.695414 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhjzp\" (UniqueName: \"kubernetes.io/projected/e4df6984-60f1-4ed6-910c-2608e94ab9db-kube-api-access-lhjzp\") pod \"payload-pre-processing-e2e-ipp-b-962e2e-76469968d7-bx7fz\" (UID: \"e4df6984-60f1-4ed6-910c-2608e94ab9db\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-962e2e-76469968d7-bx7fz" Oct 30 09:23:37.754704 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:37.754610 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e70dc648-2cba-4c25-af40-e83ee85a58a1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-962e2e-756c88587b-dnvhs\" (UID: \"e70dc648-2cba-4c25-af40-e83ee85a58a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-962e2e-756c88587b-dnvhs" Oct 30 09:23:37.754873 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:37.754844 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xqtv\" (UniqueName: \"kubernetes.io/projected/e70dc648-2cba-4c25-af40-e83ee85a58a1-kube-api-access-4xqtv\") pod \"payload-processing-e2e-ipp-b-962e2e-756c88587b-dnvhs\" (UID: \"e70dc648-2cba-4c25-af40-e83ee85a58a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-962e2e-756c88587b-dnvhs" Oct 30 09:23:37.755536 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:37.755514 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e70dc648-2cba-4c25-af40-e83ee85a58a1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-962e2e-756c88587b-dnvhs\" (UID: \"e70dc648-2cba-4c25-af40-e83ee85a58a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-962e2e-756c88587b-dnvhs" Oct 30 09:23:37.774754 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:37.774720 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xqtv\" (UniqueName: \"kubernetes.io/projected/e70dc648-2cba-4c25-af40-e83ee85a58a1-kube-api-access-4xqtv\") pod \"payload-processing-e2e-ipp-b-962e2e-756c88587b-dnvhs\" (UID: \"e70dc648-2cba-4c25-af40-e83ee85a58a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-962e2e-756c88587b-dnvhs" Oct 30 09:23:37.819267 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:37.819237 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-962e2e-76469968d7-bx7fz" Oct 30 09:23:37.868304 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:37.868270 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-962e2e-756c88587b-dnvhs" Oct 30 09:23:38.123154 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:23:38.123124 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4df6984_60f1_4ed6_910c_2608e94ab9db.slice/crio-f9738733f8734bf6af9ac5df975ff8445e7c33650142d5bf7d8d3791a3ef8bbd WatchSource:0}: Error finding container f9738733f8734bf6af9ac5df975ff8445e7c33650142d5bf7d8d3791a3ef8bbd: Status 404 returned error can't find the container with id f9738733f8734bf6af9ac5df975ff8445e7c33650142d5bf7d8d3791a3ef8bbd Oct 30 09:23:38.125134 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:38.125107 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-962e2e-76469968d7-bx7fz"] Oct 30 09:23:38.189062 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:23:38.189040 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode70dc648_2cba_4c25_af40_e83ee85a58a1.slice/crio-04d716f80fbe4608bd5f166a3f436238def684ede867ed6a74a515f8b6ae6f12 WatchSource:0}: Error finding container 04d716f80fbe4608bd5f166a3f436238def684ede867ed6a74a515f8b6ae6f12: Status 404 returned error can't find the container with id 04d716f80fbe4608bd5f166a3f436238def684ede867ed6a74a515f8b6ae6f12 Oct 30 09:23:38.191879 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:38.191841 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-962e2e-756c88587b-dnvhs"] Oct 30 09:23:38.834121 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:38.834086 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-962e2e-756c88587b-dnvhs" event={"ID":"e70dc648-2cba-4c25-af40-e83ee85a58a1","Type":"ContainerStarted","Data":"d053feb106811114fa0d85ea302c69e58d3c1a97356dbe51cddb221bfbf1c49f"} Oct 30 09:23:38.834619 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:38.834138 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-962e2e-756c88587b-dnvhs" event={"ID":"e70dc648-2cba-4c25-af40-e83ee85a58a1","Type":"ContainerStarted","Data":"04d716f80fbe4608bd5f166a3f436238def684ede867ed6a74a515f8b6ae6f12"} Oct 30 09:23:38.834619 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:38.834270 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-962e2e-756c88587b-dnvhs" Oct 30 09:23:38.836079 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:38.836047 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-962e2e-76469968d7-bx7fz" event={"ID":"e4df6984-60f1-4ed6-910c-2608e94ab9db","Type":"ContainerStarted","Data":"f824e625891be4e90a6b2467be398377547ce9b66229d2ac4e8f398007aeea69"} Oct 30 09:23:38.836219 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:38.836101 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-962e2e-76469968d7-bx7fz" Oct 30 09:23:38.836219 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:38.836124 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-962e2e-76469968d7-bx7fz" event={"ID":"e4df6984-60f1-4ed6-910c-2608e94ab9db","Type":"ContainerStarted","Data":"f9738733f8734bf6af9ac5df975ff8445e7c33650142d5bf7d8d3791a3ef8bbd"} Oct 30 09:23:38.882001 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:38.881950 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-962e2e-756c88587b-dnvhs" podStartSLOduration=1.8819371660000002 podStartE2EDuration="1.881937166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:23:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:23:38.87767287 +0000 UTC m=+1779.139233861" watchObservedRunningTime="2026-10-30 09:23:38.881937166 +0000 UTC m=+1779.143498153" Oct 30 09:23:38.924370 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:38.924313 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-962e2e-76469968d7-bx7fz" podStartSLOduration=1.924298233 podStartE2EDuration="1.924298233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:23:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:23:38.921337174 +0000 UTC m=+1779.182898163" watchObservedRunningTime="2026-10-30 09:23:38.924298233 +0000 UTC m=+1779.185859220" Oct 30 09:23:42.734993 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:42.734955 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f40a3-84f8df4f47-6p5c8" Oct 30 09:23:49.846362 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:49.846329 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-962e2e-76469968d7-bx7fz" Oct 30 09:23:49.846858 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:49.846435 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-962e2e-756c88587b-dnvhs" Oct 30 09:23:51.351949 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:51.351914 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-356553-658d75b4cf-bvlnt"] Oct 30 09:23:51.357790 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:51.357768 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-356553-658d75b4cf-bvlnt" Oct 30 09:23:51.378913 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:51.378877 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-356553-658d75b4cf-bvlnt"] Oct 30 09:23:51.387967 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:51.387935 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-356553\"" Oct 30 09:23:51.388076 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:51.387997 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-356553-dockercfg-rfkwb\"" Oct 30 09:23:51.540408 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:51.531116 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a916cf3a-673c-4244-8d65-1ab14f6328bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-356553-658d75b4cf-bvlnt\" (UID: \"a916cf3a-673c-4244-8d65-1ab14f6328bd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-356553-658d75b4cf-bvlnt" Oct 30 09:23:51.540408 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:51.531458 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7lgd\" (UniqueName: \"kubernetes.io/projected/a916cf3a-673c-4244-8d65-1ab14f6328bd-kube-api-access-n7lgd\") pod \"payload-pre-processing-e2e-ipp-b-356553-658d75b4cf-bvlnt\" (UID: \"a916cf3a-673c-4244-8d65-1ab14f6328bd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-356553-658d75b4cf-bvlnt" Oct 30 09:23:51.572406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:51.568442 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-356553-54cf695556-7vstx"] Oct 30 09:23:51.705710 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:51.688719 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-356553-54cf695556-7vstx"] Oct 30 09:23:51.705710 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:51.688843 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-356553-54cf695556-7vstx" Oct 30 09:23:51.752751 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:51.752715 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kktwd\" (UniqueName: \"kubernetes.io/projected/7a3c155f-73dc-48e5-b0fc-4d45d49670fa-kube-api-access-kktwd\") pod \"payload-processing-e2e-ipp-b-356553-54cf695556-7vstx\" (UID: \"7a3c155f-73dc-48e5-b0fc-4d45d49670fa\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-356553-54cf695556-7vstx" Oct 30 09:23:51.753100 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:51.753081 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7lgd\" (UniqueName: \"kubernetes.io/projected/a916cf3a-673c-4244-8d65-1ab14f6328bd-kube-api-access-n7lgd\") pod \"payload-pre-processing-e2e-ipp-b-356553-658d75b4cf-bvlnt\" (UID: \"a916cf3a-673c-4244-8d65-1ab14f6328bd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-356553-658d75b4cf-bvlnt" Oct 30 09:23:51.753659 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:51.753634 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a916cf3a-673c-4244-8d65-1ab14f6328bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-356553-658d75b4cf-bvlnt\" (UID: \"a916cf3a-673c-4244-8d65-1ab14f6328bd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-356553-658d75b4cf-bvlnt" Oct 30 09:23:51.766677 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:51.763591 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a3c155f-73dc-48e5-b0fc-4d45d49670fa-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-356553-54cf695556-7vstx\" (UID: \"7a3c155f-73dc-48e5-b0fc-4d45d49670fa\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-356553-54cf695556-7vstx" Oct 30 09:23:51.767116 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:51.767086 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a916cf3a-673c-4244-8d65-1ab14f6328bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-356553-658d75b4cf-bvlnt\" (UID: \"a916cf3a-673c-4244-8d65-1ab14f6328bd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-356553-658d75b4cf-bvlnt" Oct 30 09:23:51.774615 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:51.774590 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7lgd\" (UniqueName: \"kubernetes.io/projected/a916cf3a-673c-4244-8d65-1ab14f6328bd-kube-api-access-n7lgd\") pod \"payload-pre-processing-e2e-ipp-b-356553-658d75b4cf-bvlnt\" (UID: \"a916cf3a-673c-4244-8d65-1ab14f6328bd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-356553-658d75b4cf-bvlnt" Oct 30 09:23:51.864867 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:51.864834 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a3c155f-73dc-48e5-b0fc-4d45d49670fa-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-356553-54cf695556-7vstx\" (UID: \"7a3c155f-73dc-48e5-b0fc-4d45d49670fa\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-356553-54cf695556-7vstx" Oct 30 09:23:51.865069 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:51.865051 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kktwd\" (UniqueName: \"kubernetes.io/projected/7a3c155f-73dc-48e5-b0fc-4d45d49670fa-kube-api-access-kktwd\") pod \"payload-processing-e2e-ipp-b-356553-54cf695556-7vstx\" (UID: \"7a3c155f-73dc-48e5-b0fc-4d45d49670fa\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-356553-54cf695556-7vstx" Oct 30 09:23:51.865699 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:51.865660 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a3c155f-73dc-48e5-b0fc-4d45d49670fa-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-356553-54cf695556-7vstx\" (UID: \"7a3c155f-73dc-48e5-b0fc-4d45d49670fa\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-356553-54cf695556-7vstx" Oct 30 09:23:51.885408 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:51.885353 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kktwd\" (UniqueName: \"kubernetes.io/projected/7a3c155f-73dc-48e5-b0fc-4d45d49670fa-kube-api-access-kktwd\") pod \"payload-processing-e2e-ipp-b-356553-54cf695556-7vstx\" (UID: \"7a3c155f-73dc-48e5-b0fc-4d45d49670fa\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-356553-54cf695556-7vstx" Oct 30 09:23:51.972182 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:51.972093 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-356553-658d75b4cf-bvlnt" Oct 30 09:23:52.078772 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:52.078743 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-356553-54cf695556-7vstx" Oct 30 09:23:52.292771 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:23:52.292737 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda916cf3a_673c_4244_8d65_1ab14f6328bd.slice/crio-3f9b7dfc2901cee66fc699818561dd3a8bc72ec7689a5e6a38a8b886997ed08d WatchSource:0}: Error finding container 3f9b7dfc2901cee66fc699818561dd3a8bc72ec7689a5e6a38a8b886997ed08d: Status 404 returned error can't find the container with id 3f9b7dfc2901cee66fc699818561dd3a8bc72ec7689a5e6a38a8b886997ed08d Oct 30 09:23:52.293138 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:52.293112 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-356553-658d75b4cf-bvlnt"] Oct 30 09:23:52.415728 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:23:52.415702 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a3c155f_73dc_48e5_b0fc_4d45d49670fa.slice/crio-4e5ab667f713559d578767ec5e108d30533214c203e2f7a23e6fa16f4d1a21fa WatchSource:0}: Error finding container 4e5ab667f713559d578767ec5e108d30533214c203e2f7a23e6fa16f4d1a21fa: Status 404 returned error can't find the container with id 4e5ab667f713559d578767ec5e108d30533214c203e2f7a23e6fa16f4d1a21fa Oct 30 09:23:52.418548 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:52.418523 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-356553-54cf695556-7vstx"] Oct 30 09:23:53.010406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:53.007453 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-356553-658d75b4cf-bvlnt" event={"ID":"a916cf3a-673c-4244-8d65-1ab14f6328bd","Type":"ContainerStarted","Data":"08df7bf40cdae14f75506ffcf08239baa22f2207b9590cd902d0cf0fd5733513"} Oct 30 09:23:53.010406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:53.007525 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-356553-658d75b4cf-bvlnt" event={"ID":"a916cf3a-673c-4244-8d65-1ab14f6328bd","Type":"ContainerStarted","Data":"3f9b7dfc2901cee66fc699818561dd3a8bc72ec7689a5e6a38a8b886997ed08d"} Oct 30 09:23:53.010406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:53.008675 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-356553-658d75b4cf-bvlnt" Oct 30 09:23:53.024601 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:53.024556 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-356553-54cf695556-7vstx" event={"ID":"7a3c155f-73dc-48e5-b0fc-4d45d49670fa","Type":"ContainerStarted","Data":"1df14b6a135fd7f6bd4cfcd37afcb61a67f421fe4d0c2f50cdfe91ca8695cf97"} Oct 30 09:23:53.024772 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:53.024619 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-356553-54cf695556-7vstx" event={"ID":"7a3c155f-73dc-48e5-b0fc-4d45d49670fa","Type":"ContainerStarted","Data":"4e5ab667f713559d578767ec5e108d30533214c203e2f7a23e6fa16f4d1a21fa"} Oct 30 09:23:53.025120 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:53.025089 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-356553-54cf695556-7vstx" Oct 30 09:23:53.065064 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:53.060945 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-356553-658d75b4cf-bvlnt" podStartSLOduration=2.060868886 podStartE2EDuration="2.060868886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:23:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:23:53.057900914 +0000 UTC m=+1793.319461903" watchObservedRunningTime="2026-10-30 09:23:53.060868886 +0000 UTC m=+1793.322429875" Oct 30 09:23:53.119171 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:53.114067 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-356553-54cf695556-7vstx" podStartSLOduration=2.114043717 podStartE2EDuration="2.114043717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:23:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:23:53.109141002 +0000 UTC m=+1793.370701991" watchObservedRunningTime="2026-10-30 09:23:53.114043717 +0000 UTC m=+1793.375604738" Oct 30 09:23:58.597512 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:23:58.597475 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:23:58.654052 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:58.654016 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9371b7-66bbf587b-8tg85"] Oct 30 09:23:58.658830 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:58.658796 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9371b7-66bbf587b-8tg85" Oct 30 09:23:58.678076 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:58.678048 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9371b7-66bbf587b-8tg85"] Oct 30 09:23:58.686799 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:58.686768 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-9371b7-serving-cert\"" Oct 30 09:23:58.744116 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:58.744085 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1aa98b2-58f0-442b-9591-0f00f8a9929c-metrics-certs\") pod \"maas-api-e2e-api-b-9371b7-66bbf587b-8tg85\" (UID: \"f1aa98b2-58f0-442b-9591-0f00f8a9929c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9371b7-66bbf587b-8tg85" Oct 30 09:23:58.744295 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:58.744129 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qllz8\" (UniqueName: \"kubernetes.io/projected/f1aa98b2-58f0-442b-9591-0f00f8a9929c-kube-api-access-qllz8\") pod \"maas-api-e2e-api-b-9371b7-66bbf587b-8tg85\" (UID: \"f1aa98b2-58f0-442b-9591-0f00f8a9929c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9371b7-66bbf587b-8tg85" Oct 30 09:23:58.744295 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:58.744258 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1aa98b2-58f0-442b-9591-0f00f8a9929c-openshift-service-ca\") pod \"maas-api-e2e-api-b-9371b7-66bbf587b-8tg85\" (UID: \"f1aa98b2-58f0-442b-9591-0f00f8a9929c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9371b7-66bbf587b-8tg85" Oct 30 09:23:58.744447 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:58.744373 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1aa98b2-58f0-442b-9591-0f00f8a9929c-maas-api-tls\") pod \"maas-api-e2e-api-b-9371b7-66bbf587b-8tg85\" (UID: \"f1aa98b2-58f0-442b-9591-0f00f8a9929c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9371b7-66bbf587b-8tg85" Oct 30 09:23:58.867452 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:58.866610 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1aa98b2-58f0-442b-9591-0f00f8a9929c-metrics-certs\") pod \"maas-api-e2e-api-b-9371b7-66bbf587b-8tg85\" (UID: \"f1aa98b2-58f0-442b-9591-0f00f8a9929c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9371b7-66bbf587b-8tg85" Oct 30 09:23:58.867452 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:58.866678 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qllz8\" (UniqueName: \"kubernetes.io/projected/f1aa98b2-58f0-442b-9591-0f00f8a9929c-kube-api-access-qllz8\") pod \"maas-api-e2e-api-b-9371b7-66bbf587b-8tg85\" (UID: \"f1aa98b2-58f0-442b-9591-0f00f8a9929c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9371b7-66bbf587b-8tg85" Oct 30 09:23:58.867452 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:58.866714 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1aa98b2-58f0-442b-9591-0f00f8a9929c-openshift-service-ca\") pod \"maas-api-e2e-api-b-9371b7-66bbf587b-8tg85\" (UID: \"f1aa98b2-58f0-442b-9591-0f00f8a9929c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9371b7-66bbf587b-8tg85" Oct 30 09:23:58.867452 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:58.866817 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1aa98b2-58f0-442b-9591-0f00f8a9929c-maas-api-tls\") pod \"maas-api-e2e-api-b-9371b7-66bbf587b-8tg85\" (UID: \"f1aa98b2-58f0-442b-9591-0f00f8a9929c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9371b7-66bbf587b-8tg85" Oct 30 09:23:58.884407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:58.869898 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1aa98b2-58f0-442b-9591-0f00f8a9929c-openshift-service-ca\") pod \"maas-api-e2e-api-b-9371b7-66bbf587b-8tg85\" (UID: \"f1aa98b2-58f0-442b-9591-0f00f8a9929c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9371b7-66bbf587b-8tg85" Oct 30 09:23:58.892404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:58.891660 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qllz8\" (UniqueName: \"kubernetes.io/projected/f1aa98b2-58f0-442b-9591-0f00f8a9929c-kube-api-access-qllz8\") pod \"maas-api-e2e-api-b-9371b7-66bbf587b-8tg85\" (UID: \"f1aa98b2-58f0-442b-9591-0f00f8a9929c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9371b7-66bbf587b-8tg85" Oct 30 09:23:58.903431 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:58.898256 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1aa98b2-58f0-442b-9591-0f00f8a9929c-maas-api-tls\") pod \"maas-api-e2e-api-b-9371b7-66bbf587b-8tg85\" (UID: \"f1aa98b2-58f0-442b-9591-0f00f8a9929c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9371b7-66bbf587b-8tg85" Oct 30 09:23:58.903431 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:58.898800 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1aa98b2-58f0-442b-9591-0f00f8a9929c-metrics-certs\") pod \"maas-api-e2e-api-b-9371b7-66bbf587b-8tg85\" (UID: \"f1aa98b2-58f0-442b-9591-0f00f8a9929c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9371b7-66bbf587b-8tg85" Oct 30 09:23:58.979474 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:58.977604 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9371b7-66bbf587b-8tg85" Oct 30 09:23:59.299155 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:23:59.299123 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1aa98b2_58f0_442b_9591_0f00f8a9929c.slice/crio-4556039b79d8412e3f9c4dc81b21f985b0a68e5db75fa99c17ead0bd34c77e04 WatchSource:0}: Error finding container 4556039b79d8412e3f9c4dc81b21f985b0a68e5db75fa99c17ead0bd34c77e04: Status 404 returned error can't find the container with id 4556039b79d8412e3f9c4dc81b21f985b0a68e5db75fa99c17ead0bd34c77e04 Oct 30 09:23:59.301108 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:23:59.301083 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9371b7-66bbf587b-8tg85"] Oct 30 09:24:00.099016 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:00.098988 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6q7n_1f51db5a-5720-4dea-ad02-f336b35d43b1/ovn-acl-logging/0.log" Oct 30 09:24:00.112033 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:00.112009 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6q7n_1f51db5a-5720-4dea-ad02-f336b35d43b1/ovn-acl-logging/0.log" Oct 30 09:24:00.195808 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:00.195770 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9371b7-66bbf587b-8tg85" event={"ID":"f1aa98b2-58f0-442b-9591-0f00f8a9929c","Type":"ContainerStarted","Data":"5c0f67c0f22caa4269d1f535169418e984f291d4d09b40c5b1aa98a1a2b28c72"} Oct 30 09:24:00.195981 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:00.195826 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9371b7-66bbf587b-8tg85" event={"ID":"f1aa98b2-58f0-442b-9591-0f00f8a9929c","Type":"ContainerStarted","Data":"4556039b79d8412e3f9c4dc81b21f985b0a68e5db75fa99c17ead0bd34c77e04"} Oct 30 09:24:00.195981 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:00.195904 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9371b7-66bbf587b-8tg85" Oct 30 09:24:00.240235 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:00.240163 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9371b7-66bbf587b-8tg85" podStartSLOduration=1.9329972130000002 podStartE2EDuration="2.240111743s" totalImagesPullingTime="307.11453ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:23:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:24:00.237552965 +0000 UTC m=+1800.499113953" watchObservedRunningTime="2026-10-30 09:24:00.240111743 +0000 UTC m=+1800.501672736" Oct 30 09:24:04.035209 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:04.035175 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-356553-54cf695556-7vstx" Oct 30 09:24:05.043210 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:05.043177 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-356553-658d75b4cf-bvlnt" Oct 30 09:24:06.209209 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:06.209179 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9371b7-66bbf587b-8tg85" Oct 30 09:24:08.376404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:08.367563 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4fc2e2-84855487c8-46cpz"] Oct 30 09:24:08.376404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:08.375679 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4fc2e2-84855487c8-46cpz" Oct 30 09:24:08.406407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:08.393310 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4fc2e2-84855487c8-46cpz"] Oct 30 09:24:08.406407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:08.397662 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-4fc2e2\"" Oct 30 09:24:08.424575 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:08.412100 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-4fc2e2-dockercfg-zg9w9\"" Oct 30 09:24:08.505605 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:08.505532 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/343dbb0c-f2d3-41cf-bc2c-283eba43e35d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4fc2e2-84855487c8-46cpz\" (UID: \"343dbb0c-f2d3-41cf-bc2c-283eba43e35d\") " pod="openshift-ingress/payload-processing-e2e-api-a-4fc2e2-84855487c8-46cpz" Oct 30 09:24:08.506036 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:08.506007 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb8tj\" (UniqueName: \"kubernetes.io/projected/343dbb0c-f2d3-41cf-bc2c-283eba43e35d-kube-api-access-nb8tj\") pod \"payload-processing-e2e-api-a-4fc2e2-84855487c8-46cpz\" (UID: \"343dbb0c-f2d3-41cf-bc2c-283eba43e35d\") " pod="openshift-ingress/payload-processing-e2e-api-a-4fc2e2-84855487c8-46cpz" Oct 30 09:24:08.607655 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:08.607622 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/343dbb0c-f2d3-41cf-bc2c-283eba43e35d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4fc2e2-84855487c8-46cpz\" (UID: \"343dbb0c-f2d3-41cf-bc2c-283eba43e35d\") " pod="openshift-ingress/payload-processing-e2e-api-a-4fc2e2-84855487c8-46cpz" Oct 30 09:24:08.607836 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:08.607663 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb8tj\" (UniqueName: \"kubernetes.io/projected/343dbb0c-f2d3-41cf-bc2c-283eba43e35d-kube-api-access-nb8tj\") pod \"payload-processing-e2e-api-a-4fc2e2-84855487c8-46cpz\" (UID: \"343dbb0c-f2d3-41cf-bc2c-283eba43e35d\") " pod="openshift-ingress/payload-processing-e2e-api-a-4fc2e2-84855487c8-46cpz" Oct 30 09:24:08.608467 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:08.608447 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/343dbb0c-f2d3-41cf-bc2c-283eba43e35d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4fc2e2-84855487c8-46cpz\" (UID: \"343dbb0c-f2d3-41cf-bc2c-283eba43e35d\") " pod="openshift-ingress/payload-processing-e2e-api-a-4fc2e2-84855487c8-46cpz" Oct 30 09:24:08.628113 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:08.628047 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb8tj\" (UniqueName: \"kubernetes.io/projected/343dbb0c-f2d3-41cf-bc2c-283eba43e35d-kube-api-access-nb8tj\") pod \"payload-processing-e2e-api-a-4fc2e2-84855487c8-46cpz\" (UID: \"343dbb0c-f2d3-41cf-bc2c-283eba43e35d\") " pod="openshift-ingress/payload-processing-e2e-api-a-4fc2e2-84855487c8-46cpz" Oct 30 09:24:08.726655 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:08.726625 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4fc2e2-84855487c8-46cpz" Oct 30 09:24:09.031111 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:24:09.031079 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod343dbb0c_f2d3_41cf_bc2c_283eba43e35d.slice/crio-1d2e740bbddb557b366ef8a1ae13d682cca296e9379fcd4b7f3b7df0e25b47ec WatchSource:0}: Error finding container 1d2e740bbddb557b366ef8a1ae13d682cca296e9379fcd4b7f3b7df0e25b47ec: Status 404 returned error can't find the container with id 1d2e740bbddb557b366ef8a1ae13d682cca296e9379fcd4b7f3b7df0e25b47ec Oct 30 09:24:09.033860 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:09.033833 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4fc2e2-84855487c8-46cpz"] Oct 30 09:24:09.520467 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:09.520431 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4fc2e2-84855487c8-46cpz" event={"ID":"343dbb0c-f2d3-41cf-bc2c-283eba43e35d","Type":"ContainerStarted","Data":"339e82d667875decc0cdf6280bbcbecb19a6c7f66e2ea83640557233bbbfef9e"} Oct 30 09:24:09.520874 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:09.520491 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4fc2e2-84855487c8-46cpz" event={"ID":"343dbb0c-f2d3-41cf-bc2c-283eba43e35d","Type":"ContainerStarted","Data":"1d2e740bbddb557b366ef8a1ae13d682cca296e9379fcd4b7f3b7df0e25b47ec"} Oct 30 09:24:09.520874 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:09.520533 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-4fc2e2-84855487c8-46cpz" Oct 30 09:24:09.564650 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:09.564585 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-4fc2e2-84855487c8-46cpz" podStartSLOduration=1.564565738 podStartE2EDuration="1.564565738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:24:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:24:09.559505406 +0000 UTC m=+1809.821066411" watchObservedRunningTime="2026-10-30 09:24:09.564565738 +0000 UTC m=+1809.826126734" Oct 30 09:24:13.539794 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:13.539761 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cbe974-b85c966d6-6fqhj"] Oct 30 09:24:13.576331 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:13.576302 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cbe974-d65fcff9-6hvv7"] Oct 30 09:24:18.540939 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:18.540887 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbe974-b85c966d6-6fqhj" podUID="12a110c3-50f9-4443-96cc-fbe6245adc61" containerName="payload-processing" containerID="cri-o://50449a1654ae1bda53eb3b470bec3507f1ca898643a1ba851d51719d40da21e8" gracePeriod=25 Oct 30 09:24:18.577148 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:18.577117 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbe974-d65fcff9-6hvv7" podUID="2754d28e-7bd7-457c-b087-6074d969e0d7" containerName="payload-pre-processing" containerID="cri-o://6e75ebee0e55aecfa0154ecf834b1b8d66cbdc2a1d52ede50ccd5ff901c11b53" gracePeriod=25 Oct 30 09:24:18.940558 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:18.940524 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbe974-b85c966d6-6fqhj" Oct 30 09:24:19.004741 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:19.004714 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbe974-d65fcff9-6hvv7" Oct 30 09:24:19.073723 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:19.073693 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12a110c3-50f9-4443-96cc-fbe6245adc61-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12a110c3-50f9-4443-96cc-fbe6245adc61-plugins-config-volume\") pod \"12a110c3-50f9-4443-96cc-fbe6245adc61\" (UID: \"12a110c3-50f9-4443-96cc-fbe6245adc61\") " Oct 30 09:24:19.073917 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:19.073744 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12a110c3-50f9-4443-96cc-fbe6245adc61-kube-api-access-5sdcz\" (UniqueName: \"kubernetes.io/projected/12a110c3-50f9-4443-96cc-fbe6245adc61-kube-api-access-5sdcz\") pod \"12a110c3-50f9-4443-96cc-fbe6245adc61\" (UID: \"12a110c3-50f9-4443-96cc-fbe6245adc61\") " Oct 30 09:24:19.074539 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:19.074491 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12a110c3-50f9-4443-96cc-fbe6245adc61-plugins-config-volume" pod "12a110c3-50f9-4443-96cc-fbe6245adc61" (UID: "12a110c3-50f9-4443-96cc-fbe6245adc61"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:24:19.079760 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:19.079691 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12a110c3-50f9-4443-96cc-fbe6245adc61-kube-api-access-5sdcz" pod "12a110c3-50f9-4443-96cc-fbe6245adc61" (UID: "12a110c3-50f9-4443-96cc-fbe6245adc61"). InnerVolumeSpecName "kube-api-access-5sdcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:24:19.175782 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:19.175750 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2754d28e-7bd7-457c-b087-6074d969e0d7-kube-api-access-r4d5n\" (UniqueName: \"kubernetes.io/projected/2754d28e-7bd7-457c-b087-6074d969e0d7-kube-api-access-r4d5n\") pod \"2754d28e-7bd7-457c-b087-6074d969e0d7\" (UID: \"2754d28e-7bd7-457c-b087-6074d969e0d7\") " Oct 30 09:24:19.175963 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:19.175789 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2754d28e-7bd7-457c-b087-6074d969e0d7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2754d28e-7bd7-457c-b087-6074d969e0d7-plugins-config-volume\") pod \"2754d28e-7bd7-457c-b087-6074d969e0d7\" (UID: \"2754d28e-7bd7-457c-b087-6074d969e0d7\") " Oct 30 09:24:19.176230 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:19.176204 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2754d28e-7bd7-457c-b087-6074d969e0d7-plugins-config-volume" pod "2754d28e-7bd7-457c-b087-6074d969e0d7" (UID: "2754d28e-7bd7-457c-b087-6074d969e0d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:24:19.176628 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:19.176605 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2754d28e-7bd7-457c-b087-6074d969e0d7-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:24:19.176713 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:19.176633 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12a110c3-50f9-4443-96cc-fbe6245adc61-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:24:19.176713 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:19.176648 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sdcz\" (UniqueName: \"kubernetes.io/projected/12a110c3-50f9-4443-96cc-fbe6245adc61-kube-api-access-5sdcz\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:24:19.177971 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:19.177948 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2754d28e-7bd7-457c-b087-6074d969e0d7-kube-api-access-r4d5n" pod "2754d28e-7bd7-457c-b087-6074d969e0d7" (UID: "2754d28e-7bd7-457c-b087-6074d969e0d7"). InnerVolumeSpecName "kube-api-access-r4d5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:24:19.278343 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:19.278311 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4d5n\" (UniqueName: \"kubernetes.io/projected/2754d28e-7bd7-457c-b087-6074d969e0d7-kube-api-access-r4d5n\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:24:19.604048 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:19.604016 4562 generic.go:356] "Generic (PLEG): container finished" podID="12a110c3-50f9-4443-96cc-fbe6245adc61" containerID="50449a1654ae1bda53eb3b470bec3507f1ca898643a1ba851d51719d40da21e8" exitCode=0 Oct 30 09:24:19.604447 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:19.604060 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbe974-b85c966d6-6fqhj" Oct 30 09:24:19.604447 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:19.604096 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbe974-b85c966d6-6fqhj" event={"ID":"12a110c3-50f9-4443-96cc-fbe6245adc61","Type":"ContainerDied","Data":"50449a1654ae1bda53eb3b470bec3507f1ca898643a1ba851d51719d40da21e8"} Oct 30 09:24:19.604447 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:19.604150 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbe974-b85c966d6-6fqhj" event={"ID":"12a110c3-50f9-4443-96cc-fbe6245adc61","Type":"ContainerDied","Data":"c682103b7b659c95635a5ac4fff23747be2289f8cad8bd922684c661555d0ff4"} Oct 30 09:24:19.604447 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:19.604172 4562 scope.go:122] "RemoveContainer" containerID="50449a1654ae1bda53eb3b470bec3507f1ca898643a1ba851d51719d40da21e8" Oct 30 09:24:19.605698 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:19.605678 4562 generic.go:356] "Generic (PLEG): container finished" podID="2754d28e-7bd7-457c-b087-6074d969e0d7" containerID="6e75ebee0e55aecfa0154ecf834b1b8d66cbdc2a1d52ede50ccd5ff901c11b53" exitCode=0 Oct 30 09:24:19.605776 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:19.605706 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbe974-d65fcff9-6hvv7" Oct 30 09:24:19.605826 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:19.605769 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbe974-d65fcff9-6hvv7" event={"ID":"2754d28e-7bd7-457c-b087-6074d969e0d7","Type":"ContainerDied","Data":"6e75ebee0e55aecfa0154ecf834b1b8d66cbdc2a1d52ede50ccd5ff901c11b53"} Oct 30 09:24:19.605826 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:19.605812 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbe974-d65fcff9-6hvv7" event={"ID":"2754d28e-7bd7-457c-b087-6074d969e0d7","Type":"ContainerDied","Data":"171c54dc78061553f87dc0a0702fcb876228c07d1709da91f61399ebede50146"} Oct 30 09:24:19.615659 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:19.615200 4562 scope.go:122] "RemoveContainer" containerID="50449a1654ae1bda53eb3b470bec3507f1ca898643a1ba851d51719d40da21e8" Oct 30 09:24:19.615659 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:24:19.615559 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50449a1654ae1bda53eb3b470bec3507f1ca898643a1ba851d51719d40da21e8\": container with ID starting with 50449a1654ae1bda53eb3b470bec3507f1ca898643a1ba851d51719d40da21e8 not found: ID does not exist" containerID="50449a1654ae1bda53eb3b470bec3507f1ca898643a1ba851d51719d40da21e8" Oct 30 09:24:19.615659 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:19.615594 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50449a1654ae1bda53eb3b470bec3507f1ca898643a1ba851d51719d40da21e8"} err="failed to get container status \"50449a1654ae1bda53eb3b470bec3507f1ca898643a1ba851d51719d40da21e8\": rpc error: code = NotFound desc = could not find container \"50449a1654ae1bda53eb3b470bec3507f1ca898643a1ba851d51719d40da21e8\": container with ID starting with 50449a1654ae1bda53eb3b470bec3507f1ca898643a1ba851d51719d40da21e8 not found: ID does not exist" Oct 30 09:24:19.615659 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:19.615613 4562 scope.go:122] "RemoveContainer" containerID="6e75ebee0e55aecfa0154ecf834b1b8d66cbdc2a1d52ede50ccd5ff901c11b53" Oct 30 09:24:19.625418 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:19.625371 4562 scope.go:122] "RemoveContainer" containerID="6e75ebee0e55aecfa0154ecf834b1b8d66cbdc2a1d52ede50ccd5ff901c11b53" Oct 30 09:24:19.625747 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:24:19.625723 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e75ebee0e55aecfa0154ecf834b1b8d66cbdc2a1d52ede50ccd5ff901c11b53\": container with ID starting with 6e75ebee0e55aecfa0154ecf834b1b8d66cbdc2a1d52ede50ccd5ff901c11b53 not found: ID does not exist" containerID="6e75ebee0e55aecfa0154ecf834b1b8d66cbdc2a1d52ede50ccd5ff901c11b53" Oct 30 09:24:19.625853 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:19.625752 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e75ebee0e55aecfa0154ecf834b1b8d66cbdc2a1d52ede50ccd5ff901c11b53"} err="failed to get container status \"6e75ebee0e55aecfa0154ecf834b1b8d66cbdc2a1d52ede50ccd5ff901c11b53\": rpc error: code = NotFound desc = could not find container \"6e75ebee0e55aecfa0154ecf834b1b8d66cbdc2a1d52ede50ccd5ff901c11b53\": container with ID starting with 6e75ebee0e55aecfa0154ecf834b1b8d66cbdc2a1d52ede50ccd5ff901c11b53 not found: ID does not exist" Oct 30 09:24:19.658074 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:19.658049 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cbe974-d65fcff9-6hvv7"] Oct 30 09:24:19.674635 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:19.671881 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cbe974-d65fcff9-6hvv7"] Oct 30 09:24:19.695432 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:19.695404 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cbe974-b85c966d6-6fqhj"] Oct 30 09:24:19.700927 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:19.700902 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cbe974-b85c966d6-6fqhj"] Oct 30 09:24:19.877681 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:19.877609 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12a110c3-50f9-4443-96cc-fbe6245adc61" path="/var/lib/kubelet/pods/12a110c3-50f9-4443-96cc-fbe6245adc61/volumes" Oct 30 09:24:19.878079 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:19.878066 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2754d28e-7bd7-457c-b087-6074d969e0d7" path="/var/lib/kubelet/pods/2754d28e-7bd7-457c-b087-6074d969e0d7/volumes" Oct 30 09:24:20.532399 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:20.532358 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-4fc2e2-84855487c8-46cpz" Oct 30 09:24:26.161405 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:26.160715 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-962e2e-756c88587b-dnvhs"] Oct 30 09:24:26.192809 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:26.192776 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-962e2e-76469968d7-bx7fz"] Oct 30 09:24:28.645412 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:24:28.644581 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:24:31.161550 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:31.161514 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-962e2e-756c88587b-dnvhs" podUID="e70dc648-2cba-4c25-af40-e83ee85a58a1" containerName="payload-processing" containerID="cri-o://d053feb106811114fa0d85ea302c69e58d3c1a97356dbe51cddb221bfbf1c49f" gracePeriod=25 Oct 30 09:24:31.193332 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:31.193300 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-962e2e-76469968d7-bx7fz" podUID="e4df6984-60f1-4ed6-910c-2608e94ab9db" containerName="payload-pre-processing" containerID="cri-o://f824e625891be4e90a6b2467be398377547ce9b66229d2ac4e8f398007aeea69" gracePeriod=25 Oct 30 09:24:31.494955 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:31.494921 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-962e2e-756c88587b-dnvhs" Oct 30 09:24:31.572116 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:31.572085 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-962e2e-76469968d7-bx7fz" Oct 30 09:24:31.647523 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:31.647487 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e70dc648-2cba-4c25-af40-e83ee85a58a1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e70dc648-2cba-4c25-af40-e83ee85a58a1-plugins-config-volume\") pod \"e70dc648-2cba-4c25-af40-e83ee85a58a1\" (UID: \"e70dc648-2cba-4c25-af40-e83ee85a58a1\") " Oct 30 09:24:31.647752 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:31.647734 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e70dc648-2cba-4c25-af40-e83ee85a58a1-kube-api-access-4xqtv\" (UniqueName: \"kubernetes.io/projected/e70dc648-2cba-4c25-af40-e83ee85a58a1-kube-api-access-4xqtv\") pod \"e70dc648-2cba-4c25-af40-e83ee85a58a1\" (UID: \"e70dc648-2cba-4c25-af40-e83ee85a58a1\") " Oct 30 09:24:31.648008 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:31.647975 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e70dc648-2cba-4c25-af40-e83ee85a58a1-plugins-config-volume" pod "e70dc648-2cba-4c25-af40-e83ee85a58a1" (UID: "e70dc648-2cba-4c25-af40-e83ee85a58a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:24:31.648551 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:31.648526 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e70dc648-2cba-4c25-af40-e83ee85a58a1-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:24:31.650136 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:31.650108 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e70dc648-2cba-4c25-af40-e83ee85a58a1-kube-api-access-4xqtv" pod "e70dc648-2cba-4c25-af40-e83ee85a58a1" (UID: "e70dc648-2cba-4c25-af40-e83ee85a58a1"). InnerVolumeSpecName "kube-api-access-4xqtv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:24:31.749678 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:31.749648 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4df6984-60f1-4ed6-910c-2608e94ab9db-kube-api-access-lhjzp\" (UniqueName: \"kubernetes.io/projected/e4df6984-60f1-4ed6-910c-2608e94ab9db-kube-api-access-lhjzp\") pod \"e4df6984-60f1-4ed6-910c-2608e94ab9db\" (UID: \"e4df6984-60f1-4ed6-910c-2608e94ab9db\") " Oct 30 09:24:31.755720 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:31.749785 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4df6984-60f1-4ed6-910c-2608e94ab9db-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4df6984-60f1-4ed6-910c-2608e94ab9db-plugins-config-volume\") pod \"e4df6984-60f1-4ed6-910c-2608e94ab9db\" (UID: \"e4df6984-60f1-4ed6-910c-2608e94ab9db\") " Oct 30 09:24:31.755720 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:31.750200 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4df6984-60f1-4ed6-910c-2608e94ab9db-plugins-config-volume" pod "e4df6984-60f1-4ed6-910c-2608e94ab9db" (UID: "e4df6984-60f1-4ed6-910c-2608e94ab9db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:24:31.755720 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:31.750581 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4df6984-60f1-4ed6-910c-2608e94ab9db-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:24:31.755720 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:31.750601 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xqtv\" (UniqueName: \"kubernetes.io/projected/e70dc648-2cba-4c25-af40-e83ee85a58a1-kube-api-access-4xqtv\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:24:31.755720 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:31.751727 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4df6984-60f1-4ed6-910c-2608e94ab9db-kube-api-access-lhjzp" pod "e4df6984-60f1-4ed6-910c-2608e94ab9db" (UID: "e4df6984-60f1-4ed6-910c-2608e94ab9db"). InnerVolumeSpecName "kube-api-access-lhjzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:24:31.847954 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:31.847903 4562 generic.go:356] "Generic (PLEG): container finished" podID="e70dc648-2cba-4c25-af40-e83ee85a58a1" containerID="d053feb106811114fa0d85ea302c69e58d3c1a97356dbe51cddb221bfbf1c49f" exitCode=0 Oct 30 09:24:31.848044 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:31.848013 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-962e2e-756c88587b-dnvhs" event={"ID":"e70dc648-2cba-4c25-af40-e83ee85a58a1","Type":"ContainerDied","Data":"d053feb106811114fa0d85ea302c69e58d3c1a97356dbe51cddb221bfbf1c49f"} Oct 30 09:24:31.848137 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:31.848119 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-962e2e-756c88587b-dnvhs" event={"ID":"e70dc648-2cba-4c25-af40-e83ee85a58a1","Type":"ContainerDied","Data":"04d716f80fbe4608bd5f166a3f436238def684ede867ed6a74a515f8b6ae6f12"} Oct 30 09:24:31.848193 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:31.848167 4562 scope.go:122] "RemoveContainer" containerID="d053feb106811114fa0d85ea302c69e58d3c1a97356dbe51cddb221bfbf1c49f" Oct 30 09:24:31.848494 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:31.848474 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-962e2e-756c88587b-dnvhs" Oct 30 09:24:31.854010 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:31.853964 4562 generic.go:356] "Generic (PLEG): container finished" podID="e4df6984-60f1-4ed6-910c-2608e94ab9db" containerID="f824e625891be4e90a6b2467be398377547ce9b66229d2ac4e8f398007aeea69" exitCode=0 Oct 30 09:24:31.854141 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:31.854054 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-962e2e-76469968d7-bx7fz" Oct 30 09:24:31.854254 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:31.854206 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-962e2e-76469968d7-bx7fz" event={"ID":"e4df6984-60f1-4ed6-910c-2608e94ab9db","Type":"ContainerDied","Data":"f824e625891be4e90a6b2467be398377547ce9b66229d2ac4e8f398007aeea69"} Oct 30 09:24:31.854325 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:31.854265 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-962e2e-76469968d7-bx7fz" event={"ID":"e4df6984-60f1-4ed6-910c-2608e94ab9db","Type":"ContainerDied","Data":"f9738733f8734bf6af9ac5df975ff8445e7c33650142d5bf7d8d3791a3ef8bbd"} Oct 30 09:24:31.855412 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:31.855373 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhjzp\" (UniqueName: \"kubernetes.io/projected/e4df6984-60f1-4ed6-910c-2608e94ab9db-kube-api-access-lhjzp\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:24:31.896785 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:31.896763 4562 scope.go:122] "RemoveContainer" containerID="d053feb106811114fa0d85ea302c69e58d3c1a97356dbe51cddb221bfbf1c49f" Oct 30 09:24:31.897092 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:24:31.897069 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d053feb106811114fa0d85ea302c69e58d3c1a97356dbe51cddb221bfbf1c49f\": container with ID starting with d053feb106811114fa0d85ea302c69e58d3c1a97356dbe51cddb221bfbf1c49f not found: ID does not exist" containerID="d053feb106811114fa0d85ea302c69e58d3c1a97356dbe51cddb221bfbf1c49f" Oct 30 09:24:31.897174 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:31.897104 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d053feb106811114fa0d85ea302c69e58d3c1a97356dbe51cddb221bfbf1c49f"} err="failed to get container status \"d053feb106811114fa0d85ea302c69e58d3c1a97356dbe51cddb221bfbf1c49f\": rpc error: code = NotFound desc = could not find container \"d053feb106811114fa0d85ea302c69e58d3c1a97356dbe51cddb221bfbf1c49f\": container with ID starting with d053feb106811114fa0d85ea302c69e58d3c1a97356dbe51cddb221bfbf1c49f not found: ID does not exist" Oct 30 09:24:31.897174 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:31.897127 4562 scope.go:122] "RemoveContainer" containerID="f824e625891be4e90a6b2467be398377547ce9b66229d2ac4e8f398007aeea69" Oct 30 09:24:31.906363 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:31.906346 4562 scope.go:122] "RemoveContainer" containerID="f824e625891be4e90a6b2467be398377547ce9b66229d2ac4e8f398007aeea69" Oct 30 09:24:31.906636 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:24:31.906614 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f824e625891be4e90a6b2467be398377547ce9b66229d2ac4e8f398007aeea69\": container with ID starting with f824e625891be4e90a6b2467be398377547ce9b66229d2ac4e8f398007aeea69 not found: ID does not exist" containerID="f824e625891be4e90a6b2467be398377547ce9b66229d2ac4e8f398007aeea69" Oct 30 09:24:31.906706 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:31.906643 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f824e625891be4e90a6b2467be398377547ce9b66229d2ac4e8f398007aeea69"} err="failed to get container status \"f824e625891be4e90a6b2467be398377547ce9b66229d2ac4e8f398007aeea69\": rpc error: code = NotFound desc = could not find container \"f824e625891be4e90a6b2467be398377547ce9b66229d2ac4e8f398007aeea69\": container with ID starting with f824e625891be4e90a6b2467be398377547ce9b66229d2ac4e8f398007aeea69 not found: ID does not exist" Oct 30 09:24:31.938130 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:31.938106 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-962e2e-76469968d7-bx7fz"] Oct 30 09:24:31.941667 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:31.941647 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-962e2e-76469968d7-bx7fz"] Oct 30 09:24:31.976751 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:31.976725 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-962e2e-756c88587b-dnvhs"] Oct 30 09:24:31.980040 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:31.980020 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-962e2e-756c88587b-dnvhs"] Oct 30 09:24:32.176079 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:32.175991 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4fc2e2-66476768fb-rzqx7"] Oct 30 09:24:32.177198 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:32.177174 4562 cpu_manager.go:414] "containerMap: removing container" podUID="2754d28e-7bd7-457c-b087-6074d969e0d7" containerName="payload-pre-processing" Oct 30 09:24:32.177246 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:32.177203 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2754d28e-7bd7-457c-b087-6074d969e0d7" containerName="payload-pre-processing" Oct 30 09:24:32.177246 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:32.177234 4562 cpu_manager.go:414] "containerMap: removing container" podUID="e4df6984-60f1-4ed6-910c-2608e94ab9db" containerName="payload-pre-processing" Oct 30 09:24:32.177246 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:32.177243 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4df6984-60f1-4ed6-910c-2608e94ab9db" containerName="payload-pre-processing" Oct 30 09:24:32.177342 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:32.177260 4562 cpu_manager.go:414] "containerMap: removing container" podUID="12a110c3-50f9-4443-96cc-fbe6245adc61" containerName="payload-processing" Oct 30 09:24:32.177342 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:32.177268 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12a110c3-50f9-4443-96cc-fbe6245adc61" containerName="payload-processing" Oct 30 09:24:32.177342 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:32.177291 4562 cpu_manager.go:414] "containerMap: removing container" podUID="e70dc648-2cba-4c25-af40-e83ee85a58a1" containerName="payload-processing" Oct 30 09:24:32.177342 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:32.177300 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e70dc648-2cba-4c25-af40-e83ee85a58a1" containerName="payload-processing" Oct 30 09:24:32.177518 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:32.177498 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="2754d28e-7bd7-457c-b087-6074d969e0d7" containerName="payload-pre-processing" Oct 30 09:24:32.177518 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:32.177519 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4df6984-60f1-4ed6-910c-2608e94ab9db" containerName="payload-pre-processing" Oct 30 09:24:32.177681 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:32.177529 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="12a110c3-50f9-4443-96cc-fbe6245adc61" containerName="payload-processing" Oct 30 09:24:32.177681 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:32.177550 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="e70dc648-2cba-4c25-af40-e83ee85a58a1" containerName="payload-processing" Oct 30 09:24:32.183304 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:32.183286 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4fc2e2-66476768fb-rzqx7" Oct 30 09:24:32.199257 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:32.199233 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4fc2e2-66476768fb-rzqx7"] Oct 30 09:24:32.199583 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:32.199566 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-4fc2e2-serving-cert\"" Oct 30 09:24:32.362573 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:32.362542 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5a258ce-a81b-4605-8e83-ad8a63d4bc9c-openshift-service-ca\") pod \"maas-api-e2e-api-a-4fc2e2-66476768fb-rzqx7\" (UID: \"b5a258ce-a81b-4605-8e83-ad8a63d4bc9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4fc2e2-66476768fb-rzqx7" Oct 30 09:24:32.362736 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:32.362654 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crkfh\" (UniqueName: \"kubernetes.io/projected/b5a258ce-a81b-4605-8e83-ad8a63d4bc9c-kube-api-access-crkfh\") pod \"maas-api-e2e-api-a-4fc2e2-66476768fb-rzqx7\" (UID: \"b5a258ce-a81b-4605-8e83-ad8a63d4bc9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4fc2e2-66476768fb-rzqx7" Oct 30 09:24:32.362851 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:32.362830 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5a258ce-a81b-4605-8e83-ad8a63d4bc9c-metrics-certs\") pod \"maas-api-e2e-api-a-4fc2e2-66476768fb-rzqx7\" (UID: \"b5a258ce-a81b-4605-8e83-ad8a63d4bc9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4fc2e2-66476768fb-rzqx7" Oct 30 09:24:32.362903 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:32.362889 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5a258ce-a81b-4605-8e83-ad8a63d4bc9c-maas-api-tls\") pod \"maas-api-e2e-api-a-4fc2e2-66476768fb-rzqx7\" (UID: \"b5a258ce-a81b-4605-8e83-ad8a63d4bc9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4fc2e2-66476768fb-rzqx7" Oct 30 09:24:32.472286 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:32.472242 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5a258ce-a81b-4605-8e83-ad8a63d4bc9c-openshift-service-ca\") pod \"maas-api-e2e-api-a-4fc2e2-66476768fb-rzqx7\" (UID: \"b5a258ce-a81b-4605-8e83-ad8a63d4bc9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4fc2e2-66476768fb-rzqx7" Oct 30 09:24:32.472715 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:32.472689 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crkfh\" (UniqueName: \"kubernetes.io/projected/b5a258ce-a81b-4605-8e83-ad8a63d4bc9c-kube-api-access-crkfh\") pod \"maas-api-e2e-api-a-4fc2e2-66476768fb-rzqx7\" (UID: \"b5a258ce-a81b-4605-8e83-ad8a63d4bc9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4fc2e2-66476768fb-rzqx7" Oct 30 09:24:32.472859 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:32.472842 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5a258ce-a81b-4605-8e83-ad8a63d4bc9c-metrics-certs\") pod \"maas-api-e2e-api-a-4fc2e2-66476768fb-rzqx7\" (UID: \"b5a258ce-a81b-4605-8e83-ad8a63d4bc9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4fc2e2-66476768fb-rzqx7" Oct 30 09:24:32.472933 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:32.472903 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5a258ce-a81b-4605-8e83-ad8a63d4bc9c-maas-api-tls\") pod \"maas-api-e2e-api-a-4fc2e2-66476768fb-rzqx7\" (UID: \"b5a258ce-a81b-4605-8e83-ad8a63d4bc9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4fc2e2-66476768fb-rzqx7" Oct 30 09:24:32.474606 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:32.474581 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5a258ce-a81b-4605-8e83-ad8a63d4bc9c-openshift-service-ca\") pod \"maas-api-e2e-api-a-4fc2e2-66476768fb-rzqx7\" (UID: \"b5a258ce-a81b-4605-8e83-ad8a63d4bc9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4fc2e2-66476768fb-rzqx7" Oct 30 09:24:32.476349 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:32.476323 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5a258ce-a81b-4605-8e83-ad8a63d4bc9c-maas-api-tls\") pod \"maas-api-e2e-api-a-4fc2e2-66476768fb-rzqx7\" (UID: \"b5a258ce-a81b-4605-8e83-ad8a63d4bc9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4fc2e2-66476768fb-rzqx7" Oct 30 09:24:32.476531 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:32.476513 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5a258ce-a81b-4605-8e83-ad8a63d4bc9c-metrics-certs\") pod \"maas-api-e2e-api-a-4fc2e2-66476768fb-rzqx7\" (UID: \"b5a258ce-a81b-4605-8e83-ad8a63d4bc9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4fc2e2-66476768fb-rzqx7" Oct 30 09:24:32.553310 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:32.553269 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crkfh\" (UniqueName: \"kubernetes.io/projected/b5a258ce-a81b-4605-8e83-ad8a63d4bc9c-kube-api-access-crkfh\") pod \"maas-api-e2e-api-a-4fc2e2-66476768fb-rzqx7\" (UID: \"b5a258ce-a81b-4605-8e83-ad8a63d4bc9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4fc2e2-66476768fb-rzqx7" Oct 30 09:24:32.805702 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:32.805607 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4fc2e2-66476768fb-rzqx7" Oct 30 09:24:32.869741 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:32.869654 4562 generic.go:356] "Generic (PLEG): container finished" podID="286eb740-46f4-44ba-80f1-0de69bd89861" containerID="bce982b30d40131ba088ef0c2518e3b3d42db0314ab6426dde722029d794bcd8" exitCode=0 Oct 30 09:24:32.869741 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:32.869725 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f" event={"ID":"286eb740-46f4-44ba-80f1-0de69bd89861","Type":"ContainerDied","Data":"bce982b30d40131ba088ef0c2518e3b3d42db0314ab6426dde722029d794bcd8"} Oct 30 09:24:32.870685 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:32.870179 4562 scope.go:122] "RemoveContainer" containerID="bce982b30d40131ba088ef0c2518e3b3d42db0314ab6426dde722029d794bcd8" Oct 30 09:24:32.876198 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:32.876162 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:24:32.876426 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:32.876401 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:24:32.876636 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:32.876614 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:24:33.158336 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:33.158148 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4fc2e2-66476768fb-rzqx7"] Oct 30 09:24:33.796044 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:33.796011 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f" Oct 30 09:24:33.880467 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:33.879483 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4df6984-60f1-4ed6-910c-2608e94ab9db" path="/var/lib/kubelet/pods/e4df6984-60f1-4ed6-910c-2608e94ab9db/volumes" Oct 30 09:24:33.880467 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:33.880161 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e70dc648-2cba-4c25-af40-e83ee85a58a1" path="/var/lib/kubelet/pods/e70dc648-2cba-4c25-af40-e83ee85a58a1/volumes" Oct 30 09:24:33.902883 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:33.902841 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4fc2e2-66476768fb-rzqx7" event={"ID":"b5a258ce-a81b-4605-8e83-ad8a63d4bc9c","Type":"ContainerStarted","Data":"baabe2cdd3b6860090cfe8e0fe1dd664dcd6423d0e2bedd41237e863c3e09f9d"} Oct 30 09:24:33.903079 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:33.902916 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4fc2e2-66476768fb-rzqx7" Oct 30 09:24:33.903079 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:33.902946 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4fc2e2-66476768fb-rzqx7" event={"ID":"b5a258ce-a81b-4605-8e83-ad8a63d4bc9c","Type":"ContainerStarted","Data":"00e2d38efa5218755bc4037daba6ee331ea91bb8303fafb2b80bae9c78b9b6d0"} Oct 30 09:24:33.905305 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:33.905276 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f" event={"ID":"286eb740-46f4-44ba-80f1-0de69bd89861","Type":"ContainerStarted","Data":"088b8fdd949b1a7c4e46586e842aedc814000f85df6fc2659b3ba3679e0d9217"} Oct 30 09:24:33.968291 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:33.968222 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4fc2e2-66476768fb-rzqx7" podStartSLOduration=1.5695076860000001 podStartE2EDuration="1.968204305s" totalImagesPullingTime="398.696619ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:24:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:24:33.96490878 +0000 UTC m=+1834.226469792" watchObservedRunningTime="2026-10-30 09:24:33.968204305 +0000 UTC m=+1834.229765294" Oct 30 09:24:34.163311 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:34.163227 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f" Oct 30 09:24:34.176403 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:34.172515 4562 patch_prober.go:29] interesting pod/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 30 09:24:34.176403 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:34.172596 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f" podUID="286eb740-46f4-44ba-80f1-0de69bd89861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 30 09:24:35.163763 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:35.163731 4562 patch_prober.go:29] interesting pod/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 30 09:24:35.164141 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:35.163782 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f" podUID="286eb740-46f4-44ba-80f1-0de69bd89861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 30 09:24:36.173406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:36.152707 4562 patch_prober.go:29] interesting pod/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 30 09:24:36.173406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:36.152766 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f" podUID="286eb740-46f4-44ba-80f1-0de69bd89861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 30 09:24:36.256408 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:36.249208 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cbe974-6d74bdc9f6-w4szp"] Oct 30 09:24:36.353691 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:36.338793 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbe974-6d74bdc9f6-w4szp" Oct 30 09:24:36.353691 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:36.340060 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cbe974-6d74bdc9f6-w4szp"] Oct 30 09:24:36.385606 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:36.363779 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cbe974-86947478d7-f4rmw"] Oct 30 09:24:36.385606 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:36.379085 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-cbe974-dockercfg-lgkcs\"" Oct 30 09:24:36.385606 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:36.381154 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-cbe974\"" Oct 30 09:24:36.387032 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:36.387009 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cbe974-86947478d7-f4rmw"] Oct 30 09:24:36.394780 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:36.387268 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbe974-86947478d7-f4rmw" Oct 30 09:24:36.483430 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:36.483396 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1502d0e9-69e7-4c24-b69c-76ccc9d382f0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-cbe974-6d74bdc9f6-w4szp\" (UID: \"1502d0e9-69e7-4c24-b69c-76ccc9d382f0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cbe974-6d74bdc9f6-w4szp" Oct 30 09:24:36.483678 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:36.483656 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2t6z\" (UniqueName: \"kubernetes.io/projected/1502d0e9-69e7-4c24-b69c-76ccc9d382f0-kube-api-access-b2t6z\") pod \"payload-processing-e2e-ipp-b-cbe974-6d74bdc9f6-w4szp\" (UID: \"1502d0e9-69e7-4c24-b69c-76ccc9d382f0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cbe974-6d74bdc9f6-w4szp" Oct 30 09:24:36.585633 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:36.585589 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cb673aa-7113-4265-a9a4-e8e78651f02b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-cbe974-86947478d7-f4rmw\" (UID: \"7cb673aa-7113-4265-a9a4-e8e78651f02b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbe974-86947478d7-f4rmw" Oct 30 09:24:36.585828 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:36.585657 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1502d0e9-69e7-4c24-b69c-76ccc9d382f0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-cbe974-6d74bdc9f6-w4szp\" (UID: \"1502d0e9-69e7-4c24-b69c-76ccc9d382f0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cbe974-6d74bdc9f6-w4szp" Oct 30 09:24:36.585900 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:36.585824 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2t6z\" (UniqueName: \"kubernetes.io/projected/1502d0e9-69e7-4c24-b69c-76ccc9d382f0-kube-api-access-b2t6z\") pod \"payload-processing-e2e-ipp-b-cbe974-6d74bdc9f6-w4szp\" (UID: \"1502d0e9-69e7-4c24-b69c-76ccc9d382f0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cbe974-6d74bdc9f6-w4szp" Oct 30 09:24:36.585900 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:36.585873 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znjtr\" (UniqueName: \"kubernetes.io/projected/7cb673aa-7113-4265-a9a4-e8e78651f02b-kube-api-access-znjtr\") pod \"payload-pre-processing-e2e-ipp-b-cbe974-86947478d7-f4rmw\" (UID: \"7cb673aa-7113-4265-a9a4-e8e78651f02b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbe974-86947478d7-f4rmw" Oct 30 09:24:36.586405 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:36.586369 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1502d0e9-69e7-4c24-b69c-76ccc9d382f0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-cbe974-6d74bdc9f6-w4szp\" (UID: \"1502d0e9-69e7-4c24-b69c-76ccc9d382f0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cbe974-6d74bdc9f6-w4szp" Oct 30 09:24:36.605759 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:36.605730 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2t6z\" (UniqueName: \"kubernetes.io/projected/1502d0e9-69e7-4c24-b69c-76ccc9d382f0-kube-api-access-b2t6z\") pod \"payload-processing-e2e-ipp-b-cbe974-6d74bdc9f6-w4szp\" (UID: \"1502d0e9-69e7-4c24-b69c-76ccc9d382f0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cbe974-6d74bdc9f6-w4szp" Oct 30 09:24:36.687489 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:36.687453 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cb673aa-7113-4265-a9a4-e8e78651f02b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-cbe974-86947478d7-f4rmw\" (UID: \"7cb673aa-7113-4265-a9a4-e8e78651f02b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbe974-86947478d7-f4rmw" Oct 30 09:24:36.687656 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:36.687607 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znjtr\" (UniqueName: \"kubernetes.io/projected/7cb673aa-7113-4265-a9a4-e8e78651f02b-kube-api-access-znjtr\") pod \"payload-pre-processing-e2e-ipp-b-cbe974-86947478d7-f4rmw\" (UID: \"7cb673aa-7113-4265-a9a4-e8e78651f02b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbe974-86947478d7-f4rmw" Oct 30 09:24:36.688197 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:36.688176 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cb673aa-7113-4265-a9a4-e8e78651f02b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-cbe974-86947478d7-f4rmw\" (UID: \"7cb673aa-7113-4265-a9a4-e8e78651f02b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbe974-86947478d7-f4rmw" Oct 30 09:24:36.708229 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:36.708203 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znjtr\" (UniqueName: \"kubernetes.io/projected/7cb673aa-7113-4265-a9a4-e8e78651f02b-kube-api-access-znjtr\") pod \"payload-pre-processing-e2e-ipp-b-cbe974-86947478d7-f4rmw\" (UID: \"7cb673aa-7113-4265-a9a4-e8e78651f02b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbe974-86947478d7-f4rmw" Oct 30 09:24:36.714899 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:36.714872 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbe974-6d74bdc9f6-w4szp" Oct 30 09:24:36.721909 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:36.721882 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbe974-86947478d7-f4rmw" Oct 30 09:24:37.215438 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:37.200345 4562 patch_prober.go:29] interesting pod/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 30 09:24:37.215438 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:37.200521 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f" podUID="286eb740-46f4-44ba-80f1-0de69bd89861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 30 09:24:37.283839 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:24:37.283791 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1502d0e9_69e7_4c24_b69c_76ccc9d382f0.slice/crio-a14641044e60aefef56a5857450a9704917d5ea7b30ea113f49690ca408ba7eb WatchSource:0}: Error finding container a14641044e60aefef56a5857450a9704917d5ea7b30ea113f49690ca408ba7eb: Status 404 returned error can't find the container with id a14641044e60aefef56a5857450a9704917d5ea7b30ea113f49690ca408ba7eb Oct 30 09:24:37.286306 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:37.286284 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cbe974-6d74bdc9f6-w4szp"] Oct 30 09:24:37.297932 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:37.297905 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cbe974-86947478d7-f4rmw"] Oct 30 09:24:37.299559 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:24:37.299538 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cb673aa_7113_4265_a9a4_e8e78651f02b.slice/crio-4a826173a567f51b75c2d935f9eb1ba3ff45a3ccbf60babee14c16818c11df01 WatchSource:0}: Error finding container 4a826173a567f51b75c2d935f9eb1ba3ff45a3ccbf60babee14c16818c11df01: Status 404 returned error can't find the container with id 4a826173a567f51b75c2d935f9eb1ba3ff45a3ccbf60babee14c16818c11df01 Oct 30 09:24:38.148751 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:38.148710 4562 patch_prober.go:29] interesting pod/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 30 09:24:38.149043 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:38.149012 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f" podUID="286eb740-46f4-44ba-80f1-0de69bd89861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 30 09:24:38.227295 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:38.227257 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbe974-6d74bdc9f6-w4szp" event={"ID":"1502d0e9-69e7-4c24-b69c-76ccc9d382f0","Type":"ContainerStarted","Data":"7a8c0a6e08bf7b1abac71887ccf7e2f9125ded1d357ff36c842ba09426a5c8fb"} Oct 30 09:24:38.227762 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:38.227356 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbe974-6d74bdc9f6-w4szp" event={"ID":"1502d0e9-69e7-4c24-b69c-76ccc9d382f0","Type":"ContainerStarted","Data":"a14641044e60aefef56a5857450a9704917d5ea7b30ea113f49690ca408ba7eb"} Oct 30 09:24:38.227762 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:38.227421 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbe974-6d74bdc9f6-w4szp" Oct 30 09:24:38.228986 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:38.228957 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbe974-86947478d7-f4rmw" event={"ID":"7cb673aa-7113-4265-a9a4-e8e78651f02b","Type":"ContainerStarted","Data":"240e3f3e81c6d394040380232276bddf4638dec6a47127c6467dacd83b025746"} Oct 30 09:24:38.229133 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:38.229004 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbe974-86947478d7-f4rmw" event={"ID":"7cb673aa-7113-4265-a9a4-e8e78651f02b","Type":"ContainerStarted","Data":"4a826173a567f51b75c2d935f9eb1ba3ff45a3ccbf60babee14c16818c11df01"} Oct 30 09:24:38.229133 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:38.229055 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbe974-86947478d7-f4rmw" Oct 30 09:24:38.285395 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:38.285321 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbe974-6d74bdc9f6-w4szp" podStartSLOduration=3.285308984 podStartE2EDuration="3.285308984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:24:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:24:38.282612822 +0000 UTC m=+1838.544173811" watchObservedRunningTime="2026-10-30 09:24:38.285308984 +0000 UTC m=+1838.546869972" Oct 30 09:24:38.322954 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:38.322903 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbe974-86947478d7-f4rmw" podStartSLOduration=2.322889864 podStartE2EDuration="2.322889864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:24:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:24:38.321690613 +0000 UTC m=+1838.583251601" watchObservedRunningTime="2026-10-30 09:24:38.322889864 +0000 UTC m=+1838.584450852" Oct 30 09:24:39.110464 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:39.110430 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-58fadf-7748bcb6d6-9pwn4"] Oct 30 09:24:39.161407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:39.161110 4562 patch_prober.go:29] interesting pod/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 30 09:24:39.161407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:39.161176 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f" podUID="286eb740-46f4-44ba-80f1-0de69bd89861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 30 09:24:39.169471 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:39.169437 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-58fadf-5968d7d495-gprm9"] Oct 30 09:24:39.984404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:39.983662 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4fc2e2-66476768fb-rzqx7" Oct 30 09:24:40.179626 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:40.176729 4562 patch_prober.go:29] interesting pod/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 30 09:24:40.179626 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:40.176789 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f" podUID="286eb740-46f4-44ba-80f1-0de69bd89861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 30 09:24:41.150516 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:41.150486 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f" Oct 30 09:24:41.151026 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:41.151002 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f" Oct 30 09:24:41.151610 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:41.151590 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f" Oct 30 09:24:44.110787 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:44.110695 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-58fadf-7748bcb6d6-9pwn4" podUID="2fd05ff0-d658-452d-8b4a-399aca8f7876" containerName="payload-processing" containerID="cri-o://5c60c0e665f6f406eff6c8026dc02a810f4fdcceb784688edafa9d5a7e6c14b6" gracePeriod=25 Oct 30 09:24:44.170714 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:44.170670 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-58fadf-5968d7d495-gprm9" podUID="5bc980b8-abbe-4e78-bf92-cd6654479cbe" containerName="payload-pre-processing" containerID="cri-o://414f71432cf30512f53cd0b55bc7e3c0167c846de4fca35d86d68ad57dd93a6c" gracePeriod=25 Oct 30 09:24:44.281950 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:44.281923 4562 generic.go:356] "Generic (PLEG): container finished" podID="2fd05ff0-d658-452d-8b4a-399aca8f7876" containerID="5c60c0e665f6f406eff6c8026dc02a810f4fdcceb784688edafa9d5a7e6c14b6" exitCode=0 Oct 30 09:24:44.282079 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:44.281994 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-58fadf-7748bcb6d6-9pwn4" event={"ID":"2fd05ff0-d658-452d-8b4a-399aca8f7876","Type":"ContainerDied","Data":"5c60c0e665f6f406eff6c8026dc02a810f4fdcceb784688edafa9d5a7e6c14b6"} Oct 30 09:24:44.474396 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:44.474357 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-58fadf-7748bcb6d6-9pwn4" Oct 30 09:24:44.498637 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:44.498611 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-58fadf-5968d7d495-gprm9" Oct 30 09:24:44.579339 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:44.579299 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bc980b8-abbe-4e78-bf92-cd6654479cbe-kube-api-access-bc9hj\" (UniqueName: \"kubernetes.io/projected/5bc980b8-abbe-4e78-bf92-cd6654479cbe-kube-api-access-bc9hj\") pod \"5bc980b8-abbe-4e78-bf92-cd6654479cbe\" (UID: \"5bc980b8-abbe-4e78-bf92-cd6654479cbe\") " Oct 30 09:24:44.579542 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:44.579349 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bc980b8-abbe-4e78-bf92-cd6654479cbe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bc980b8-abbe-4e78-bf92-cd6654479cbe-plugins-config-volume\") pod \"5bc980b8-abbe-4e78-bf92-cd6654479cbe\" (UID: \"5bc980b8-abbe-4e78-bf92-cd6654479cbe\") " Oct 30 09:24:44.579542 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:44.579493 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fd05ff0-d658-452d-8b4a-399aca8f7876-kube-api-access-kh2np\" (UniqueName: \"kubernetes.io/projected/2fd05ff0-d658-452d-8b4a-399aca8f7876-kube-api-access-kh2np\") pod \"2fd05ff0-d658-452d-8b4a-399aca8f7876\" (UID: \"2fd05ff0-d658-452d-8b4a-399aca8f7876\") " Oct 30 09:24:44.579664 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:44.579614 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fd05ff0-d658-452d-8b4a-399aca8f7876-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fd05ff0-d658-452d-8b4a-399aca8f7876-plugins-config-volume\") pod \"2fd05ff0-d658-452d-8b4a-399aca8f7876\" (UID: \"2fd05ff0-d658-452d-8b4a-399aca8f7876\") " Oct 30 09:24:44.579989 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:44.579954 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bc980b8-abbe-4e78-bf92-cd6654479cbe-plugins-config-volume" pod "5bc980b8-abbe-4e78-bf92-cd6654479cbe" (UID: "5bc980b8-abbe-4e78-bf92-cd6654479cbe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:24:44.580102 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:44.580042 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fd05ff0-d658-452d-8b4a-399aca8f7876-plugins-config-volume" pod "2fd05ff0-d658-452d-8b4a-399aca8f7876" (UID: "2fd05ff0-d658-452d-8b4a-399aca8f7876"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:24:44.580452 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:44.580398 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bc980b8-abbe-4e78-bf92-cd6654479cbe-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:24:44.580452 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:44.580427 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fd05ff0-d658-452d-8b4a-399aca8f7876-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:24:44.581858 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:44.581831 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fd05ff0-d658-452d-8b4a-399aca8f7876-kube-api-access-kh2np" pod "2fd05ff0-d658-452d-8b4a-399aca8f7876" (UID: "2fd05ff0-d658-452d-8b4a-399aca8f7876"). InnerVolumeSpecName "kube-api-access-kh2np". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:24:44.582083 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:44.582061 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bc980b8-abbe-4e78-bf92-cd6654479cbe-kube-api-access-bc9hj" pod "5bc980b8-abbe-4e78-bf92-cd6654479cbe" (UID: "5bc980b8-abbe-4e78-bf92-cd6654479cbe"). InnerVolumeSpecName "kube-api-access-bc9hj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:24:44.682287 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:44.682220 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bc9hj\" (UniqueName: \"kubernetes.io/projected/5bc980b8-abbe-4e78-bf92-cd6654479cbe-kube-api-access-bc9hj\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:24:44.682287 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:44.682247 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kh2np\" (UniqueName: \"kubernetes.io/projected/2fd05ff0-d658-452d-8b4a-399aca8f7876-kube-api-access-kh2np\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:24:45.290827 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:45.290790 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-58fadf-7748bcb6d6-9pwn4" Oct 30 09:24:45.291232 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:45.290826 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-58fadf-7748bcb6d6-9pwn4" event={"ID":"2fd05ff0-d658-452d-8b4a-399aca8f7876","Type":"ContainerDied","Data":"1cc797d36fe6eed0a8629527e2684279dfb5c27e1bcc4cc63cc044bbbe45daa8"} Oct 30 09:24:45.291232 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:45.290885 4562 scope.go:122] "RemoveContainer" containerID="5c60c0e665f6f406eff6c8026dc02a810f4fdcceb784688edafa9d5a7e6c14b6" Oct 30 09:24:45.292600 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:45.292559 4562 generic.go:356] "Generic (PLEG): container finished" podID="5bc980b8-abbe-4e78-bf92-cd6654479cbe" containerID="414f71432cf30512f53cd0b55bc7e3c0167c846de4fca35d86d68ad57dd93a6c" exitCode=0 Oct 30 09:24:45.292717 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:45.292597 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-58fadf-5968d7d495-gprm9" Oct 30 09:24:45.292717 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:45.292614 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-58fadf-5968d7d495-gprm9" event={"ID":"5bc980b8-abbe-4e78-bf92-cd6654479cbe","Type":"ContainerDied","Data":"414f71432cf30512f53cd0b55bc7e3c0167c846de4fca35d86d68ad57dd93a6c"} Oct 30 09:24:45.292717 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:45.292651 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-58fadf-5968d7d495-gprm9" event={"ID":"5bc980b8-abbe-4e78-bf92-cd6654479cbe","Type":"ContainerDied","Data":"574dc4d247bbad456d68fbe6c0a5e2a22a8652ef7c2ed3ff4696796623903089"} Oct 30 09:24:45.303291 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:45.303274 4562 scope.go:122] "RemoveContainer" containerID="414f71432cf30512f53cd0b55bc7e3c0167c846de4fca35d86d68ad57dd93a6c" Oct 30 09:24:45.315300 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:45.315267 4562 scope.go:122] "RemoveContainer" containerID="414f71432cf30512f53cd0b55bc7e3c0167c846de4fca35d86d68ad57dd93a6c" Oct 30 09:24:45.315652 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:24:45.315632 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"414f71432cf30512f53cd0b55bc7e3c0167c846de4fca35d86d68ad57dd93a6c\": container with ID starting with 414f71432cf30512f53cd0b55bc7e3c0167c846de4fca35d86d68ad57dd93a6c not found: ID does not exist" containerID="414f71432cf30512f53cd0b55bc7e3c0167c846de4fca35d86d68ad57dd93a6c" Oct 30 09:24:45.315722 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:45.315657 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"414f71432cf30512f53cd0b55bc7e3c0167c846de4fca35d86d68ad57dd93a6c"} err="failed to get container status \"414f71432cf30512f53cd0b55bc7e3c0167c846de4fca35d86d68ad57dd93a6c\": rpc error: code = NotFound desc = could not find container \"414f71432cf30512f53cd0b55bc7e3c0167c846de4fca35d86d68ad57dd93a6c\": container with ID starting with 414f71432cf30512f53cd0b55bc7e3c0167c846de4fca35d86d68ad57dd93a6c not found: ID does not exist" Oct 30 09:24:45.345562 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:45.345538 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-58fadf-5968d7d495-gprm9"] Oct 30 09:24:45.349079 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:45.349059 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-58fadf-5968d7d495-gprm9"] Oct 30 09:24:45.392442 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:45.392350 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-58fadf-7748bcb6d6-9pwn4"] Oct 30 09:24:45.398516 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:45.398492 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-58fadf-7748bcb6d6-9pwn4"] Oct 30 09:24:45.641404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:45.640947 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-356553-54cf695556-7vstx"] Oct 30 09:24:45.651618 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:45.647878 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-356553-658d75b4cf-bvlnt"] Oct 30 09:24:45.936453 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:45.934441 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fd05ff0-d658-452d-8b4a-399aca8f7876" path="/var/lib/kubelet/pods/2fd05ff0-d658-452d-8b4a-399aca8f7876/volumes" Oct 30 09:24:45.936453 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:45.935290 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bc980b8-abbe-4e78-bf92-cd6654479cbe" path="/var/lib/kubelet/pods/5bc980b8-abbe-4e78-bf92-cd6654479cbe/volumes" Oct 30 09:24:49.240555 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:49.240523 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbe974-86947478d7-f4rmw" Oct 30 09:24:49.241055 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:49.240639 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbe974-6d74bdc9f6-w4szp" Oct 30 09:24:50.642108 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:50.642051 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-356553-54cf695556-7vstx" podUID="7a3c155f-73dc-48e5-b0fc-4d45d49670fa" containerName="payload-processing" containerID="cri-o://1df14b6a135fd7f6bd4cfcd37afcb61a67f421fe4d0c2f50cdfe91ca8695cf97" gracePeriod=25 Oct 30 09:24:50.648603 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:50.648574 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-356553-658d75b4cf-bvlnt" podUID="a916cf3a-673c-4244-8d65-1ab14f6328bd" containerName="payload-pre-processing" containerID="cri-o://08df7bf40cdae14f75506ffcf08239baa22f2207b9590cd902d0cf0fd5733513" gracePeriod=25 Oct 30 09:24:50.961288 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:50.961263 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-356553-54cf695556-7vstx" Oct 30 09:24:50.995263 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:50.995238 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-356553-658d75b4cf-bvlnt" Oct 30 09:24:51.017936 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:51.017903 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a3c155f-73dc-48e5-b0fc-4d45d49670fa-kube-api-access-kktwd\" (UniqueName: \"kubernetes.io/projected/7a3c155f-73dc-48e5-b0fc-4d45d49670fa-kube-api-access-kktwd\") pod \"7a3c155f-73dc-48e5-b0fc-4d45d49670fa\" (UID: \"7a3c155f-73dc-48e5-b0fc-4d45d49670fa\") " Oct 30 09:24:51.018150 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:51.018127 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a3c155f-73dc-48e5-b0fc-4d45d49670fa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a3c155f-73dc-48e5-b0fc-4d45d49670fa-plugins-config-volume\") pod \"7a3c155f-73dc-48e5-b0fc-4d45d49670fa\" (UID: \"7a3c155f-73dc-48e5-b0fc-4d45d49670fa\") " Oct 30 09:24:51.019335 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:51.018931 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a3c155f-73dc-48e5-b0fc-4d45d49670fa-plugins-config-volume" pod "7a3c155f-73dc-48e5-b0fc-4d45d49670fa" (UID: "7a3c155f-73dc-48e5-b0fc-4d45d49670fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:24:51.025326 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:51.025298 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a3c155f-73dc-48e5-b0fc-4d45d49670fa-kube-api-access-kktwd" pod "7a3c155f-73dc-48e5-b0fc-4d45d49670fa" (UID: "7a3c155f-73dc-48e5-b0fc-4d45d49670fa"). InnerVolumeSpecName "kube-api-access-kktwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:24:51.119799 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:51.119759 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a916cf3a-673c-4244-8d65-1ab14f6328bd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a916cf3a-673c-4244-8d65-1ab14f6328bd-plugins-config-volume\") pod \"a916cf3a-673c-4244-8d65-1ab14f6328bd\" (UID: \"a916cf3a-673c-4244-8d65-1ab14f6328bd\") " Oct 30 09:24:51.119989 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:51.119977 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a916cf3a-673c-4244-8d65-1ab14f6328bd-kube-api-access-n7lgd\" (UniqueName: \"kubernetes.io/projected/a916cf3a-673c-4244-8d65-1ab14f6328bd-kube-api-access-n7lgd\") pod \"a916cf3a-673c-4244-8d65-1ab14f6328bd\" (UID: \"a916cf3a-673c-4244-8d65-1ab14f6328bd\") " Oct 30 09:24:51.120119 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:51.120095 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a916cf3a-673c-4244-8d65-1ab14f6328bd-plugins-config-volume" pod "a916cf3a-673c-4244-8d65-1ab14f6328bd" (UID: "a916cf3a-673c-4244-8d65-1ab14f6328bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:24:51.120451 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:51.120433 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kktwd\" (UniqueName: \"kubernetes.io/projected/7a3c155f-73dc-48e5-b0fc-4d45d49670fa-kube-api-access-kktwd\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:24:51.120451 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:51.120450 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a916cf3a-673c-4244-8d65-1ab14f6328bd-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:24:51.120625 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:51.120459 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a3c155f-73dc-48e5-b0fc-4d45d49670fa-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:24:51.122045 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:51.122024 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a916cf3a-673c-4244-8d65-1ab14f6328bd-kube-api-access-n7lgd" pod "a916cf3a-673c-4244-8d65-1ab14f6328bd" (UID: "a916cf3a-673c-4244-8d65-1ab14f6328bd"). InnerVolumeSpecName "kube-api-access-n7lgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:24:51.222133 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:51.222098 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7lgd\" (UniqueName: \"kubernetes.io/projected/a916cf3a-673c-4244-8d65-1ab14f6328bd-kube-api-access-n7lgd\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:24:51.339699 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:51.339663 4562 generic.go:356] "Generic (PLEG): container finished" podID="a916cf3a-673c-4244-8d65-1ab14f6328bd" containerID="08df7bf40cdae14f75506ffcf08239baa22f2207b9590cd902d0cf0fd5733513" exitCode=0 Oct 30 09:24:51.339864 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:51.339716 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-356553-658d75b4cf-bvlnt" Oct 30 09:24:51.339864 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:51.339745 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-356553-658d75b4cf-bvlnt" event={"ID":"a916cf3a-673c-4244-8d65-1ab14f6328bd","Type":"ContainerDied","Data":"08df7bf40cdae14f75506ffcf08239baa22f2207b9590cd902d0cf0fd5733513"} Oct 30 09:24:51.339864 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:51.339802 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-356553-658d75b4cf-bvlnt" event={"ID":"a916cf3a-673c-4244-8d65-1ab14f6328bd","Type":"ContainerDied","Data":"3f9b7dfc2901cee66fc699818561dd3a8bc72ec7689a5e6a38a8b886997ed08d"} Oct 30 09:24:51.339864 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:51.339820 4562 scope.go:122] "RemoveContainer" containerID="08df7bf40cdae14f75506ffcf08239baa22f2207b9590cd902d0cf0fd5733513" Oct 30 09:24:51.341294 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:51.341267 4562 generic.go:356] "Generic (PLEG): container finished" podID="7a3c155f-73dc-48e5-b0fc-4d45d49670fa" containerID="1df14b6a135fd7f6bd4cfcd37afcb61a67f421fe4d0c2f50cdfe91ca8695cf97" exitCode=0 Oct 30 09:24:51.341426 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:51.341300 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-356553-54cf695556-7vstx" event={"ID":"7a3c155f-73dc-48e5-b0fc-4d45d49670fa","Type":"ContainerDied","Data":"1df14b6a135fd7f6bd4cfcd37afcb61a67f421fe4d0c2f50cdfe91ca8695cf97"} Oct 30 09:24:51.341426 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:51.341318 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-356553-54cf695556-7vstx" Oct 30 09:24:51.341426 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:51.341344 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-356553-54cf695556-7vstx" event={"ID":"7a3c155f-73dc-48e5-b0fc-4d45d49670fa","Type":"ContainerDied","Data":"4e5ab667f713559d578767ec5e108d30533214c203e2f7a23e6fa16f4d1a21fa"} Oct 30 09:24:51.350697 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:51.350678 4562 scope.go:122] "RemoveContainer" containerID="08df7bf40cdae14f75506ffcf08239baa22f2207b9590cd902d0cf0fd5733513" Oct 30 09:24:51.351016 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:24:51.350953 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08df7bf40cdae14f75506ffcf08239baa22f2207b9590cd902d0cf0fd5733513\": container with ID starting with 08df7bf40cdae14f75506ffcf08239baa22f2207b9590cd902d0cf0fd5733513 not found: ID does not exist" containerID="08df7bf40cdae14f75506ffcf08239baa22f2207b9590cd902d0cf0fd5733513" Oct 30 09:24:51.351016 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:51.350980 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08df7bf40cdae14f75506ffcf08239baa22f2207b9590cd902d0cf0fd5733513"} err="failed to get container status \"08df7bf40cdae14f75506ffcf08239baa22f2207b9590cd902d0cf0fd5733513\": rpc error: code = NotFound desc = could not find container \"08df7bf40cdae14f75506ffcf08239baa22f2207b9590cd902d0cf0fd5733513\": container with ID starting with 08df7bf40cdae14f75506ffcf08239baa22f2207b9590cd902d0cf0fd5733513 not found: ID does not exist" Oct 30 09:24:51.351016 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:51.351007 4562 scope.go:122] "RemoveContainer" containerID="1df14b6a135fd7f6bd4cfcd37afcb61a67f421fe4d0c2f50cdfe91ca8695cf97" Oct 30 09:24:51.360977 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:51.360961 4562 scope.go:122] "RemoveContainer" containerID="1df14b6a135fd7f6bd4cfcd37afcb61a67f421fe4d0c2f50cdfe91ca8695cf97" Oct 30 09:24:51.361223 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:24:51.361206 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1df14b6a135fd7f6bd4cfcd37afcb61a67f421fe4d0c2f50cdfe91ca8695cf97\": container with ID starting with 1df14b6a135fd7f6bd4cfcd37afcb61a67f421fe4d0c2f50cdfe91ca8695cf97 not found: ID does not exist" containerID="1df14b6a135fd7f6bd4cfcd37afcb61a67f421fe4d0c2f50cdfe91ca8695cf97" Oct 30 09:24:51.361264 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:51.361229 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1df14b6a135fd7f6bd4cfcd37afcb61a67f421fe4d0c2f50cdfe91ca8695cf97"} err="failed to get container status \"1df14b6a135fd7f6bd4cfcd37afcb61a67f421fe4d0c2f50cdfe91ca8695cf97\": rpc error: code = NotFound desc = could not find container \"1df14b6a135fd7f6bd4cfcd37afcb61a67f421fe4d0c2f50cdfe91ca8695cf97\": container with ID starting with 1df14b6a135fd7f6bd4cfcd37afcb61a67f421fe4d0c2f50cdfe91ca8695cf97 not found: ID does not exist" Oct 30 09:24:51.399200 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:51.399178 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-356553-54cf695556-7vstx"] Oct 30 09:24:51.404256 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:51.404237 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-356553-54cf695556-7vstx"] Oct 30 09:24:51.439317 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:51.439291 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-356553-658d75b4cf-bvlnt"] Oct 30 09:24:51.443014 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:51.442991 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-356553-658d75b4cf-bvlnt"] Oct 30 09:24:51.875459 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:51.875422 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a3c155f-73dc-48e5-b0fc-4d45d49670fa" path="/var/lib/kubelet/pods/7a3c155f-73dc-48e5-b0fc-4d45d49670fa/volumes" Oct 30 09:24:51.875912 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:51.875897 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a916cf3a-673c-4244-8d65-1ab14f6328bd" path="/var/lib/kubelet/pods/a916cf3a-673c-4244-8d65-1ab14f6328bd/volumes" Oct 30 09:24:53.143406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:53.135750 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-962e2e-5788599b6c-96x72"] Oct 30 09:24:53.143406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:53.140905 4562 cpu_manager.go:414] "containerMap: removing container" podUID="7a3c155f-73dc-48e5-b0fc-4d45d49670fa" containerName="payload-processing" Oct 30 09:24:53.143406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:53.140928 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a3c155f-73dc-48e5-b0fc-4d45d49670fa" containerName="payload-processing" Oct 30 09:24:53.143406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:53.140960 4562 cpu_manager.go:414] "containerMap: removing container" podUID="5bc980b8-abbe-4e78-bf92-cd6654479cbe" containerName="payload-pre-processing" Oct 30 09:24:53.143406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:53.140969 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bc980b8-abbe-4e78-bf92-cd6654479cbe" containerName="payload-pre-processing" Oct 30 09:24:53.143406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:53.140991 4562 cpu_manager.go:414] "containerMap: removing container" podUID="2fd05ff0-d658-452d-8b4a-399aca8f7876" containerName="payload-processing" Oct 30 09:24:53.143406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:53.140998 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fd05ff0-d658-452d-8b4a-399aca8f7876" containerName="payload-processing" Oct 30 09:24:53.143406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:53.141043 4562 cpu_manager.go:414] "containerMap: removing container" podUID="a916cf3a-673c-4244-8d65-1ab14f6328bd" containerName="payload-pre-processing" Oct 30 09:24:53.143406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:53.141050 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a916cf3a-673c-4244-8d65-1ab14f6328bd" containerName="payload-pre-processing" Oct 30 09:24:53.143406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:53.141237 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="a916cf3a-673c-4244-8d65-1ab14f6328bd" containerName="payload-pre-processing" Oct 30 09:24:53.143406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:53.141256 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a3c155f-73dc-48e5-b0fc-4d45d49670fa" containerName="payload-processing" Oct 30 09:24:53.143406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:53.141271 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bc980b8-abbe-4e78-bf92-cd6654479cbe" containerName="payload-pre-processing" Oct 30 09:24:53.143406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:53.141283 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fd05ff0-d658-452d-8b4a-399aca8f7876" containerName="payload-processing" Oct 30 09:24:53.179152 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:53.173572 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-962e2e-5788599b6c-96x72"] Oct 30 09:24:53.179152 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:53.173693 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-962e2e-5788599b6c-96x72" Oct 30 09:24:53.276905 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:53.236584 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-962e2e-dockercfg-h8qdg\"" Oct 30 09:24:53.276905 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:53.236795 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-962e2e\"" Oct 30 09:24:53.328821 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:53.319872 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/120fcc9d-1535-497e-94ca-827940c4323c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-962e2e-5788599b6c-96x72\" (UID: \"120fcc9d-1535-497e-94ca-827940c4323c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-962e2e-5788599b6c-96x72" Oct 30 09:24:53.329456 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:53.329418 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmbvl\" (UniqueName: \"kubernetes.io/projected/120fcc9d-1535-497e-94ca-827940c4323c-kube-api-access-hmbvl\") pod \"payload-pre-processing-e2e-ipp-b-962e2e-5788599b6c-96x72\" (UID: \"120fcc9d-1535-497e-94ca-827940c4323c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-962e2e-5788599b6c-96x72" Oct 30 09:24:53.378710 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:53.374904 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-962e2e-7f64d7567d-h86tm"] Oct 30 09:24:53.385132 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:53.384537 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-962e2e-7f64d7567d-h86tm" Oct 30 09:24:53.389311 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:53.389280 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-962e2e-7f64d7567d-h86tm"] Oct 30 09:24:53.431267 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:53.431211 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0cefc21-e1d4-45ed-b81a-68365ab8e716-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-962e2e-7f64d7567d-h86tm\" (UID: \"b0cefc21-e1d4-45ed-b81a-68365ab8e716\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-962e2e-7f64d7567d-h86tm" Oct 30 09:24:53.431490 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:53.431358 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/120fcc9d-1535-497e-94ca-827940c4323c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-962e2e-5788599b6c-96x72\" (UID: \"120fcc9d-1535-497e-94ca-827940c4323c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-962e2e-5788599b6c-96x72" Oct 30 09:24:53.431490 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:53.431447 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmbvl\" (UniqueName: \"kubernetes.io/projected/120fcc9d-1535-497e-94ca-827940c4323c-kube-api-access-hmbvl\") pod \"payload-pre-processing-e2e-ipp-b-962e2e-5788599b6c-96x72\" (UID: \"120fcc9d-1535-497e-94ca-827940c4323c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-962e2e-5788599b6c-96x72" Oct 30 09:24:53.431681 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:53.431657 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45648\" (UniqueName: \"kubernetes.io/projected/b0cefc21-e1d4-45ed-b81a-68365ab8e716-kube-api-access-45648\") pod \"payload-processing-e2e-ipp-b-962e2e-7f64d7567d-h86tm\" (UID: \"b0cefc21-e1d4-45ed-b81a-68365ab8e716\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-962e2e-7f64d7567d-h86tm" Oct 30 09:24:53.432801 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:53.432774 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/120fcc9d-1535-497e-94ca-827940c4323c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-962e2e-5788599b6c-96x72\" (UID: \"120fcc9d-1535-497e-94ca-827940c4323c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-962e2e-5788599b6c-96x72" Oct 30 09:24:53.456695 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:53.456655 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmbvl\" (UniqueName: \"kubernetes.io/projected/120fcc9d-1535-497e-94ca-827940c4323c-kube-api-access-hmbvl\") pod \"payload-pre-processing-e2e-ipp-b-962e2e-5788599b6c-96x72\" (UID: \"120fcc9d-1535-497e-94ca-827940c4323c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-962e2e-5788599b6c-96x72" Oct 30 09:24:53.533145 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:53.533100 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0cefc21-e1d4-45ed-b81a-68365ab8e716-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-962e2e-7f64d7567d-h86tm\" (UID: \"b0cefc21-e1d4-45ed-b81a-68365ab8e716\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-962e2e-7f64d7567d-h86tm" Oct 30 09:24:53.533356 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:53.533336 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45648\" (UniqueName: \"kubernetes.io/projected/b0cefc21-e1d4-45ed-b81a-68365ab8e716-kube-api-access-45648\") pod \"payload-processing-e2e-ipp-b-962e2e-7f64d7567d-h86tm\" (UID: \"b0cefc21-e1d4-45ed-b81a-68365ab8e716\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-962e2e-7f64d7567d-h86tm" Oct 30 09:24:53.533925 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:53.533904 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0cefc21-e1d4-45ed-b81a-68365ab8e716-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-962e2e-7f64d7567d-h86tm\" (UID: \"b0cefc21-e1d4-45ed-b81a-68365ab8e716\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-962e2e-7f64d7567d-h86tm" Oct 30 09:24:53.553178 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:53.553143 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45648\" (UniqueName: \"kubernetes.io/projected/b0cefc21-e1d4-45ed-b81a-68365ab8e716-kube-api-access-45648\") pod \"payload-processing-e2e-ipp-b-962e2e-7f64d7567d-h86tm\" (UID: \"b0cefc21-e1d4-45ed-b81a-68365ab8e716\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-962e2e-7f64d7567d-h86tm" Oct 30 09:24:53.689466 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:53.689434 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-962e2e-5788599b6c-96x72" Oct 30 09:24:53.701822 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:53.701797 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-962e2e-7f64d7567d-h86tm" Oct 30 09:24:54.085642 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:54.081534 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-962e2e-5788599b6c-96x72"] Oct 30 09:24:54.138497 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:54.119821 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-962e2e-7f64d7567d-h86tm"] Oct 30 09:24:54.242034 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:24:54.241963 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0cefc21_e1d4_45ed_b81a_68365ab8e716.slice/crio-e9a621a1086a420c770802616cdff5decfa0286d8e40dfe586b7610d43e9cad8 WatchSource:0}: Error finding container e9a621a1086a420c770802616cdff5decfa0286d8e40dfe586b7610d43e9cad8: Status 404 returned error can't find the container with id e9a621a1086a420c770802616cdff5decfa0286d8e40dfe586b7610d43e9cad8 Oct 30 09:24:54.397437 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:54.397350 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-962e2e-7f64d7567d-h86tm" event={"ID":"b0cefc21-e1d4-45ed-b81a-68365ab8e716","Type":"ContainerStarted","Data":"e9a621a1086a420c770802616cdff5decfa0286d8e40dfe586b7610d43e9cad8"} Oct 30 09:24:54.399292 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:54.399252 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-962e2e-5788599b6c-96x72" event={"ID":"120fcc9d-1535-497e-94ca-827940c4323c","Type":"ContainerStarted","Data":"e5eac35dda2642236730ea4d33bbebbc76ac517a2fabdfff5e500257d5673cdc"} Oct 30 09:24:55.407453 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:55.407418 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-962e2e-7f64d7567d-h86tm" event={"ID":"b0cefc21-e1d4-45ed-b81a-68365ab8e716","Type":"ContainerStarted","Data":"8deebfb29cae9ce612467c3c05a08fb1c37f5ed299a0b7397189ea809af39b89"} Oct 30 09:24:55.407914 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:55.407485 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-962e2e-7f64d7567d-h86tm" Oct 30 09:24:55.409078 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:55.409051 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-962e2e-5788599b6c-96x72" event={"ID":"120fcc9d-1535-497e-94ca-827940c4323c","Type":"ContainerStarted","Data":"c439c983adc54ba174413e11a59734e3dac3cc90e6430ddcfbda1e848512e463"} Oct 30 09:24:55.409210 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:55.409162 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-962e2e-5788599b6c-96x72" Oct 30 09:24:55.451101 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:55.451052 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-962e2e-7f64d7567d-h86tm" podStartSLOduration=2.451038386 podStartE2EDuration="2.451038386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:24:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:24:55.447583239 +0000 UTC m=+1855.709144231" watchObservedRunningTime="2026-10-30 09:24:55.451038386 +0000 UTC m=+1855.712599375" Oct 30 09:24:55.489673 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:24:55.489616 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-962e2e-5788599b6c-96x72" podStartSLOduration=2.489598176 podStartE2EDuration="2.489598176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:24:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:24:55.487416968 +0000 UTC m=+1855.748977952" watchObservedRunningTime="2026-10-30 09:24:55.489598176 +0000 UTC m=+1855.751159183" Oct 30 09:24:58.597804 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:24:58.597767 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:25:06.419182 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:06.419150 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-962e2e-5788599b6c-96x72" Oct 30 09:25:06.419689 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:06.419228 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-962e2e-7f64d7567d-h86tm" Oct 30 09:25:25.905977 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:25.905939 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8de308-64c66d9c69-7p6mx"] Oct 30 09:25:25.906500 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:25.906196 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8de308-64c66d9c69-7p6mx" podUID="f0b5325c-a526-4cbe-9d70-2dd858185eb5" containerName="maas-api" containerID="cri-o://16e065b53259ca85c8c40d128148896e4c17c534b0f2c1909426d6e51e3342de" gracePeriod=30 Oct 30 09:25:26.053112 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:26.053072 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8de308-7b79454f5d-w8kbm"] Oct 30 09:25:26.053345 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:26.053307 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-8de308-7b79454f5d-w8kbm" podUID="dacc132b-fcda-49a6-a569-4696ea206638" containerName="payload-processing" containerID="cri-o://65134e9f0a92a5edee1ccf07bc5d5397f1d8fc12b6cd48efffadc3db52ad2805" gracePeriod=30 Oct 30 09:25:26.296933 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:26.296906 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8de308-64c66d9c69-7p6mx" Oct 30 09:25:26.517265 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:26.517232 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0b5325c-a526-4cbe-9d70-2dd858185eb5-kube-api-access-d29jd\" (UniqueName: \"kubernetes.io/projected/f0b5325c-a526-4cbe-9d70-2dd858185eb5-kube-api-access-d29jd\") pod \"f0b5325c-a526-4cbe-9d70-2dd858185eb5\" (UID: \"f0b5325c-a526-4cbe-9d70-2dd858185eb5\") " Oct 30 09:25:26.517653 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:26.517631 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0b5325c-a526-4cbe-9d70-2dd858185eb5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0b5325c-a526-4cbe-9d70-2dd858185eb5-maas-api-tls\") pod \"f0b5325c-a526-4cbe-9d70-2dd858185eb5\" (UID: \"f0b5325c-a526-4cbe-9d70-2dd858185eb5\") " Oct 30 09:25:26.526520 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:26.520251 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0b5325c-a526-4cbe-9d70-2dd858185eb5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0b5325c-a526-4cbe-9d70-2dd858185eb5-metrics-certs\") pod \"f0b5325c-a526-4cbe-9d70-2dd858185eb5\" (UID: \"f0b5325c-a526-4cbe-9d70-2dd858185eb5\") " Oct 30 09:25:26.545942 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:26.526827 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0b5325c-a526-4cbe-9d70-2dd858185eb5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0b5325c-a526-4cbe-9d70-2dd858185eb5-openshift-service-ca\") pod \"f0b5325c-a526-4cbe-9d70-2dd858185eb5\" (UID: \"f0b5325c-a526-4cbe-9d70-2dd858185eb5\") " Oct 30 09:25:26.591406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:26.538067 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0b5325c-a526-4cbe-9d70-2dd858185eb5-openshift-service-ca" pod "f0b5325c-a526-4cbe-9d70-2dd858185eb5" (UID: "f0b5325c-a526-4cbe-9d70-2dd858185eb5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:25:26.663434 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:26.651840 4562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0b5325c-a526-4cbe-9d70-2dd858185eb5-openshift-service-ca\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:25:26.663434 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:26.636072 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0b5325c-a526-4cbe-9d70-2dd858185eb5-kube-api-access-d29jd" pod "f0b5325c-a526-4cbe-9d70-2dd858185eb5" (UID: "f0b5325c-a526-4cbe-9d70-2dd858185eb5"). InnerVolumeSpecName "kube-api-access-d29jd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:25:26.669469 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:26.669440 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8de308-7b79454f5d-w8kbm" Oct 30 09:25:26.690149 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:26.690094 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0b5325c-a526-4cbe-9d70-2dd858185eb5-maas-api-tls" pod "f0b5325c-a526-4cbe-9d70-2dd858185eb5" (UID: "f0b5325c-a526-4cbe-9d70-2dd858185eb5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:25:26.690833 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:26.690777 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0b5325c-a526-4cbe-9d70-2dd858185eb5-metrics-certs" pod "f0b5325c-a526-4cbe-9d70-2dd858185eb5" (UID: "f0b5325c-a526-4cbe-9d70-2dd858185eb5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:25:26.753550 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:26.753469 4562 generic.go:356] "Generic (PLEG): container finished" podID="dacc132b-fcda-49a6-a569-4696ea206638" containerID="65134e9f0a92a5edee1ccf07bc5d5397f1d8fc12b6cd48efffadc3db52ad2805" exitCode=0 Oct 30 09:25:26.753550 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:26.753532 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8de308-7b79454f5d-w8kbm" Oct 30 09:25:26.753769 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:26.753560 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8de308-7b79454f5d-w8kbm" event={"ID":"dacc132b-fcda-49a6-a569-4696ea206638","Type":"ContainerDied","Data":"65134e9f0a92a5edee1ccf07bc5d5397f1d8fc12b6cd48efffadc3db52ad2805"} Oct 30 09:25:26.753769 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:26.753625 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8de308-7b79454f5d-w8kbm" event={"ID":"dacc132b-fcda-49a6-a569-4696ea206638","Type":"ContainerDied","Data":"0d81575f9ef37e05968636a6a9f4c76972345f1c9a47bb411a7b2e69e169fd93"} Oct 30 09:25:26.753769 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:26.753660 4562 scope.go:122] "RemoveContainer" containerID="65134e9f0a92a5edee1ccf07bc5d5397f1d8fc12b6cd48efffadc3db52ad2805" Oct 30 09:25:26.755336 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:26.755315 4562 generic.go:356] "Generic (PLEG): container finished" podID="f0b5325c-a526-4cbe-9d70-2dd858185eb5" containerID="16e065b53259ca85c8c40d128148896e4c17c534b0f2c1909426d6e51e3342de" exitCode=0 Oct 30 09:25:26.755556 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:26.755460 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8de308-64c66d9c69-7p6mx" event={"ID":"f0b5325c-a526-4cbe-9d70-2dd858185eb5","Type":"ContainerDied","Data":"16e065b53259ca85c8c40d128148896e4c17c534b0f2c1909426d6e51e3342de"} Oct 30 09:25:26.755556 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:26.755503 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8de308-64c66d9c69-7p6mx" event={"ID":"f0b5325c-a526-4cbe-9d70-2dd858185eb5","Type":"ContainerDied","Data":"1f5dec07e176805f4d3bfb60263075f46d2f76638f6b03299ab4bfbcdd03f414"} Oct 30 09:25:26.755707 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:26.755573 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8de308-64c66d9c69-7p6mx" Oct 30 09:25:26.761860 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:26.761834 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dacc132b-fcda-49a6-a569-4696ea206638-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dacc132b-fcda-49a6-a569-4696ea206638-plugins-config-volume\") pod \"dacc132b-fcda-49a6-a569-4696ea206638\" (UID: \"dacc132b-fcda-49a6-a569-4696ea206638\") " Oct 30 09:25:26.761995 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:26.761911 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dacc132b-fcda-49a6-a569-4696ea206638-kube-api-access-j8nt5\" (UniqueName: \"kubernetes.io/projected/dacc132b-fcda-49a6-a569-4696ea206638-kube-api-access-j8nt5\") pod \"dacc132b-fcda-49a6-a569-4696ea206638\" (UID: \"dacc132b-fcda-49a6-a569-4696ea206638\") " Oct 30 09:25:26.762270 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:26.762242 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dacc132b-fcda-49a6-a569-4696ea206638-plugins-config-volume" pod "dacc132b-fcda-49a6-a569-4696ea206638" (UID: "dacc132b-fcda-49a6-a569-4696ea206638"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:25:26.762590 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:26.762572 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dacc132b-fcda-49a6-a569-4696ea206638-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:25:26.762590 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:26.762590 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d29jd\" (UniqueName: \"kubernetes.io/projected/f0b5325c-a526-4cbe-9d70-2dd858185eb5-kube-api-access-d29jd\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:25:26.762733 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:26.762600 4562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0b5325c-a526-4cbe-9d70-2dd858185eb5-maas-api-tls\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:25:26.762733 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:26.762609 4562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0b5325c-a526-4cbe-9d70-2dd858185eb5-metrics-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:25:26.764283 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:26.764253 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dacc132b-fcda-49a6-a569-4696ea206638-kube-api-access-j8nt5" pod "dacc132b-fcda-49a6-a569-4696ea206638" (UID: "dacc132b-fcda-49a6-a569-4696ea206638"). InnerVolumeSpecName "kube-api-access-j8nt5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:25:26.765184 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:26.765163 4562 scope.go:122] "RemoveContainer" containerID="65134e9f0a92a5edee1ccf07bc5d5397f1d8fc12b6cd48efffadc3db52ad2805" Oct 30 09:25:26.765517 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:25:26.765496 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65134e9f0a92a5edee1ccf07bc5d5397f1d8fc12b6cd48efffadc3db52ad2805\": container with ID starting with 65134e9f0a92a5edee1ccf07bc5d5397f1d8fc12b6cd48efffadc3db52ad2805 not found: ID does not exist" containerID="65134e9f0a92a5edee1ccf07bc5d5397f1d8fc12b6cd48efffadc3db52ad2805" Oct 30 09:25:26.765615 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:26.765529 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65134e9f0a92a5edee1ccf07bc5d5397f1d8fc12b6cd48efffadc3db52ad2805"} err="failed to get container status \"65134e9f0a92a5edee1ccf07bc5d5397f1d8fc12b6cd48efffadc3db52ad2805\": rpc error: code = NotFound desc = could not find container \"65134e9f0a92a5edee1ccf07bc5d5397f1d8fc12b6cd48efffadc3db52ad2805\": container with ID starting with 65134e9f0a92a5edee1ccf07bc5d5397f1d8fc12b6cd48efffadc3db52ad2805 not found: ID does not exist" Oct 30 09:25:26.765615 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:26.765551 4562 scope.go:122] "RemoveContainer" containerID="16e065b53259ca85c8c40d128148896e4c17c534b0f2c1909426d6e51e3342de" Oct 30 09:25:26.775054 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:26.775038 4562 scope.go:122] "RemoveContainer" containerID="16e065b53259ca85c8c40d128148896e4c17c534b0f2c1909426d6e51e3342de" Oct 30 09:25:26.775291 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:25:26.775275 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16e065b53259ca85c8c40d128148896e4c17c534b0f2c1909426d6e51e3342de\": container with ID starting with 16e065b53259ca85c8c40d128148896e4c17c534b0f2c1909426d6e51e3342de not found: ID does not exist" containerID="16e065b53259ca85c8c40d128148896e4c17c534b0f2c1909426d6e51e3342de" Oct 30 09:25:26.775332 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:26.775297 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16e065b53259ca85c8c40d128148896e4c17c534b0f2c1909426d6e51e3342de"} err="failed to get container status \"16e065b53259ca85c8c40d128148896e4c17c534b0f2c1909426d6e51e3342de\": rpc error: code = NotFound desc = could not find container \"16e065b53259ca85c8c40d128148896e4c17c534b0f2c1909426d6e51e3342de\": container with ID starting with 16e065b53259ca85c8c40d128148896e4c17c534b0f2c1909426d6e51e3342de not found: ID does not exist" Oct 30 09:25:26.813693 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:26.813666 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8de308-64c66d9c69-7p6mx"] Oct 30 09:25:26.817279 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:26.817254 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8de308-64c66d9c69-7p6mx"] Oct 30 09:25:26.864004 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:26.863973 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8nt5\" (UniqueName: \"kubernetes.io/projected/dacc132b-fcda-49a6-a569-4696ea206638-kube-api-access-j8nt5\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:25:27.104944 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:27.104916 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8de308-7b79454f5d-w8kbm"] Oct 30 09:25:27.108634 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:27.108609 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8de308-7b79454f5d-w8kbm"] Oct 30 09:25:27.876027 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:27.875995 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dacc132b-fcda-49a6-a569-4696ea206638" path="/var/lib/kubelet/pods/dacc132b-fcda-49a6-a569-4696ea206638/volumes" Oct 30 09:25:27.876590 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:27.876573 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0b5325c-a526-4cbe-9d70-2dd858185eb5" path="/var/lib/kubelet/pods/f0b5325c-a526-4cbe-9d70-2dd858185eb5/volumes" Oct 30 09:25:28.600335 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:25:28.600298 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:25:32.304224 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:32.303980 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-58fadf-667c9dd778-swcxh"] Oct 30 09:25:32.304739 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:32.304236 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58fadf-667c9dd778-swcxh" podUID="c5aa403a-1de7-40d7-93ef-238dac140203" containerName="maas-api" containerID="cri-o://aa138d0fd4c7087e32a534c3562253df92bd692f9a7d445dc04f7a72cb3a184a" gracePeriod=30 Oct 30 09:25:32.800408 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:32.795354 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58fadf-667c9dd778-swcxh" Oct 30 09:25:32.888968 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:32.887749 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5aa403a-1de7-40d7-93ef-238dac140203-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5aa403a-1de7-40d7-93ef-238dac140203-openshift-service-ca\") pod \"c5aa403a-1de7-40d7-93ef-238dac140203\" (UID: \"c5aa403a-1de7-40d7-93ef-238dac140203\") " Oct 30 09:25:32.888968 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:32.887853 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5aa403a-1de7-40d7-93ef-238dac140203-kube-api-access-cp9gl\" (UniqueName: \"kubernetes.io/projected/c5aa403a-1de7-40d7-93ef-238dac140203-kube-api-access-cp9gl\") pod \"c5aa403a-1de7-40d7-93ef-238dac140203\" (UID: \"c5aa403a-1de7-40d7-93ef-238dac140203\") " Oct 30 09:25:32.888968 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:32.888078 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5aa403a-1de7-40d7-93ef-238dac140203-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5aa403a-1de7-40d7-93ef-238dac140203-metrics-certs\") pod \"c5aa403a-1de7-40d7-93ef-238dac140203\" (UID: \"c5aa403a-1de7-40d7-93ef-238dac140203\") " Oct 30 09:25:32.888968 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:32.888125 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5aa403a-1de7-40d7-93ef-238dac140203-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5aa403a-1de7-40d7-93ef-238dac140203-maas-api-tls\") pod \"c5aa403a-1de7-40d7-93ef-238dac140203\" (UID: \"c5aa403a-1de7-40d7-93ef-238dac140203\") " Oct 30 09:25:32.897135 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:32.891963 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5aa403a-1de7-40d7-93ef-238dac140203-kube-api-access-cp9gl" pod "c5aa403a-1de7-40d7-93ef-238dac140203" (UID: "c5aa403a-1de7-40d7-93ef-238dac140203"). InnerVolumeSpecName "kube-api-access-cp9gl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:25:32.897135 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:32.896772 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5aa403a-1de7-40d7-93ef-238dac140203-openshift-service-ca" pod "c5aa403a-1de7-40d7-93ef-238dac140203" (UID: "c5aa403a-1de7-40d7-93ef-238dac140203"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:25:32.919929 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:32.919536 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5aa403a-1de7-40d7-93ef-238dac140203-maas-api-tls" pod "c5aa403a-1de7-40d7-93ef-238dac140203" (UID: "c5aa403a-1de7-40d7-93ef-238dac140203"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:25:32.920462 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:32.920357 4562 generic.go:356] "Generic (PLEG): container finished" podID="c5aa403a-1de7-40d7-93ef-238dac140203" containerID="aa138d0fd4c7087e32a534c3562253df92bd692f9a7d445dc04f7a72cb3a184a" exitCode=0 Oct 30 09:25:32.920961 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:32.920694 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58fadf-667c9dd778-swcxh" event={"ID":"c5aa403a-1de7-40d7-93ef-238dac140203","Type":"ContainerDied","Data":"aa138d0fd4c7087e32a534c3562253df92bd692f9a7d445dc04f7a72cb3a184a"} Oct 30 09:25:32.920961 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:32.920729 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58fadf-667c9dd778-swcxh" Oct 30 09:25:32.920961 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:32.920757 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58fadf-667c9dd778-swcxh" event={"ID":"c5aa403a-1de7-40d7-93ef-238dac140203","Type":"ContainerDied","Data":"9a4e0f181691f9237d842270fb5b58487077abbff76a05dfdbf14438aa7ef3ff"} Oct 30 09:25:32.920961 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:32.920789 4562 scope.go:122] "RemoveContainer" containerID="aa138d0fd4c7087e32a534c3562253df92bd692f9a7d445dc04f7a72cb3a184a" Oct 30 09:25:32.990402 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:32.975548 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5aa403a-1de7-40d7-93ef-238dac140203-metrics-certs" pod "c5aa403a-1de7-40d7-93ef-238dac140203" (UID: "c5aa403a-1de7-40d7-93ef-238dac140203"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:25:33.001020 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:33.000987 4562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5aa403a-1de7-40d7-93ef-238dac140203-openshift-service-ca\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:25:33.001335 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:33.001312 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cp9gl\" (UniqueName: \"kubernetes.io/projected/c5aa403a-1de7-40d7-93ef-238dac140203-kube-api-access-cp9gl\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:25:33.001498 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:33.001482 4562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5aa403a-1de7-40d7-93ef-238dac140203-metrics-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:25:33.001596 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:33.001584 4562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5aa403a-1de7-40d7-93ef-238dac140203-maas-api-tls\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:25:33.111921 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:33.111607 4562 scope.go:122] "RemoveContainer" containerID="aa138d0fd4c7087e32a534c3562253df92bd692f9a7d445dc04f7a72cb3a184a" Oct 30 09:25:33.127227 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:25:33.122827 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa138d0fd4c7087e32a534c3562253df92bd692f9a7d445dc04f7a72cb3a184a\": container with ID starting with aa138d0fd4c7087e32a534c3562253df92bd692f9a7d445dc04f7a72cb3a184a not found: ID does not exist" containerID="aa138d0fd4c7087e32a534c3562253df92bd692f9a7d445dc04f7a72cb3a184a" Oct 30 09:25:33.127227 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:33.122874 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa138d0fd4c7087e32a534c3562253df92bd692f9a7d445dc04f7a72cb3a184a"} err="failed to get container status \"aa138d0fd4c7087e32a534c3562253df92bd692f9a7d445dc04f7a72cb3a184a\": rpc error: code = NotFound desc = could not find container \"aa138d0fd4c7087e32a534c3562253df92bd692f9a7d445dc04f7a72cb3a184a\": container with ID starting with aa138d0fd4c7087e32a534c3562253df92bd692f9a7d445dc04f7a72cb3a184a not found: ID does not exist" Oct 30 09:25:33.270153 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:33.270123 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-58fadf-667c9dd778-swcxh"] Oct 30 09:25:33.273481 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:33.273458 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-58fadf-667c9dd778-swcxh"] Oct 30 09:25:33.877679 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:33.877646 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5aa403a-1de7-40d7-93ef-238dac140203" path="/var/lib/kubelet/pods/c5aa403a-1de7-40d7-93ef-238dac140203/volumes" Oct 30 09:25:34.899050 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:34.899020 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8"] Oct 30 09:25:34.899526 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:34.899238 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8" podUID="7b433eaf-bc2e-4a08-8e7a-527e88ef4c59" containerName="istio-proxy" containerID="cri-o://1e903368f7a3186b97d6e289bd97e71795cb7f690d1565926e5368ca4ef759e7" gracePeriod=30 Oct 30 09:25:40.224317 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:40.224292 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8" Oct 30 09:25:40.407201 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:40.369869 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7b433eaf-bc2e-4a08-8e7a-527e88ef4c59-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b433eaf-bc2e-4a08-8e7a-527e88ef4c59-istio-podinfo\") pod \"7b433eaf-bc2e-4a08-8e7a-527e88ef4c59\" (UID: \"7b433eaf-bc2e-4a08-8e7a-527e88ef4c59\") " Oct 30 09:25:40.407686 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:40.407655 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b433eaf-bc2e-4a08-8e7a-527e88ef4c59-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b433eaf-bc2e-4a08-8e7a-527e88ef4c59-credential-socket\") pod \"7b433eaf-bc2e-4a08-8e7a-527e88ef4c59\" (UID: \"7b433eaf-bc2e-4a08-8e7a-527e88ef4c59\") " Oct 30 09:25:40.408401 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:40.408359 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b433eaf-bc2e-4a08-8e7a-527e88ef4c59-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b433eaf-bc2e-4a08-8e7a-527e88ef4c59-istio-envoy\") pod \"7b433eaf-bc2e-4a08-8e7a-527e88ef4c59\" (UID: \"7b433eaf-bc2e-4a08-8e7a-527e88ef4c59\") " Oct 30 09:25:40.408573 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:40.408554 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b433eaf-bc2e-4a08-8e7a-527e88ef4c59-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b433eaf-bc2e-4a08-8e7a-527e88ef4c59-workload-socket\") pod \"7b433eaf-bc2e-4a08-8e7a-527e88ef4c59\" (UID: \"7b433eaf-bc2e-4a08-8e7a-527e88ef4c59\") " Oct 30 09:25:40.408731 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:40.408718 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b433eaf-bc2e-4a08-8e7a-527e88ef4c59-istio-token\" (UniqueName: \"kubernetes.io/projected/7b433eaf-bc2e-4a08-8e7a-527e88ef4c59-istio-token\") pod \"7b433eaf-bc2e-4a08-8e7a-527e88ef4c59\" (UID: \"7b433eaf-bc2e-4a08-8e7a-527e88ef4c59\") " Oct 30 09:25:40.408829 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:40.408816 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b433eaf-bc2e-4a08-8e7a-527e88ef4c59-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b433eaf-bc2e-4a08-8e7a-527e88ef4c59-istio-data\") pod \"7b433eaf-bc2e-4a08-8e7a-527e88ef4c59\" (UID: \"7b433eaf-bc2e-4a08-8e7a-527e88ef4c59\") " Oct 30 09:25:40.409155 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:40.409130 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b433eaf-bc2e-4a08-8e7a-527e88ef4c59-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b433eaf-bc2e-4a08-8e7a-527e88ef4c59-istiod-ca-cert\") pod \"7b433eaf-bc2e-4a08-8e7a-527e88ef4c59\" (UID: \"7b433eaf-bc2e-4a08-8e7a-527e88ef4c59\") " Oct 30 09:25:40.409472 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:40.409455 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b433eaf-bc2e-4a08-8e7a-527e88ef4c59-kube-api-access-ggwg8\" (UniqueName: \"kubernetes.io/projected/7b433eaf-bc2e-4a08-8e7a-527e88ef4c59-kube-api-access-ggwg8\") pod \"7b433eaf-bc2e-4a08-8e7a-527e88ef4c59\" (UID: \"7b433eaf-bc2e-4a08-8e7a-527e88ef4c59\") " Oct 30 09:25:40.409634 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:40.409620 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b433eaf-bc2e-4a08-8e7a-527e88ef4c59-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b433eaf-bc2e-4a08-8e7a-527e88ef4c59-workload-certs\") pod \"7b433eaf-bc2e-4a08-8e7a-527e88ef4c59\" (UID: \"7b433eaf-bc2e-4a08-8e7a-527e88ef4c59\") " Oct 30 09:25:40.412345 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:40.408237 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b433eaf-bc2e-4a08-8e7a-527e88ef4c59-credential-socket" pod "7b433eaf-bc2e-4a08-8e7a-527e88ef4c59" (UID: "7b433eaf-bc2e-4a08-8e7a-527e88ef4c59"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:40.412471 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:40.410743 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b433eaf-bc2e-4a08-8e7a-527e88ef4c59-workload-certs" pod "7b433eaf-bc2e-4a08-8e7a-527e88ef4c59" (UID: "7b433eaf-bc2e-4a08-8e7a-527e88ef4c59"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:40.412471 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:40.410764 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b433eaf-bc2e-4a08-8e7a-527e88ef4c59-istiod-ca-cert" pod "7b433eaf-bc2e-4a08-8e7a-527e88ef4c59" (UID: "7b433eaf-bc2e-4a08-8e7a-527e88ef4c59"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:25:40.412471 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:40.411032 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b433eaf-bc2e-4a08-8e7a-527e88ef4c59-istio-data" pod "7b433eaf-bc2e-4a08-8e7a-527e88ef4c59" (UID: "7b433eaf-bc2e-4a08-8e7a-527e88ef4c59"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:40.412471 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:40.412311 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b433eaf-bc2e-4a08-8e7a-527e88ef4c59-istio-podinfo" pod "7b433eaf-bc2e-4a08-8e7a-527e88ef4c59" (UID: "7b433eaf-bc2e-4a08-8e7a-527e88ef4c59"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:25:40.412696 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:40.412655 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b433eaf-bc2e-4a08-8e7a-527e88ef4c59-workload-socket" pod "7b433eaf-bc2e-4a08-8e7a-527e88ef4c59" (UID: "7b433eaf-bc2e-4a08-8e7a-527e88ef4c59"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:40.437846 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:40.437790 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b433eaf-bc2e-4a08-8e7a-527e88ef4c59-istio-envoy" pod "7b433eaf-bc2e-4a08-8e7a-527e88ef4c59" (UID: "7b433eaf-bc2e-4a08-8e7a-527e88ef4c59"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:40.437846 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:40.437806 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b433eaf-bc2e-4a08-8e7a-527e88ef4c59-kube-api-access-ggwg8" pod "7b433eaf-bc2e-4a08-8e7a-527e88ef4c59" (UID: "7b433eaf-bc2e-4a08-8e7a-527e88ef4c59"). InnerVolumeSpecName "kube-api-access-ggwg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:25:40.439497 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:40.439469 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b433eaf-bc2e-4a08-8e7a-527e88ef4c59-istio-token" pod "7b433eaf-bc2e-4a08-8e7a-527e88ef4c59" (UID: "7b433eaf-bc2e-4a08-8e7a-527e88ef4c59"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:25:40.512832 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:40.512756 4562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b433eaf-bc2e-4a08-8e7a-527e88ef4c59-istio-envoy\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:25:40.512832 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:40.512788 4562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b433eaf-bc2e-4a08-8e7a-527e88ef4c59-workload-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:25:40.512832 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:40.512806 4562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b433eaf-bc2e-4a08-8e7a-527e88ef4c59-istio-token\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:25:40.512832 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:40.512820 4562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b433eaf-bc2e-4a08-8e7a-527e88ef4c59-istio-data\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:25:40.512832 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:40.512833 4562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b433eaf-bc2e-4a08-8e7a-527e88ef4c59-istiod-ca-cert\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:25:40.513094 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:40.512846 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggwg8\" (UniqueName: \"kubernetes.io/projected/7b433eaf-bc2e-4a08-8e7a-527e88ef4c59-kube-api-access-ggwg8\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:25:40.513094 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:40.512859 4562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b433eaf-bc2e-4a08-8e7a-527e88ef4c59-workload-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:25:40.513094 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:40.512872 4562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b433eaf-bc2e-4a08-8e7a-527e88ef4c59-istio-podinfo\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:25:40.513094 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:40.512887 4562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b433eaf-bc2e-4a08-8e7a-527e88ef4c59-credential-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:25:41.047142 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:41.047106 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8f40a3-784d84f5c5-dtwh7"] Oct 30 09:25:41.047652 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:41.047452 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8f40a3-784d84f5c5-dtwh7" podUID="eada6a8f-af88-4279-ae23-1bab2a23be56" containerName="maas-api" containerID="cri-o://7eae696ae7f676e5db6bb1239b6b1616057437bcf4062ebd4417ce548cc14a81" gracePeriod=30 Oct 30 09:25:41.078098 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:41.076984 4562 generic.go:356] "Generic (PLEG): container finished" podID="7b433eaf-bc2e-4a08-8e7a-527e88ef4c59" containerID="1e903368f7a3186b97d6e289bd97e71795cb7f690d1565926e5368ca4ef759e7" exitCode=0 Oct 30 09:25:41.078098 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:41.077116 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8" event={"ID":"7b433eaf-bc2e-4a08-8e7a-527e88ef4c59","Type":"ContainerDied","Data":"1e903368f7a3186b97d6e289bd97e71795cb7f690d1565926e5368ca4ef759e7"} Oct 30 09:25:41.078098 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:41.077161 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8" event={"ID":"7b433eaf-bc2e-4a08-8e7a-527e88ef4c59","Type":"ContainerDied","Data":"99685a22e14fb86a27c52799e40a9da7f7037ebc680de2246e1f0e80c3daab40"} Oct 30 09:25:41.078098 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:41.077225 4562 scope.go:122] "RemoveContainer" containerID="1e903368f7a3186b97d6e289bd97e71795cb7f690d1565926e5368ca4ef759e7" Oct 30 09:25:41.078098 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:41.077449 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8" Oct 30 09:25:41.179330 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:41.179305 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8"] Oct 30 09:25:41.185707 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:41.185677 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-8de308-openshift-default-75b4598799-45qh8"] Oct 30 09:25:41.187759 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:41.187736 4562 scope.go:122] "RemoveContainer" containerID="4a49092088ec33406b51283b5dc3b0064f3bbd844e34fe625074b009ee0ca805" Oct 30 09:25:41.199147 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:41.199126 4562 scope.go:122] "RemoveContainer" containerID="1e903368f7a3186b97d6e289bd97e71795cb7f690d1565926e5368ca4ef759e7" Oct 30 09:25:41.199499 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:25:41.199477 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e903368f7a3186b97d6e289bd97e71795cb7f690d1565926e5368ca4ef759e7\": container with ID starting with 1e903368f7a3186b97d6e289bd97e71795cb7f690d1565926e5368ca4ef759e7 not found: ID does not exist" containerID="1e903368f7a3186b97d6e289bd97e71795cb7f690d1565926e5368ca4ef759e7" Oct 30 09:25:41.199562 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:41.199510 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e903368f7a3186b97d6e289bd97e71795cb7f690d1565926e5368ca4ef759e7"} err="failed to get container status \"1e903368f7a3186b97d6e289bd97e71795cb7f690d1565926e5368ca4ef759e7\": rpc error: code = NotFound desc = could not find container \"1e903368f7a3186b97d6e289bd97e71795cb7f690d1565926e5368ca4ef759e7\": container with ID starting with 1e903368f7a3186b97d6e289bd97e71795cb7f690d1565926e5368ca4ef759e7 not found: ID does not exist" Oct 30 09:25:41.199562 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:41.199527 4562 scope.go:122] "RemoveContainer" containerID="4a49092088ec33406b51283b5dc3b0064f3bbd844e34fe625074b009ee0ca805" Oct 30 09:25:41.199835 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:25:41.199804 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a49092088ec33406b51283b5dc3b0064f3bbd844e34fe625074b009ee0ca805\": container with ID starting with 4a49092088ec33406b51283b5dc3b0064f3bbd844e34fe625074b009ee0ca805 not found: ID does not exist" containerID="4a49092088ec33406b51283b5dc3b0064f3bbd844e34fe625074b009ee0ca805" Oct 30 09:25:41.199904 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:41.199844 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a49092088ec33406b51283b5dc3b0064f3bbd844e34fe625074b009ee0ca805"} err="failed to get container status \"4a49092088ec33406b51283b5dc3b0064f3bbd844e34fe625074b009ee0ca805\": rpc error: code = NotFound desc = could not find container \"4a49092088ec33406b51283b5dc3b0064f3bbd844e34fe625074b009ee0ca805\": container with ID starting with 4a49092088ec33406b51283b5dc3b0064f3bbd844e34fe625074b009ee0ca805 not found: ID does not exist" Oct 30 09:25:41.228700 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:41.228665 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8f40a3-84f8df4f47-6p5c8"] Oct 30 09:25:41.229153 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:41.228899 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f40a3-84f8df4f47-6p5c8" podUID="bdfbe6ee-d5c2-48e6-a7d8-a05ea3bc8e01" containerName="payload-pre-processing" containerID="cri-o://d74cb5b1825d8a598692fd0b330440b6c825515046469df8dff77ba53f5bbb4b" gracePeriod=30 Oct 30 09:25:41.447814 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:41.447784 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8f40a3-784d84f5c5-dtwh7" Oct 30 09:25:41.504358 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:41.504320 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c44644-8555bd6cd4-pmg85"] Oct 30 09:25:41.504606 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:41.504576 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c44644-8555bd6cd4-pmg85" podUID="39375e4b-ce51-48bc-b63f-fee3fd03f3b7" containerName="maas-api" containerID="cri-o://0ec7aa7aeeedb4dcfba5df11bd88d52b77a4c8596b78f92e942abbb2904f6229" gracePeriod=30 Oct 30 09:25:41.544999 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:41.544947 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eada6a8f-af88-4279-ae23-1bab2a23be56-kube-api-access-xswvz\" (UniqueName: \"kubernetes.io/projected/eada6a8f-af88-4279-ae23-1bab2a23be56-kube-api-access-xswvz\") pod \"eada6a8f-af88-4279-ae23-1bab2a23be56\" (UID: \"eada6a8f-af88-4279-ae23-1bab2a23be56\") " Oct 30 09:25:41.545206 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:41.545111 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eada6a8f-af88-4279-ae23-1bab2a23be56-metrics-certs\" (UniqueName: \"kubernetes.io/secret/eada6a8f-af88-4279-ae23-1bab2a23be56-metrics-certs\") pod \"eada6a8f-af88-4279-ae23-1bab2a23be56\" (UID: \"eada6a8f-af88-4279-ae23-1bab2a23be56\") " Oct 30 09:25:41.545206 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:41.545181 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eada6a8f-af88-4279-ae23-1bab2a23be56-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eada6a8f-af88-4279-ae23-1bab2a23be56-maas-api-tls\") pod \"eada6a8f-af88-4279-ae23-1bab2a23be56\" (UID: \"eada6a8f-af88-4279-ae23-1bab2a23be56\") " Oct 30 09:25:41.545405 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:41.545284 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eada6a8f-af88-4279-ae23-1bab2a23be56-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eada6a8f-af88-4279-ae23-1bab2a23be56-openshift-service-ca\") pod \"eada6a8f-af88-4279-ae23-1bab2a23be56\" (UID: \"eada6a8f-af88-4279-ae23-1bab2a23be56\") " Oct 30 09:25:41.546765 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:41.546729 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eada6a8f-af88-4279-ae23-1bab2a23be56-openshift-service-ca" pod "eada6a8f-af88-4279-ae23-1bab2a23be56" (UID: "eada6a8f-af88-4279-ae23-1bab2a23be56"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:25:41.548612 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:41.548460 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eada6a8f-af88-4279-ae23-1bab2a23be56-kube-api-access-xswvz" pod "eada6a8f-af88-4279-ae23-1bab2a23be56" (UID: "eada6a8f-af88-4279-ae23-1bab2a23be56"). InnerVolumeSpecName "kube-api-access-xswvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:25:41.549339 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:41.549306 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eada6a8f-af88-4279-ae23-1bab2a23be56-metrics-certs" pod "eada6a8f-af88-4279-ae23-1bab2a23be56" (UID: "eada6a8f-af88-4279-ae23-1bab2a23be56"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:25:41.550474 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:41.550423 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eada6a8f-af88-4279-ae23-1bab2a23be56-maas-api-tls" pod "eada6a8f-af88-4279-ae23-1bab2a23be56" (UID: "eada6a8f-af88-4279-ae23-1bab2a23be56"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:25:41.624268 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:41.624237 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f40a3-84f8df4f47-6p5c8" Oct 30 09:25:41.647161 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:41.647129 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdfbe6ee-d5c2-48e6-a7d8-a05ea3bc8e01-kube-api-access-45bmd\" (UniqueName: \"kubernetes.io/projected/bdfbe6ee-d5c2-48e6-a7d8-a05ea3bc8e01-kube-api-access-45bmd\") pod \"bdfbe6ee-d5c2-48e6-a7d8-a05ea3bc8e01\" (UID: \"bdfbe6ee-d5c2-48e6-a7d8-a05ea3bc8e01\") " Oct 30 09:25:41.647303 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:41.647274 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdfbe6ee-d5c2-48e6-a7d8-a05ea3bc8e01-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdfbe6ee-d5c2-48e6-a7d8-a05ea3bc8e01-plugins-config-volume\") pod \"bdfbe6ee-d5c2-48e6-a7d8-a05ea3bc8e01\" (UID: \"bdfbe6ee-d5c2-48e6-a7d8-a05ea3bc8e01\") " Oct 30 09:25:41.647824 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:41.647770 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdfbe6ee-d5c2-48e6-a7d8-a05ea3bc8e01-plugins-config-volume" pod "bdfbe6ee-d5c2-48e6-a7d8-a05ea3bc8e01" (UID: "bdfbe6ee-d5c2-48e6-a7d8-a05ea3bc8e01"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:25:41.648120 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:41.648103 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdfbe6ee-d5c2-48e6-a7d8-a05ea3bc8e01-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:25:41.648195 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:41.648123 4562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eada6a8f-af88-4279-ae23-1bab2a23be56-openshift-service-ca\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:25:41.648195 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:41.648134 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xswvz\" (UniqueName: \"kubernetes.io/projected/eada6a8f-af88-4279-ae23-1bab2a23be56-kube-api-access-xswvz\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:25:41.648195 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:41.648143 4562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eada6a8f-af88-4279-ae23-1bab2a23be56-metrics-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:25:41.648195 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:41.648151 4562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eada6a8f-af88-4279-ae23-1bab2a23be56-maas-api-tls\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:25:41.649810 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:41.649783 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdfbe6ee-d5c2-48e6-a7d8-a05ea3bc8e01-kube-api-access-45bmd" pod "bdfbe6ee-d5c2-48e6-a7d8-a05ea3bc8e01" (UID: "bdfbe6ee-d5c2-48e6-a7d8-a05ea3bc8e01"). InnerVolumeSpecName "kube-api-access-45bmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:25:41.715851 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:41.715800 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c44644-76499c9b99-ld6jd"] Oct 30 09:25:41.716466 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:41.716435 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c44644-76499c9b99-ld6jd" podUID="f9dfb6a8-2d84-49d2-b5b3-3289259cc7bf" containerName="payload-pre-processing" containerID="cri-o://8211361f376d5ac49eb8df59cd69d9aeb51ccecf3928cea1b2b6a47ec4bb2d21" gracePeriod=30 Oct 30 09:25:41.750885 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:41.750475 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45bmd\" (UniqueName: \"kubernetes.io/projected/bdfbe6ee-d5c2-48e6-a7d8-a05ea3bc8e01-kube-api-access-45bmd\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:25:41.836313 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:41.836283 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c44644-8555bd6cd4-pmg85" Oct 30 09:25:41.852420 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:41.851009 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39375e4b-ce51-48bc-b63f-fee3fd03f3b7-kube-api-access-lv9dj\" (UniqueName: \"kubernetes.io/projected/39375e4b-ce51-48bc-b63f-fee3fd03f3b7-kube-api-access-lv9dj\") pod \"39375e4b-ce51-48bc-b63f-fee3fd03f3b7\" (UID: \"39375e4b-ce51-48bc-b63f-fee3fd03f3b7\") " Oct 30 09:25:41.852420 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:41.851078 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39375e4b-ce51-48bc-b63f-fee3fd03f3b7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39375e4b-ce51-48bc-b63f-fee3fd03f3b7-openshift-service-ca\") pod \"39375e4b-ce51-48bc-b63f-fee3fd03f3b7\" (UID: \"39375e4b-ce51-48bc-b63f-fee3fd03f3b7\") " Oct 30 09:25:41.852420 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:41.851165 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39375e4b-ce51-48bc-b63f-fee3fd03f3b7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39375e4b-ce51-48bc-b63f-fee3fd03f3b7-maas-api-tls\") pod \"39375e4b-ce51-48bc-b63f-fee3fd03f3b7\" (UID: \"39375e4b-ce51-48bc-b63f-fee3fd03f3b7\") " Oct 30 09:25:41.852420 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:41.851196 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39375e4b-ce51-48bc-b63f-fee3fd03f3b7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/39375e4b-ce51-48bc-b63f-fee3fd03f3b7-metrics-certs\") pod \"39375e4b-ce51-48bc-b63f-fee3fd03f3b7\" (UID: \"39375e4b-ce51-48bc-b63f-fee3fd03f3b7\") " Oct 30 09:25:41.853963 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:41.853921 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39375e4b-ce51-48bc-b63f-fee3fd03f3b7-openshift-service-ca" pod "39375e4b-ce51-48bc-b63f-fee3fd03f3b7" (UID: "39375e4b-ce51-48bc-b63f-fee3fd03f3b7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:25:41.858747 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:41.858707 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39375e4b-ce51-48bc-b63f-fee3fd03f3b7-maas-api-tls" pod "39375e4b-ce51-48bc-b63f-fee3fd03f3b7" (UID: "39375e4b-ce51-48bc-b63f-fee3fd03f3b7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:25:41.858747 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:41.858734 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39375e4b-ce51-48bc-b63f-fee3fd03f3b7-metrics-certs" pod "39375e4b-ce51-48bc-b63f-fee3fd03f3b7" (UID: "39375e4b-ce51-48bc-b63f-fee3fd03f3b7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:25:41.864340 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:41.864291 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39375e4b-ce51-48bc-b63f-fee3fd03f3b7-kube-api-access-lv9dj" pod "39375e4b-ce51-48bc-b63f-fee3fd03f3b7" (UID: "39375e4b-ce51-48bc-b63f-fee3fd03f3b7"). InnerVolumeSpecName "kube-api-access-lv9dj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:25:41.883730 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:41.883699 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b433eaf-bc2e-4a08-8e7a-527e88ef4c59" path="/var/lib/kubelet/pods/7b433eaf-bc2e-4a08-8e7a-527e88ef4c59/volumes" Oct 30 09:25:41.960775 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:41.960623 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lv9dj\" (UniqueName: \"kubernetes.io/projected/39375e4b-ce51-48bc-b63f-fee3fd03f3b7-kube-api-access-lv9dj\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:25:41.960775 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:41.960660 4562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39375e4b-ce51-48bc-b63f-fee3fd03f3b7-openshift-service-ca\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:25:41.960775 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:41.960676 4562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39375e4b-ce51-48bc-b63f-fee3fd03f3b7-maas-api-tls\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:25:41.960775 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:41.960689 4562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39375e4b-ce51-48bc-b63f-fee3fd03f3b7-metrics-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:25:42.232478 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:42.231179 4562 generic.go:356] "Generic (PLEG): container finished" podID="eada6a8f-af88-4279-ae23-1bab2a23be56" containerID="7eae696ae7f676e5db6bb1239b6b1616057437bcf4062ebd4417ce548cc14a81" exitCode=0 Oct 30 09:25:42.232478 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:42.231447 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8f40a3-784d84f5c5-dtwh7" event={"ID":"eada6a8f-af88-4279-ae23-1bab2a23be56","Type":"ContainerDied","Data":"7eae696ae7f676e5db6bb1239b6b1616057437bcf4062ebd4417ce548cc14a81"} Oct 30 09:25:42.232478 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:42.231506 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8f40a3-784d84f5c5-dtwh7" event={"ID":"eada6a8f-af88-4279-ae23-1bab2a23be56","Type":"ContainerDied","Data":"e7b4a692828ca7598ba11d98554cf0ef4039e6f40e6cba9acd2e4b50c30983f1"} Oct 30 09:25:42.232478 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:42.231534 4562 scope.go:122] "RemoveContainer" containerID="7eae696ae7f676e5db6bb1239b6b1616057437bcf4062ebd4417ce548cc14a81" Oct 30 09:25:42.232478 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:42.231639 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8f40a3-784d84f5c5-dtwh7" Oct 30 09:25:42.240790 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:42.239982 4562 generic.go:356] "Generic (PLEG): container finished" podID="39375e4b-ce51-48bc-b63f-fee3fd03f3b7" containerID="0ec7aa7aeeedb4dcfba5df11bd88d52b77a4c8596b78f92e942abbb2904f6229" exitCode=0 Oct 30 09:25:42.240790 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:42.240072 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c44644-8555bd6cd4-pmg85" event={"ID":"39375e4b-ce51-48bc-b63f-fee3fd03f3b7","Type":"ContainerDied","Data":"0ec7aa7aeeedb4dcfba5df11bd88d52b77a4c8596b78f92e942abbb2904f6229"} Oct 30 09:25:42.240790 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:42.240108 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c44644-8555bd6cd4-pmg85" event={"ID":"39375e4b-ce51-48bc-b63f-fee3fd03f3b7","Type":"ContainerDied","Data":"81b1fdcef2c7b58b876651b5740d8609829a96aca9b8f5d7cfbf44304df0bd39"} Oct 30 09:25:42.240790 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:42.240171 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c44644-8555bd6cd4-pmg85" Oct 30 09:25:42.243169 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:42.242598 4562 generic.go:356] "Generic (PLEG): container finished" podID="f9dfb6a8-2d84-49d2-b5b3-3289259cc7bf" containerID="8211361f376d5ac49eb8df59cd69d9aeb51ccecf3928cea1b2b6a47ec4bb2d21" exitCode=0 Oct 30 09:25:42.243169 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:42.242771 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c44644-76499c9b99-ld6jd" event={"ID":"f9dfb6a8-2d84-49d2-b5b3-3289259cc7bf","Type":"ContainerDied","Data":"8211361f376d5ac49eb8df59cd69d9aeb51ccecf3928cea1b2b6a47ec4bb2d21"} Oct 30 09:25:42.245693 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:42.244689 4562 generic.go:356] "Generic (PLEG): container finished" podID="bdfbe6ee-d5c2-48e6-a7d8-a05ea3bc8e01" containerID="d74cb5b1825d8a598692fd0b330440b6c825515046469df8dff77ba53f5bbb4b" exitCode=0 Oct 30 09:25:42.245693 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:42.244728 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f40a3-84f8df4f47-6p5c8" event={"ID":"bdfbe6ee-d5c2-48e6-a7d8-a05ea3bc8e01","Type":"ContainerDied","Data":"d74cb5b1825d8a598692fd0b330440b6c825515046469df8dff77ba53f5bbb4b"} Oct 30 09:25:42.245693 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:42.244765 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f40a3-84f8df4f47-6p5c8" event={"ID":"bdfbe6ee-d5c2-48e6-a7d8-a05ea3bc8e01","Type":"ContainerDied","Data":"1286f69aa59c208b1816264b0a8f4e0b1b8e05bef659cd4aff2f1b14a12cdd9d"} Oct 30 09:25:42.245693 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:42.244829 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f40a3-84f8df4f47-6p5c8" Oct 30 09:25:42.316370 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:42.311517 4562 scope.go:122] "RemoveContainer" containerID="7eae696ae7f676e5db6bb1239b6b1616057437bcf4062ebd4417ce548cc14a81" Oct 30 09:25:42.316370 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:25:42.311851 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7eae696ae7f676e5db6bb1239b6b1616057437bcf4062ebd4417ce548cc14a81\": container with ID starting with 7eae696ae7f676e5db6bb1239b6b1616057437bcf4062ebd4417ce548cc14a81 not found: ID does not exist" containerID="7eae696ae7f676e5db6bb1239b6b1616057437bcf4062ebd4417ce548cc14a81" Oct 30 09:25:42.316370 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:42.311877 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7eae696ae7f676e5db6bb1239b6b1616057437bcf4062ebd4417ce548cc14a81"} err="failed to get container status \"7eae696ae7f676e5db6bb1239b6b1616057437bcf4062ebd4417ce548cc14a81\": rpc error: code = NotFound desc = could not find container \"7eae696ae7f676e5db6bb1239b6b1616057437bcf4062ebd4417ce548cc14a81\": container with ID starting with 7eae696ae7f676e5db6bb1239b6b1616057437bcf4062ebd4417ce548cc14a81 not found: ID does not exist" Oct 30 09:25:42.316370 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:42.311894 4562 scope.go:122] "RemoveContainer" containerID="0ec7aa7aeeedb4dcfba5df11bd88d52b77a4c8596b78f92e942abbb2904f6229" Oct 30 09:25:42.321251 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:42.321226 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c44644-76499c9b99-ld6jd" Oct 30 09:25:42.323164 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:42.323147 4562 scope.go:122] "RemoveContainer" containerID="0ec7aa7aeeedb4dcfba5df11bd88d52b77a4c8596b78f92e942abbb2904f6229" Oct 30 09:25:42.323452 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:25:42.323427 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ec7aa7aeeedb4dcfba5df11bd88d52b77a4c8596b78f92e942abbb2904f6229\": container with ID starting with 0ec7aa7aeeedb4dcfba5df11bd88d52b77a4c8596b78f92e942abbb2904f6229 not found: ID does not exist" containerID="0ec7aa7aeeedb4dcfba5df11bd88d52b77a4c8596b78f92e942abbb2904f6229" Oct 30 09:25:42.323556 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:42.323457 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ec7aa7aeeedb4dcfba5df11bd88d52b77a4c8596b78f92e942abbb2904f6229"} err="failed to get container status \"0ec7aa7aeeedb4dcfba5df11bd88d52b77a4c8596b78f92e942abbb2904f6229\": rpc error: code = NotFound desc = could not find container \"0ec7aa7aeeedb4dcfba5df11bd88d52b77a4c8596b78f92e942abbb2904f6229\": container with ID starting with 0ec7aa7aeeedb4dcfba5df11bd88d52b77a4c8596b78f92e942abbb2904f6229 not found: ID does not exist" Oct 30 09:25:42.323556 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:42.323474 4562 scope.go:122] "RemoveContainer" containerID="d74cb5b1825d8a598692fd0b330440b6c825515046469df8dff77ba53f5bbb4b" Oct 30 09:25:42.333684 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:42.333666 4562 scope.go:122] "RemoveContainer" containerID="d74cb5b1825d8a598692fd0b330440b6c825515046469df8dff77ba53f5bbb4b" Oct 30 09:25:42.333972 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:25:42.333953 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d74cb5b1825d8a598692fd0b330440b6c825515046469df8dff77ba53f5bbb4b\": container with ID starting with d74cb5b1825d8a598692fd0b330440b6c825515046469df8dff77ba53f5bbb4b not found: ID does not exist" containerID="d74cb5b1825d8a598692fd0b330440b6c825515046469df8dff77ba53f5bbb4b" Oct 30 09:25:42.334034 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:42.333980 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d74cb5b1825d8a598692fd0b330440b6c825515046469df8dff77ba53f5bbb4b"} err="failed to get container status \"d74cb5b1825d8a598692fd0b330440b6c825515046469df8dff77ba53f5bbb4b\": rpc error: code = NotFound desc = could not find container \"d74cb5b1825d8a598692fd0b330440b6c825515046469df8dff77ba53f5bbb4b\": container with ID starting with d74cb5b1825d8a598692fd0b330440b6c825515046469df8dff77ba53f5bbb4b not found: ID does not exist" Oct 30 09:25:42.394671 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:42.394634 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c44644-8555bd6cd4-pmg85"] Oct 30 09:25:42.420360 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:42.420329 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c44644-8555bd6cd4-pmg85"] Oct 30 09:25:42.450855 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:42.450824 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9dfb6a8-2d84-49d2-b5b3-3289259cc7bf-kube-api-access-tzswb\" (UniqueName: \"kubernetes.io/projected/f9dfb6a8-2d84-49d2-b5b3-3289259cc7bf-kube-api-access-tzswb\") pod \"f9dfb6a8-2d84-49d2-b5b3-3289259cc7bf\" (UID: \"f9dfb6a8-2d84-49d2-b5b3-3289259cc7bf\") " Oct 30 09:25:42.451079 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:42.451058 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9dfb6a8-2d84-49d2-b5b3-3289259cc7bf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9dfb6a8-2d84-49d2-b5b3-3289259cc7bf-plugins-config-volume\") pod \"f9dfb6a8-2d84-49d2-b5b3-3289259cc7bf\" (UID: \"f9dfb6a8-2d84-49d2-b5b3-3289259cc7bf\") " Oct 30 09:25:42.451441 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:42.451416 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9dfb6a8-2d84-49d2-b5b3-3289259cc7bf-plugins-config-volume" pod "f9dfb6a8-2d84-49d2-b5b3-3289259cc7bf" (UID: "f9dfb6a8-2d84-49d2-b5b3-3289259cc7bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:25:42.451700 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:42.451682 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9dfb6a8-2d84-49d2-b5b3-3289259cc7bf-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:25:42.453003 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:42.452984 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9dfb6a8-2d84-49d2-b5b3-3289259cc7bf-kube-api-access-tzswb" pod "f9dfb6a8-2d84-49d2-b5b3-3289259cc7bf" (UID: "f9dfb6a8-2d84-49d2-b5b3-3289259cc7bf"). InnerVolumeSpecName "kube-api-access-tzswb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:25:42.552743 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:42.552669 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzswb\" (UniqueName: \"kubernetes.io/projected/f9dfb6a8-2d84-49d2-b5b3-3289259cc7bf-kube-api-access-tzswb\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:25:42.555986 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:42.555960 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8f40a3-784d84f5c5-dtwh7"] Oct 30 09:25:42.562689 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:42.562668 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8f40a3-784d84f5c5-dtwh7"] Oct 30 09:25:42.610929 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:42.610902 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8f40a3-84f8df4f47-6p5c8"] Oct 30 09:25:42.617241 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:42.617217 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8f40a3-84f8df4f47-6p5c8"] Oct 30 09:25:43.257327 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:43.257300 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c44644-76499c9b99-ld6jd" Oct 30 09:25:43.257858 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:43.257329 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c44644-76499c9b99-ld6jd" event={"ID":"f9dfb6a8-2d84-49d2-b5b3-3289259cc7bf","Type":"ContainerDied","Data":"02be1e00c9eaa742160b25ee8ee3b5205b6c05c2fc0110937ebaf3779c9b6660"} Oct 30 09:25:43.257858 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:43.257431 4562 scope.go:122] "RemoveContainer" containerID="8211361f376d5ac49eb8df59cd69d9aeb51ccecf3928cea1b2b6a47ec4bb2d21" Oct 30 09:25:43.313827 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:43.313796 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c44644-76499c9b99-ld6jd"] Oct 30 09:25:43.326854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:43.326829 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c44644-76499c9b99-ld6jd"] Oct 30 09:25:43.877156 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:43.877121 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39375e4b-ce51-48bc-b63f-fee3fd03f3b7" path="/var/lib/kubelet/pods/39375e4b-ce51-48bc-b63f-fee3fd03f3b7/volumes" Oct 30 09:25:43.877569 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:43.877554 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdfbe6ee-d5c2-48e6-a7d8-a05ea3bc8e01" path="/var/lib/kubelet/pods/bdfbe6ee-d5c2-48e6-a7d8-a05ea3bc8e01/volumes" Oct 30 09:25:43.877968 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:43.877955 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eada6a8f-af88-4279-ae23-1bab2a23be56" path="/var/lib/kubelet/pods/eada6a8f-af88-4279-ae23-1bab2a23be56/volumes" Oct 30 09:25:43.878469 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:43.878451 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9dfb6a8-2d84-49d2-b5b3-3289259cc7bf" path="/var/lib/kubelet/pods/f9dfb6a8-2d84-49d2-b5b3-3289259cc7bf/volumes" Oct 30 09:25:49.983773 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:49.983739 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc"] Oct 30 09:25:49.984152 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:49.983951 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc" podUID="a104dc46-cd60-483a-b159-4512b460125a" containerName="istio-proxy" containerID="cri-o://21cd33cb88caab9445664f94ddb97d7a765690eecc3b37ee1b4dd2401ab1a0ef" gracePeriod=30 Oct 30 09:25:51.967848 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:51.967809 4562 patch_prober.go:29] interesting pod/e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 09:25:51.968358 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:51.967904 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc" podUID="a104dc46-cd60-483a-b159-4512b460125a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 09:25:53.612768 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:53.612726 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cbe974-6d74bdc9f6-w4szp"] Oct 30 09:25:53.613301 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:53.612972 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbe974-6d74bdc9f6-w4szp" podUID="1502d0e9-69e7-4c24-b69c-76ccc9d382f0" containerName="payload-processing" containerID="cri-o://7a8c0a6e08bf7b1abac71887ccf7e2f9125ded1d357ff36c842ba09426a5c8fb" gracePeriod=30 Oct 30 09:25:53.650124 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:53.650093 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cbe974-86947478d7-f4rmw"] Oct 30 09:25:53.650403 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:53.650326 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbe974-86947478d7-f4rmw" podUID="7cb673aa-7113-4265-a9a4-e8e78651f02b" containerName="payload-pre-processing" containerID="cri-o://240e3f3e81c6d394040380232276bddf4638dec6a47127c6467dacd83b025746" gracePeriod=30 Oct 30 09:25:54.242407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:54.241127 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbe974-86947478d7-f4rmw" Oct 30 09:25:54.243546 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:54.243519 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbe974-6d74bdc9f6-w4szp" Oct 30 09:25:54.357300 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:54.357266 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cb673aa-7113-4265-a9a4-e8e78651f02b-kube-api-access-znjtr\" (UniqueName: \"kubernetes.io/projected/7cb673aa-7113-4265-a9a4-e8e78651f02b-kube-api-access-znjtr\") pod \"7cb673aa-7113-4265-a9a4-e8e78651f02b\" (UID: \"7cb673aa-7113-4265-a9a4-e8e78651f02b\") " Oct 30 09:25:54.357489 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:54.357331 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cb673aa-7113-4265-a9a4-e8e78651f02b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cb673aa-7113-4265-a9a4-e8e78651f02b-plugins-config-volume\") pod \"7cb673aa-7113-4265-a9a4-e8e78651f02b\" (UID: \"7cb673aa-7113-4265-a9a4-e8e78651f02b\") " Oct 30 09:25:54.357489 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:54.357445 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1502d0e9-69e7-4c24-b69c-76ccc9d382f0-kube-api-access-b2t6z\" (UniqueName: \"kubernetes.io/projected/1502d0e9-69e7-4c24-b69c-76ccc9d382f0-kube-api-access-b2t6z\") pod \"1502d0e9-69e7-4c24-b69c-76ccc9d382f0\" (UID: \"1502d0e9-69e7-4c24-b69c-76ccc9d382f0\") " Oct 30 09:25:54.357623 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:54.357495 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1502d0e9-69e7-4c24-b69c-76ccc9d382f0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1502d0e9-69e7-4c24-b69c-76ccc9d382f0-plugins-config-volume\") pod \"1502d0e9-69e7-4c24-b69c-76ccc9d382f0\" (UID: \"1502d0e9-69e7-4c24-b69c-76ccc9d382f0\") " Oct 30 09:25:54.358297 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:54.357803 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cb673aa-7113-4265-a9a4-e8e78651f02b-plugins-config-volume" pod "7cb673aa-7113-4265-a9a4-e8e78651f02b" (UID: "7cb673aa-7113-4265-a9a4-e8e78651f02b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:25:54.358297 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:54.358056 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1502d0e9-69e7-4c24-b69c-76ccc9d382f0-plugins-config-volume" pod "1502d0e9-69e7-4c24-b69c-76ccc9d382f0" (UID: "1502d0e9-69e7-4c24-b69c-76ccc9d382f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:25:54.358297 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:54.358249 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1502d0e9-69e7-4c24-b69c-76ccc9d382f0-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:25:54.358297 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:54.358270 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cb673aa-7113-4265-a9a4-e8e78651f02b-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:25:54.359752 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:54.359727 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cb673aa-7113-4265-a9a4-e8e78651f02b-kube-api-access-znjtr" pod "7cb673aa-7113-4265-a9a4-e8e78651f02b" (UID: "7cb673aa-7113-4265-a9a4-e8e78651f02b"). InnerVolumeSpecName "kube-api-access-znjtr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:25:54.360136 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:54.360117 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1502d0e9-69e7-4c24-b69c-76ccc9d382f0-kube-api-access-b2t6z" pod "1502d0e9-69e7-4c24-b69c-76ccc9d382f0" (UID: "1502d0e9-69e7-4c24-b69c-76ccc9d382f0"). InnerVolumeSpecName "kube-api-access-b2t6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:25:54.398675 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:54.398647 4562 generic.go:356] "Generic (PLEG): container finished" podID="1502d0e9-69e7-4c24-b69c-76ccc9d382f0" containerID="7a8c0a6e08bf7b1abac71887ccf7e2f9125ded1d357ff36c842ba09426a5c8fb" exitCode=0 Oct 30 09:25:54.398816 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:54.398693 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbe974-6d74bdc9f6-w4szp" Oct 30 09:25:54.398816 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:54.398735 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbe974-6d74bdc9f6-w4szp" event={"ID":"1502d0e9-69e7-4c24-b69c-76ccc9d382f0","Type":"ContainerDied","Data":"7a8c0a6e08bf7b1abac71887ccf7e2f9125ded1d357ff36c842ba09426a5c8fb"} Oct 30 09:25:54.398816 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:54.398794 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbe974-6d74bdc9f6-w4szp" event={"ID":"1502d0e9-69e7-4c24-b69c-76ccc9d382f0","Type":"ContainerDied","Data":"a14641044e60aefef56a5857450a9704917d5ea7b30ea113f49690ca408ba7eb"} Oct 30 09:25:54.398966 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:54.398830 4562 scope.go:122] "RemoveContainer" containerID="7a8c0a6e08bf7b1abac71887ccf7e2f9125ded1d357ff36c842ba09426a5c8fb" Oct 30 09:25:54.400345 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:54.400320 4562 generic.go:356] "Generic (PLEG): container finished" podID="7cb673aa-7113-4265-a9a4-e8e78651f02b" containerID="240e3f3e81c6d394040380232276bddf4638dec6a47127c6467dacd83b025746" exitCode=0 Oct 30 09:25:54.400466 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:54.400367 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbe974-86947478d7-f4rmw" Oct 30 09:25:54.400466 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:54.400360 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbe974-86947478d7-f4rmw" event={"ID":"7cb673aa-7113-4265-a9a4-e8e78651f02b","Type":"ContainerDied","Data":"240e3f3e81c6d394040380232276bddf4638dec6a47127c6467dacd83b025746"} Oct 30 09:25:54.400561 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:54.400486 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbe974-86947478d7-f4rmw" event={"ID":"7cb673aa-7113-4265-a9a4-e8e78651f02b","Type":"ContainerDied","Data":"4a826173a567f51b75c2d935f9eb1ba3ff45a3ccbf60babee14c16818c11df01"} Oct 30 09:25:54.411991 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:54.411971 4562 scope.go:122] "RemoveContainer" containerID="7a8c0a6e08bf7b1abac71887ccf7e2f9125ded1d357ff36c842ba09426a5c8fb" Oct 30 09:25:54.412335 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:25:54.412314 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a8c0a6e08bf7b1abac71887ccf7e2f9125ded1d357ff36c842ba09426a5c8fb\": container with ID starting with 7a8c0a6e08bf7b1abac71887ccf7e2f9125ded1d357ff36c842ba09426a5c8fb not found: ID does not exist" containerID="7a8c0a6e08bf7b1abac71887ccf7e2f9125ded1d357ff36c842ba09426a5c8fb" Oct 30 09:25:54.412416 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:54.412345 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a8c0a6e08bf7b1abac71887ccf7e2f9125ded1d357ff36c842ba09426a5c8fb"} err="failed to get container status \"7a8c0a6e08bf7b1abac71887ccf7e2f9125ded1d357ff36c842ba09426a5c8fb\": rpc error: code = NotFound desc = could not find container \"7a8c0a6e08bf7b1abac71887ccf7e2f9125ded1d357ff36c842ba09426a5c8fb\": container with ID starting with 7a8c0a6e08bf7b1abac71887ccf7e2f9125ded1d357ff36c842ba09426a5c8fb not found: ID does not exist" Oct 30 09:25:54.412416 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:54.412361 4562 scope.go:122] "RemoveContainer" containerID="240e3f3e81c6d394040380232276bddf4638dec6a47127c6467dacd83b025746" Oct 30 09:25:54.423203 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:54.423180 4562 scope.go:122] "RemoveContainer" containerID="240e3f3e81c6d394040380232276bddf4638dec6a47127c6467dacd83b025746" Oct 30 09:25:54.423524 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:25:54.423500 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"240e3f3e81c6d394040380232276bddf4638dec6a47127c6467dacd83b025746\": container with ID starting with 240e3f3e81c6d394040380232276bddf4638dec6a47127c6467dacd83b025746 not found: ID does not exist" containerID="240e3f3e81c6d394040380232276bddf4638dec6a47127c6467dacd83b025746" Oct 30 09:25:54.423598 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:54.423532 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"240e3f3e81c6d394040380232276bddf4638dec6a47127c6467dacd83b025746"} err="failed to get container status \"240e3f3e81c6d394040380232276bddf4638dec6a47127c6467dacd83b025746\": rpc error: code = NotFound desc = could not find container \"240e3f3e81c6d394040380232276bddf4638dec6a47127c6467dacd83b025746\": container with ID starting with 240e3f3e81c6d394040380232276bddf4638dec6a47127c6467dacd83b025746 not found: ID does not exist" Oct 30 09:25:54.454737 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:54.454677 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cbe974-6d74bdc9f6-w4szp"] Oct 30 09:25:54.458033 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:54.458010 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cbe974-6d74bdc9f6-w4szp"] Oct 30 09:25:54.459963 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:54.459942 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2t6z\" (UniqueName: \"kubernetes.io/projected/1502d0e9-69e7-4c24-b69c-76ccc9d382f0-kube-api-access-b2t6z\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:25:54.459963 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:54.459965 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znjtr\" (UniqueName: \"kubernetes.io/projected/7cb673aa-7113-4265-a9a4-e8e78651f02b-kube-api-access-znjtr\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:25:54.491322 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:54.491298 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cbe974-86947478d7-f4rmw"] Oct 30 09:25:54.495258 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:54.495238 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cbe974-86947478d7-f4rmw"] Oct 30 09:25:55.259222 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:55.259157 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc" Oct 30 09:25:55.376268 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:55.376237 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a104dc46-cd60-483a-b159-4512b460125a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a104dc46-cd60-483a-b159-4512b460125a-istio-podinfo\") pod \"a104dc46-cd60-483a-b159-4512b460125a\" (UID: \"a104dc46-cd60-483a-b159-4512b460125a\") " Oct 30 09:25:55.376425 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:55.376278 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a104dc46-cd60-483a-b159-4512b460125a-kube-api-access-b8drg\" (UniqueName: \"kubernetes.io/projected/a104dc46-cd60-483a-b159-4512b460125a-kube-api-access-b8drg\") pod \"a104dc46-cd60-483a-b159-4512b460125a\" (UID: \"a104dc46-cd60-483a-b159-4512b460125a\") " Oct 30 09:25:55.376468 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:55.376441 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a104dc46-cd60-483a-b159-4512b460125a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a104dc46-cd60-483a-b159-4512b460125a-istio-data\") pod \"a104dc46-cd60-483a-b159-4512b460125a\" (UID: \"a104dc46-cd60-483a-b159-4512b460125a\") " Oct 30 09:25:55.376535 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:55.376484 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a104dc46-cd60-483a-b159-4512b460125a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a104dc46-cd60-483a-b159-4512b460125a-credential-socket\") pod \"a104dc46-cd60-483a-b159-4512b460125a\" (UID: \"a104dc46-cd60-483a-b159-4512b460125a\") " Oct 30 09:25:55.376535 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:55.376522 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a104dc46-cd60-483a-b159-4512b460125a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a104dc46-cd60-483a-b159-4512b460125a-istiod-ca-cert\") pod \"a104dc46-cd60-483a-b159-4512b460125a\" (UID: \"a104dc46-cd60-483a-b159-4512b460125a\") " Oct 30 09:25:55.376646 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:55.376606 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a104dc46-cd60-483a-b159-4512b460125a-istio-token\" (UniqueName: \"kubernetes.io/projected/a104dc46-cd60-483a-b159-4512b460125a-istio-token\") pod \"a104dc46-cd60-483a-b159-4512b460125a\" (UID: \"a104dc46-cd60-483a-b159-4512b460125a\") " Oct 30 09:25:55.376696 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:55.376682 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a104dc46-cd60-483a-b159-4512b460125a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a104dc46-cd60-483a-b159-4512b460125a-workload-socket\") pod \"a104dc46-cd60-483a-b159-4512b460125a\" (UID: \"a104dc46-cd60-483a-b159-4512b460125a\") " Oct 30 09:25:55.376774 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:55.376730 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a104dc46-cd60-483a-b159-4512b460125a-istio-data" pod "a104dc46-cd60-483a-b159-4512b460125a" (UID: "a104dc46-cd60-483a-b159-4512b460125a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:55.376851 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:55.376790 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a104dc46-cd60-483a-b159-4512b460125a-credential-socket" pod "a104dc46-cd60-483a-b159-4512b460125a" (UID: "a104dc46-cd60-483a-b159-4512b460125a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:55.376851 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:55.376828 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a104dc46-cd60-483a-b159-4512b460125a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a104dc46-cd60-483a-b159-4512b460125a-istio-envoy\") pod \"a104dc46-cd60-483a-b159-4512b460125a\" (UID: \"a104dc46-cd60-483a-b159-4512b460125a\") " Oct 30 09:25:55.376962 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:55.376873 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a104dc46-cd60-483a-b159-4512b460125a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a104dc46-cd60-483a-b159-4512b460125a-workload-certs\") pod \"a104dc46-cd60-483a-b159-4512b460125a\" (UID: \"a104dc46-cd60-483a-b159-4512b460125a\") " Oct 30 09:25:55.377049 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:55.377029 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a104dc46-cd60-483a-b159-4512b460125a-workload-socket" pod "a104dc46-cd60-483a-b159-4512b460125a" (UID: "a104dc46-cd60-483a-b159-4512b460125a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:55.377116 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:55.377086 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a104dc46-cd60-483a-b159-4512b460125a-istiod-ca-cert" pod "a104dc46-cd60-483a-b159-4512b460125a" (UID: "a104dc46-cd60-483a-b159-4512b460125a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:25:55.377320 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:55.377300 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a104dc46-cd60-483a-b159-4512b460125a-workload-certs" pod "a104dc46-cd60-483a-b159-4512b460125a" (UID: "a104dc46-cd60-483a-b159-4512b460125a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:55.377600 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:55.377582 4562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a104dc46-cd60-483a-b159-4512b460125a-istio-data\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:25:55.377600 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:55.377606 4562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a104dc46-cd60-483a-b159-4512b460125a-credential-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:25:55.377727 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:55.377620 4562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a104dc46-cd60-483a-b159-4512b460125a-istiod-ca-cert\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:25:55.377727 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:55.377633 4562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a104dc46-cd60-483a-b159-4512b460125a-workload-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:25:55.377727 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:55.377646 4562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a104dc46-cd60-483a-b159-4512b460125a-workload-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:25:55.378957 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:55.378925 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a104dc46-cd60-483a-b159-4512b460125a-istio-podinfo" pod "a104dc46-cd60-483a-b159-4512b460125a" (UID: "a104dc46-cd60-483a-b159-4512b460125a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:25:55.379146 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:55.379114 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a104dc46-cd60-483a-b159-4512b460125a-kube-api-access-b8drg" pod "a104dc46-cd60-483a-b159-4512b460125a" (UID: "a104dc46-cd60-483a-b159-4512b460125a"). InnerVolumeSpecName "kube-api-access-b8drg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:25:55.379516 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:55.379486 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a104dc46-cd60-483a-b159-4512b460125a-istio-envoy" pod "a104dc46-cd60-483a-b159-4512b460125a" (UID: "a104dc46-cd60-483a-b159-4512b460125a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:25:55.379987 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:55.379964 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a104dc46-cd60-483a-b159-4512b460125a-istio-token" pod "a104dc46-cd60-483a-b159-4512b460125a" (UID: "a104dc46-cd60-483a-b159-4512b460125a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:25:55.408854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:55.408819 4562 generic.go:356] "Generic (PLEG): container finished" podID="a104dc46-cd60-483a-b159-4512b460125a" containerID="21cd33cb88caab9445664f94ddb97d7a765690eecc3b37ee1b4dd2401ab1a0ef" exitCode=0 Oct 30 09:25:55.408977 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:55.408872 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc" Oct 30 09:25:55.408977 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:55.408901 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc" event={"ID":"a104dc46-cd60-483a-b159-4512b460125a","Type":"ContainerDied","Data":"21cd33cb88caab9445664f94ddb97d7a765690eecc3b37ee1b4dd2401ab1a0ef"} Oct 30 09:25:55.408977 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:55.408952 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc" event={"ID":"a104dc46-cd60-483a-b159-4512b460125a","Type":"ContainerDied","Data":"2020d4f08ef7b99348a0e7d8a93669c404b4f2b0c0ef7ec48a9046f1c2fe683f"} Oct 30 09:25:55.409125 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:55.408980 4562 scope.go:122] "RemoveContainer" containerID="21cd33cb88caab9445664f94ddb97d7a765690eecc3b37ee1b4dd2401ab1a0ef" Oct 30 09:25:55.419568 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:55.419550 4562 scope.go:122] "RemoveContainer" containerID="3b6eb58a9768a11f8e18b1b69dc60bc1d2ac09c48dff91ca04ad0667be45f33a" Oct 30 09:25:55.429087 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:55.429067 4562 scope.go:122] "RemoveContainer" containerID="21cd33cb88caab9445664f94ddb97d7a765690eecc3b37ee1b4dd2401ab1a0ef" Oct 30 09:25:55.429346 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:25:55.429325 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21cd33cb88caab9445664f94ddb97d7a765690eecc3b37ee1b4dd2401ab1a0ef\": container with ID starting with 21cd33cb88caab9445664f94ddb97d7a765690eecc3b37ee1b4dd2401ab1a0ef not found: ID does not exist" containerID="21cd33cb88caab9445664f94ddb97d7a765690eecc3b37ee1b4dd2401ab1a0ef" Oct 30 09:25:55.429408 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:55.429353 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21cd33cb88caab9445664f94ddb97d7a765690eecc3b37ee1b4dd2401ab1a0ef"} err="failed to get container status \"21cd33cb88caab9445664f94ddb97d7a765690eecc3b37ee1b4dd2401ab1a0ef\": rpc error: code = NotFound desc = could not find container \"21cd33cb88caab9445664f94ddb97d7a765690eecc3b37ee1b4dd2401ab1a0ef\": container with ID starting with 21cd33cb88caab9445664f94ddb97d7a765690eecc3b37ee1b4dd2401ab1a0ef not found: ID does not exist" Oct 30 09:25:55.429408 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:55.429371 4562 scope.go:122] "RemoveContainer" containerID="3b6eb58a9768a11f8e18b1b69dc60bc1d2ac09c48dff91ca04ad0667be45f33a" Oct 30 09:25:55.429654 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:25:55.429634 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b6eb58a9768a11f8e18b1b69dc60bc1d2ac09c48dff91ca04ad0667be45f33a\": container with ID starting with 3b6eb58a9768a11f8e18b1b69dc60bc1d2ac09c48dff91ca04ad0667be45f33a not found: ID does not exist" containerID="3b6eb58a9768a11f8e18b1b69dc60bc1d2ac09c48dff91ca04ad0667be45f33a" Oct 30 09:25:55.429698 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:55.429660 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b6eb58a9768a11f8e18b1b69dc60bc1d2ac09c48dff91ca04ad0667be45f33a"} err="failed to get container status \"3b6eb58a9768a11f8e18b1b69dc60bc1d2ac09c48dff91ca04ad0667be45f33a\": rpc error: code = NotFound desc = could not find container \"3b6eb58a9768a11f8e18b1b69dc60bc1d2ac09c48dff91ca04ad0667be45f33a\": container with ID starting with 3b6eb58a9768a11f8e18b1b69dc60bc1d2ac09c48dff91ca04ad0667be45f33a not found: ID does not exist" Oct 30 09:25:55.465998 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:55.465968 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc"] Oct 30 09:25:55.470583 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:55.470553 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-c44644-openshift-default-5cbdc5799c-pmljc"] Oct 30 09:25:55.479133 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:55.479108 4562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a104dc46-cd60-483a-b159-4512b460125a-istio-podinfo\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:25:55.479247 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:55.479137 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8drg\" (UniqueName: \"kubernetes.io/projected/a104dc46-cd60-483a-b159-4512b460125a-kube-api-access-b8drg\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:25:55.479247 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:55.479155 4562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a104dc46-cd60-483a-b159-4512b460125a-istio-token\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:25:55.479247 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:55.479170 4562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a104dc46-cd60-483a-b159-4512b460125a-istio-envoy\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:25:55.876441 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:55.876412 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1502d0e9-69e7-4c24-b69c-76ccc9d382f0" path="/var/lib/kubelet/pods/1502d0e9-69e7-4c24-b69c-76ccc9d382f0/volumes" Oct 30 09:25:55.876816 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:55.876803 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cb673aa-7113-4265-a9a4-e8e78651f02b" path="/var/lib/kubelet/pods/7cb673aa-7113-4265-a9a4-e8e78651f02b/volumes" Oct 30 09:25:55.877223 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:25:55.877210 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a104dc46-cd60-483a-b159-4512b460125a" path="/var/lib/kubelet/pods/a104dc46-cd60-483a-b159-4512b460125a/volumes" Oct 30 09:25:58.597700 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:25:58.597659 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:26:05.276966 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:05.276930 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-962e2e-7f64d7567d-h86tm"] Oct 30 09:26:05.277468 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:05.277139 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-962e2e-7f64d7567d-h86tm" podUID="b0cefc21-e1d4-45ed-b81a-68365ab8e716" containerName="payload-processing" containerID="cri-o://8deebfb29cae9ce612467c3c05a08fb1c37f5ed299a0b7397189ea809af39b89" gracePeriod=30 Oct 30 09:26:05.300682 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:05.300645 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-962e2e-5788599b6c-96x72"] Oct 30 09:26:05.301662 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:05.301367 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-962e2e-5788599b6c-96x72" podUID="120fcc9d-1535-497e-94ca-827940c4323c" containerName="payload-pre-processing" containerID="cri-o://c439c983adc54ba174413e11a59734e3dac3cc90e6430ddcfbda1e848512e463" gracePeriod=30 Oct 30 09:26:05.865407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:05.862200 4562 generic.go:356] "Generic (PLEG): container finished" podID="120fcc9d-1535-497e-94ca-827940c4323c" containerID="c439c983adc54ba174413e11a59734e3dac3cc90e6430ddcfbda1e848512e463" exitCode=0 Oct 30 09:26:05.865407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:05.862525 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-962e2e-5788599b6c-96x72" event={"ID":"120fcc9d-1535-497e-94ca-827940c4323c","Type":"ContainerDied","Data":"c439c983adc54ba174413e11a59734e3dac3cc90e6430ddcfbda1e848512e463"} Oct 30 09:26:05.894403 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:05.894083 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-962e2e-5788599b6c-96x72" Oct 30 09:26:05.941521 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:05.937101 4562 generic.go:356] "Generic (PLEG): container finished" podID="b0cefc21-e1d4-45ed-b81a-68365ab8e716" containerID="8deebfb29cae9ce612467c3c05a08fb1c37f5ed299a0b7397189ea809af39b89" exitCode=0 Oct 30 09:26:05.963441 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:05.963258 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-962e2e-7f64d7567d-h86tm" event={"ID":"b0cefc21-e1d4-45ed-b81a-68365ab8e716","Type":"ContainerDied","Data":"8deebfb29cae9ce612467c3c05a08fb1c37f5ed299a0b7397189ea809af39b89"} Oct 30 09:26:05.975036 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:05.975010 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-962e2e-7f64d7567d-h86tm" Oct 30 09:26:06.007302 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:06.007209 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/120fcc9d-1535-497e-94ca-827940c4323c-kube-api-access-hmbvl\" (UniqueName: \"kubernetes.io/projected/120fcc9d-1535-497e-94ca-827940c4323c-kube-api-access-hmbvl\") pod \"120fcc9d-1535-497e-94ca-827940c4323c\" (UID: \"120fcc9d-1535-497e-94ca-827940c4323c\") " Oct 30 09:26:06.007566 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:06.007394 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/120fcc9d-1535-497e-94ca-827940c4323c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/120fcc9d-1535-497e-94ca-827940c4323c-plugins-config-volume\") pod \"120fcc9d-1535-497e-94ca-827940c4323c\" (UID: \"120fcc9d-1535-497e-94ca-827940c4323c\") " Oct 30 09:26:06.008370 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:06.008344 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/120fcc9d-1535-497e-94ca-827940c4323c-plugins-config-volume" pod "120fcc9d-1535-497e-94ca-827940c4323c" (UID: "120fcc9d-1535-497e-94ca-827940c4323c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:26:06.009671 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:06.009643 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/120fcc9d-1535-497e-94ca-827940c4323c-kube-api-access-hmbvl" pod "120fcc9d-1535-497e-94ca-827940c4323c" (UID: "120fcc9d-1535-497e-94ca-827940c4323c"). InnerVolumeSpecName "kube-api-access-hmbvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:26:06.109131 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:06.109096 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0cefc21-e1d4-45ed-b81a-68365ab8e716-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0cefc21-e1d4-45ed-b81a-68365ab8e716-plugins-config-volume\") pod \"b0cefc21-e1d4-45ed-b81a-68365ab8e716\" (UID: \"b0cefc21-e1d4-45ed-b81a-68365ab8e716\") " Oct 30 09:26:06.109324 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:06.109152 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0cefc21-e1d4-45ed-b81a-68365ab8e716-kube-api-access-45648\" (UniqueName: \"kubernetes.io/projected/b0cefc21-e1d4-45ed-b81a-68365ab8e716-kube-api-access-45648\") pod \"b0cefc21-e1d4-45ed-b81a-68365ab8e716\" (UID: \"b0cefc21-e1d4-45ed-b81a-68365ab8e716\") " Oct 30 09:26:06.109520 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:06.109495 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0cefc21-e1d4-45ed-b81a-68365ab8e716-plugins-config-volume" pod "b0cefc21-e1d4-45ed-b81a-68365ab8e716" (UID: "b0cefc21-e1d4-45ed-b81a-68365ab8e716"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:26:06.110079 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:06.110055 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0cefc21-e1d4-45ed-b81a-68365ab8e716-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:26:06.110130 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:06.110087 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/120fcc9d-1535-497e-94ca-827940c4323c-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:26:06.110130 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:06.110102 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmbvl\" (UniqueName: \"kubernetes.io/projected/120fcc9d-1535-497e-94ca-827940c4323c-kube-api-access-hmbvl\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:26:06.111316 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:06.111293 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0cefc21-e1d4-45ed-b81a-68365ab8e716-kube-api-access-45648" pod "b0cefc21-e1d4-45ed-b81a-68365ab8e716" (UID: "b0cefc21-e1d4-45ed-b81a-68365ab8e716"). InnerVolumeSpecName "kube-api-access-45648". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:26:06.211542 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:06.211505 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45648\" (UniqueName: \"kubernetes.io/projected/b0cefc21-e1d4-45ed-b81a-68365ab8e716-kube-api-access-45648\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:26:06.945055 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:06.945027 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-962e2e-7f64d7567d-h86tm" Oct 30 09:26:06.945055 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:06.945043 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-962e2e-7f64d7567d-h86tm" event={"ID":"b0cefc21-e1d4-45ed-b81a-68365ab8e716","Type":"ContainerDied","Data":"e9a621a1086a420c770802616cdff5decfa0286d8e40dfe586b7610d43e9cad8"} Oct 30 09:26:06.945642 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:06.945098 4562 scope.go:122] "RemoveContainer" containerID="8deebfb29cae9ce612467c3c05a08fb1c37f5ed299a0b7397189ea809af39b89" Oct 30 09:26:06.946537 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:06.946512 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-962e2e-5788599b6c-96x72" event={"ID":"120fcc9d-1535-497e-94ca-827940c4323c","Type":"ContainerDied","Data":"e5eac35dda2642236730ea4d33bbebbc76ac517a2fabdfff5e500257d5673cdc"} Oct 30 09:26:06.946604 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:06.946556 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-962e2e-5788599b6c-96x72" Oct 30 09:26:06.958469 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:06.958444 4562 scope.go:122] "RemoveContainer" containerID="c439c983adc54ba174413e11a59734e3dac3cc90e6430ddcfbda1e848512e463" Oct 30 09:26:07.001620 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:07.001592 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-962e2e-7f64d7567d-h86tm"] Oct 30 09:26:07.013920 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:07.013895 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-962e2e-7f64d7567d-h86tm"] Oct 30 09:26:07.043140 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:07.043108 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-962e2e-5788599b6c-96x72"] Oct 30 09:26:07.048466 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:07.048442 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-962e2e-5788599b6c-96x72"] Oct 30 09:26:07.875781 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:07.875745 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="120fcc9d-1535-497e-94ca-827940c4323c" path="/var/lib/kubelet/pods/120fcc9d-1535-497e-94ca-827940c4323c/volumes" Oct 30 09:26:07.876309 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:07.876287 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0cefc21-e1d4-45ed-b81a-68365ab8e716" path="/var/lib/kubelet/pods/b0cefc21-e1d4-45ed-b81a-68365ab8e716/volumes" Oct 30 09:26:13.364806 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:13.364719 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f"] Oct 30 09:26:13.365345 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:13.364980 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f" podUID="286eb740-46f4-44ba-80f1-0de69bd89861" containerName="istio-proxy" containerID="cri-o://088b8fdd949b1a7c4e46586e842aedc814000f85df6fc2659b3ba3679e0d9217" gracePeriod=30 Oct 30 09:26:18.640855 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:18.640790 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f" Oct 30 09:26:18.763639 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:18.763597 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/286eb740-46f4-44ba-80f1-0de69bd89861-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/286eb740-46f4-44ba-80f1-0de69bd89861-istio-podinfo\") pod \"286eb740-46f4-44ba-80f1-0de69bd89861\" (UID: \"286eb740-46f4-44ba-80f1-0de69bd89861\") " Oct 30 09:26:18.763831 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:18.763652 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/286eb740-46f4-44ba-80f1-0de69bd89861-kube-api-access-6xnwh\" (UniqueName: \"kubernetes.io/projected/286eb740-46f4-44ba-80f1-0de69bd89861-kube-api-access-6xnwh\") pod \"286eb740-46f4-44ba-80f1-0de69bd89861\" (UID: \"286eb740-46f4-44ba-80f1-0de69bd89861\") " Oct 30 09:26:18.763831 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:18.763730 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/286eb740-46f4-44ba-80f1-0de69bd89861-istio-token\" (UniqueName: \"kubernetes.io/projected/286eb740-46f4-44ba-80f1-0de69bd89861-istio-token\") pod \"286eb740-46f4-44ba-80f1-0de69bd89861\" (UID: \"286eb740-46f4-44ba-80f1-0de69bd89861\") " Oct 30 09:26:18.763831 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:18.763759 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/286eb740-46f4-44ba-80f1-0de69bd89861-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/286eb740-46f4-44ba-80f1-0de69bd89861-workload-certs\") pod \"286eb740-46f4-44ba-80f1-0de69bd89861\" (UID: \"286eb740-46f4-44ba-80f1-0de69bd89861\") " Oct 30 09:26:18.763995 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:18.763899 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/286eb740-46f4-44ba-80f1-0de69bd89861-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/286eb740-46f4-44ba-80f1-0de69bd89861-istio-envoy\") pod \"286eb740-46f4-44ba-80f1-0de69bd89861\" (UID: \"286eb740-46f4-44ba-80f1-0de69bd89861\") " Oct 30 09:26:18.763995 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:18.763967 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/286eb740-46f4-44ba-80f1-0de69bd89861-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/286eb740-46f4-44ba-80f1-0de69bd89861-credential-socket\") pod \"286eb740-46f4-44ba-80f1-0de69bd89861\" (UID: \"286eb740-46f4-44ba-80f1-0de69bd89861\") " Oct 30 09:26:18.764097 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:18.764039 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/286eb740-46f4-44ba-80f1-0de69bd89861-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/286eb740-46f4-44ba-80f1-0de69bd89861-istiod-ca-cert\") pod \"286eb740-46f4-44ba-80f1-0de69bd89861\" (UID: \"286eb740-46f4-44ba-80f1-0de69bd89861\") " Oct 30 09:26:18.764150 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:18.764121 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/286eb740-46f4-44ba-80f1-0de69bd89861-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/286eb740-46f4-44ba-80f1-0de69bd89861-workload-socket\") pod \"286eb740-46f4-44ba-80f1-0de69bd89861\" (UID: \"286eb740-46f4-44ba-80f1-0de69bd89861\") " Oct 30 09:26:18.764685 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:18.764257 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/286eb740-46f4-44ba-80f1-0de69bd89861-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/286eb740-46f4-44ba-80f1-0de69bd89861-istio-data\") pod \"286eb740-46f4-44ba-80f1-0de69bd89861\" (UID: \"286eb740-46f4-44ba-80f1-0de69bd89861\") " Oct 30 09:26:18.764685 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:18.764501 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/286eb740-46f4-44ba-80f1-0de69bd89861-workload-socket" pod "286eb740-46f4-44ba-80f1-0de69bd89861" (UID: "286eb740-46f4-44ba-80f1-0de69bd89861"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:26:18.764685 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:18.764562 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/286eb740-46f4-44ba-80f1-0de69bd89861-istiod-ca-cert" pod "286eb740-46f4-44ba-80f1-0de69bd89861" (UID: "286eb740-46f4-44ba-80f1-0de69bd89861"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:26:18.764899 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:18.764719 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/286eb740-46f4-44ba-80f1-0de69bd89861-istio-data" pod "286eb740-46f4-44ba-80f1-0de69bd89861" (UID: "286eb740-46f4-44ba-80f1-0de69bd89861"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:26:18.764899 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:18.764783 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/286eb740-46f4-44ba-80f1-0de69bd89861-credential-socket" pod "286eb740-46f4-44ba-80f1-0de69bd89861" (UID: "286eb740-46f4-44ba-80f1-0de69bd89861"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:26:18.764899 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:18.764844 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/286eb740-46f4-44ba-80f1-0de69bd89861-workload-certs" pod "286eb740-46f4-44ba-80f1-0de69bd89861" (UID: "286eb740-46f4-44ba-80f1-0de69bd89861"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:26:18.765222 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:18.765199 4562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/286eb740-46f4-44ba-80f1-0de69bd89861-istio-data\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:26:18.765271 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:18.765229 4562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/286eb740-46f4-44ba-80f1-0de69bd89861-workload-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:26:18.765271 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:18.765244 4562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/286eb740-46f4-44ba-80f1-0de69bd89861-credential-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:26:18.765271 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:18.765260 4562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/286eb740-46f4-44ba-80f1-0de69bd89861-istiod-ca-cert\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:26:18.765371 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:18.765273 4562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/286eb740-46f4-44ba-80f1-0de69bd89861-workload-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:26:18.766120 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:18.766097 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/286eb740-46f4-44ba-80f1-0de69bd89861-istio-token" pod "286eb740-46f4-44ba-80f1-0de69bd89861" (UID: "286eb740-46f4-44ba-80f1-0de69bd89861"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:26:18.766281 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:18.766264 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/286eb740-46f4-44ba-80f1-0de69bd89861-istio-podinfo" pod "286eb740-46f4-44ba-80f1-0de69bd89861" (UID: "286eb740-46f4-44ba-80f1-0de69bd89861"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:26:18.766360 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:18.766339 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/286eb740-46f4-44ba-80f1-0de69bd89861-istio-envoy" pod "286eb740-46f4-44ba-80f1-0de69bd89861" (UID: "286eb740-46f4-44ba-80f1-0de69bd89861"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:26:18.766422 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:18.766353 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/286eb740-46f4-44ba-80f1-0de69bd89861-kube-api-access-6xnwh" pod "286eb740-46f4-44ba-80f1-0de69bd89861" (UID: "286eb740-46f4-44ba-80f1-0de69bd89861"). InnerVolumeSpecName "kube-api-access-6xnwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:26:18.866911 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:18.866864 4562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/286eb740-46f4-44ba-80f1-0de69bd89861-istio-podinfo\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:26:18.866911 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:18.866907 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xnwh\" (UniqueName: \"kubernetes.io/projected/286eb740-46f4-44ba-80f1-0de69bd89861-kube-api-access-6xnwh\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:26:18.866911 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:18.866924 4562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/286eb740-46f4-44ba-80f1-0de69bd89861-istio-token\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:26:18.867232 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:18.866937 4562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/286eb740-46f4-44ba-80f1-0de69bd89861-istio-envoy\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:26:19.101468 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:19.101432 4562 generic.go:356] "Generic (PLEG): container finished" podID="286eb740-46f4-44ba-80f1-0de69bd89861" containerID="088b8fdd949b1a7c4e46586e842aedc814000f85df6fc2659b3ba3679e0d9217" exitCode=0 Oct 30 09:26:19.101651 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:19.101483 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f" Oct 30 09:26:19.101651 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:19.101492 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f" event={"ID":"286eb740-46f4-44ba-80f1-0de69bd89861","Type":"ContainerDied","Data":"088b8fdd949b1a7c4e46586e842aedc814000f85df6fc2659b3ba3679e0d9217"} Oct 30 09:26:19.101651 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:19.101555 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f" event={"ID":"286eb740-46f4-44ba-80f1-0de69bd89861","Type":"ContainerDied","Data":"4a1f386c91ba3924fddbc6318f32313ee055b30bccd5270f5d13c1ec1232ce1a"} Oct 30 09:26:19.101651 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:19.101585 4562 scope.go:122] "RemoveContainer" containerID="088b8fdd949b1a7c4e46586e842aedc814000f85df6fc2659b3ba3679e0d9217" Oct 30 09:26:19.112763 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:19.112747 4562 scope.go:122] "RemoveContainer" containerID="bce982b30d40131ba088ef0c2518e3b3d42db0314ab6426dde722029d794bcd8" Oct 30 09:26:19.122487 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:19.122406 4562 scope.go:122] "RemoveContainer" containerID="088b8fdd949b1a7c4e46586e842aedc814000f85df6fc2659b3ba3679e0d9217" Oct 30 09:26:19.122712 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:26:19.122692 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"088b8fdd949b1a7c4e46586e842aedc814000f85df6fc2659b3ba3679e0d9217\": container with ID starting with 088b8fdd949b1a7c4e46586e842aedc814000f85df6fc2659b3ba3679e0d9217 not found: ID does not exist" containerID="088b8fdd949b1a7c4e46586e842aedc814000f85df6fc2659b3ba3679e0d9217" Oct 30 09:26:19.122763 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:19.122720 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"088b8fdd949b1a7c4e46586e842aedc814000f85df6fc2659b3ba3679e0d9217"} err="failed to get container status \"088b8fdd949b1a7c4e46586e842aedc814000f85df6fc2659b3ba3679e0d9217\": rpc error: code = NotFound desc = could not find container \"088b8fdd949b1a7c4e46586e842aedc814000f85df6fc2659b3ba3679e0d9217\": container with ID starting with 088b8fdd949b1a7c4e46586e842aedc814000f85df6fc2659b3ba3679e0d9217 not found: ID does not exist" Oct 30 09:26:19.122763 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:19.122736 4562 scope.go:122] "RemoveContainer" containerID="bce982b30d40131ba088ef0c2518e3b3d42db0314ab6426dde722029d794bcd8" Oct 30 09:26:19.122979 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:26:19.122960 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bce982b30d40131ba088ef0c2518e3b3d42db0314ab6426dde722029d794bcd8\": container with ID starting with bce982b30d40131ba088ef0c2518e3b3d42db0314ab6426dde722029d794bcd8 not found: ID does not exist" containerID="bce982b30d40131ba088ef0c2518e3b3d42db0314ab6426dde722029d794bcd8" Oct 30 09:26:19.123043 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:19.122988 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bce982b30d40131ba088ef0c2518e3b3d42db0314ab6426dde722029d794bcd8"} err="failed to get container status \"bce982b30d40131ba088ef0c2518e3b3d42db0314ab6426dde722029d794bcd8\": rpc error: code = NotFound desc = could not find container \"bce982b30d40131ba088ef0c2518e3b3d42db0314ab6426dde722029d794bcd8\": container with ID starting with bce982b30d40131ba088ef0c2518e3b3d42db0314ab6426dde722029d794bcd8 not found: ID does not exist" Oct 30 09:26:19.156905 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:19.156879 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f"] Oct 30 09:26:19.170447 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:19.170427 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-962e2e-openshift-default-746cc469ff-vfd8f"] Oct 30 09:26:19.877089 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:19.877056 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="286eb740-46f4-44ba-80f1-0de69bd89861" path="/var/lib/kubelet/pods/286eb740-46f4-44ba-80f1-0de69bd89861/volumes" Oct 30 09:26:21.494368 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:21.494334 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4fc2e2-84855487c8-46cpz"] Oct 30 09:26:21.667189 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:21.667151 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9371b7-66bbf587b-8tg85"] Oct 30 09:26:21.667459 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:21.667409 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9371b7-66bbf587b-8tg85" podUID="f1aa98b2-58f0-442b-9591-0f00f8a9929c" containerName="maas-api" containerID="cri-o://5c0f67c0f22caa4269d1f535169418e984f291d4d09b40c5b1aa98a1a2b28c72" gracePeriod=30 Oct 30 09:26:22.106659 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:22.106585 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9371b7-66bbf587b-8tg85" Oct 30 09:26:22.139917 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:22.139878 4562 generic.go:356] "Generic (PLEG): container finished" podID="f1aa98b2-58f0-442b-9591-0f00f8a9929c" containerID="5c0f67c0f22caa4269d1f535169418e984f291d4d09b40c5b1aa98a1a2b28c72" exitCode=0 Oct 30 09:26:22.140193 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:22.140169 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9371b7-66bbf587b-8tg85" event={"ID":"f1aa98b2-58f0-442b-9591-0f00f8a9929c","Type":"ContainerDied","Data":"5c0f67c0f22caa4269d1f535169418e984f291d4d09b40c5b1aa98a1a2b28c72"} Oct 30 09:26:22.140427 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:22.140403 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9371b7-66bbf587b-8tg85" event={"ID":"f1aa98b2-58f0-442b-9591-0f00f8a9929c","Type":"ContainerDied","Data":"4556039b79d8412e3f9c4dc81b21f985b0a68e5db75fa99c17ead0bd34c77e04"} Oct 30 09:26:22.140892 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:22.140868 4562 scope.go:122] "RemoveContainer" containerID="5c0f67c0f22caa4269d1f535169418e984f291d4d09b40c5b1aa98a1a2b28c72" Oct 30 09:26:22.141254 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:22.141237 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9371b7-66bbf587b-8tg85" Oct 30 09:26:22.170221 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:22.170176 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1aa98b2-58f0-442b-9591-0f00f8a9929c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1aa98b2-58f0-442b-9591-0f00f8a9929c-openshift-service-ca\") pod \"f1aa98b2-58f0-442b-9591-0f00f8a9929c\" (UID: \"f1aa98b2-58f0-442b-9591-0f00f8a9929c\") " Oct 30 09:26:22.175181 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:22.175140 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1aa98b2-58f0-442b-9591-0f00f8a9929c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1aa98b2-58f0-442b-9591-0f00f8a9929c-maas-api-tls\") pod \"f1aa98b2-58f0-442b-9591-0f00f8a9929c\" (UID: \"f1aa98b2-58f0-442b-9591-0f00f8a9929c\") " Oct 30 09:26:22.218406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:22.195411 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1aa98b2-58f0-442b-9591-0f00f8a9929c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1aa98b2-58f0-442b-9591-0f00f8a9929c-metrics-certs\") pod \"f1aa98b2-58f0-442b-9591-0f00f8a9929c\" (UID: \"f1aa98b2-58f0-442b-9591-0f00f8a9929c\") " Oct 30 09:26:22.218406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:22.195567 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1aa98b2-58f0-442b-9591-0f00f8a9929c-kube-api-access-qllz8\" (UniqueName: \"kubernetes.io/projected/f1aa98b2-58f0-442b-9591-0f00f8a9929c-kube-api-access-qllz8\") pod \"f1aa98b2-58f0-442b-9591-0f00f8a9929c\" (UID: \"f1aa98b2-58f0-442b-9591-0f00f8a9929c\") " Oct 30 09:26:22.218406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:22.174976 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1aa98b2-58f0-442b-9591-0f00f8a9929c-openshift-service-ca" pod "f1aa98b2-58f0-442b-9591-0f00f8a9929c" (UID: "f1aa98b2-58f0-442b-9591-0f00f8a9929c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:26:22.246611 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:22.224921 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1aa98b2-58f0-442b-9591-0f00f8a9929c-maas-api-tls" pod "f1aa98b2-58f0-442b-9591-0f00f8a9929c" (UID: "f1aa98b2-58f0-442b-9591-0f00f8a9929c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:26:22.276915 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:22.276876 4562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1aa98b2-58f0-442b-9591-0f00f8a9929c-openshift-service-ca\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:26:22.277119 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:22.277103 4562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1aa98b2-58f0-442b-9591-0f00f8a9929c-maas-api-tls\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:26:22.306902 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:22.306872 4562 scope.go:122] "RemoveContainer" containerID="5c0f67c0f22caa4269d1f535169418e984f291d4d09b40c5b1aa98a1a2b28c72" Oct 30 09:26:22.320950 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:22.320903 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1aa98b2-58f0-442b-9591-0f00f8a9929c-metrics-certs" pod "f1aa98b2-58f0-442b-9591-0f00f8a9929c" (UID: "f1aa98b2-58f0-442b-9591-0f00f8a9929c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:26:22.341295 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:22.340014 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1aa98b2-58f0-442b-9591-0f00f8a9929c-kube-api-access-qllz8" pod "f1aa98b2-58f0-442b-9591-0f00f8a9929c" (UID: "f1aa98b2-58f0-442b-9591-0f00f8a9929c"). InnerVolumeSpecName "kube-api-access-qllz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:26:22.341295 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:26:22.340485 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c0f67c0f22caa4269d1f535169418e984f291d4d09b40c5b1aa98a1a2b28c72\": container with ID starting with 5c0f67c0f22caa4269d1f535169418e984f291d4d09b40c5b1aa98a1a2b28c72 not found: ID does not exist" containerID="5c0f67c0f22caa4269d1f535169418e984f291d4d09b40c5b1aa98a1a2b28c72" Oct 30 09:26:22.341295 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:22.340523 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c0f67c0f22caa4269d1f535169418e984f291d4d09b40c5b1aa98a1a2b28c72"} err="failed to get container status \"5c0f67c0f22caa4269d1f535169418e984f291d4d09b40c5b1aa98a1a2b28c72\": rpc error: code = NotFound desc = could not find container \"5c0f67c0f22caa4269d1f535169418e984f291d4d09b40c5b1aa98a1a2b28c72\": container with ID starting with 5c0f67c0f22caa4269d1f535169418e984f291d4d09b40c5b1aa98a1a2b28c72 not found: ID does not exist" Oct 30 09:26:22.380057 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:22.379961 4562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1aa98b2-58f0-442b-9591-0f00f8a9929c-metrics-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:26:22.380057 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:22.380005 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qllz8\" (UniqueName: \"kubernetes.io/projected/f1aa98b2-58f0-442b-9591-0f00f8a9929c-kube-api-access-qllz8\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:26:22.489466 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:22.489433 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9371b7-66bbf587b-8tg85"] Oct 30 09:26:22.494568 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:22.494539 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9371b7-66bbf587b-8tg85"] Oct 30 09:26:23.974400 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:23.974187 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1aa98b2-58f0-442b-9591-0f00f8a9929c" path="/var/lib/kubelet/pods/f1aa98b2-58f0-442b-9591-0f00f8a9929c/volumes" Oct 30 09:26:26.495217 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:26.495136 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-4fc2e2-84855487c8-46cpz" podUID="343dbb0c-f2d3-41cf-bc2c-283eba43e35d" containerName="payload-processing" containerID="cri-o://339e82d667875decc0cdf6280bbcbecb19a6c7f66e2ea83640557233bbbfef9e" gracePeriod=25 Oct 30 09:26:26.778436 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:26.778403 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4fc2e2-84855487c8-46cpz" Oct 30 09:26:26.846991 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:26.846955 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/343dbb0c-f2d3-41cf-bc2c-283eba43e35d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/343dbb0c-f2d3-41cf-bc2c-283eba43e35d-plugins-config-volume\") pod \"343dbb0c-f2d3-41cf-bc2c-283eba43e35d\" (UID: \"343dbb0c-f2d3-41cf-bc2c-283eba43e35d\") " Oct 30 09:26:26.847180 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:26.847026 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/343dbb0c-f2d3-41cf-bc2c-283eba43e35d-kube-api-access-nb8tj\" (UniqueName: \"kubernetes.io/projected/343dbb0c-f2d3-41cf-bc2c-283eba43e35d-kube-api-access-nb8tj\") pod \"343dbb0c-f2d3-41cf-bc2c-283eba43e35d\" (UID: \"343dbb0c-f2d3-41cf-bc2c-283eba43e35d\") " Oct 30 09:26:26.847476 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:26.847444 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/343dbb0c-f2d3-41cf-bc2c-283eba43e35d-plugins-config-volume" pod "343dbb0c-f2d3-41cf-bc2c-283eba43e35d" (UID: "343dbb0c-f2d3-41cf-bc2c-283eba43e35d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:26:26.847689 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:26.847665 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/343dbb0c-f2d3-41cf-bc2c-283eba43e35d-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:26:26.849707 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:26.849680 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/343dbb0c-f2d3-41cf-bc2c-283eba43e35d-kube-api-access-nb8tj" pod "343dbb0c-f2d3-41cf-bc2c-283eba43e35d" (UID: "343dbb0c-f2d3-41cf-bc2c-283eba43e35d"). InnerVolumeSpecName "kube-api-access-nb8tj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:26:26.948833 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:26.948799 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nb8tj\" (UniqueName: \"kubernetes.io/projected/343dbb0c-f2d3-41cf-bc2c-283eba43e35d-kube-api-access-nb8tj\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:26:27.227672 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:27.227620 4562 generic.go:356] "Generic (PLEG): container finished" podID="343dbb0c-f2d3-41cf-bc2c-283eba43e35d" containerID="339e82d667875decc0cdf6280bbcbecb19a6c7f66e2ea83640557233bbbfef9e" exitCode=0 Oct 30 09:26:27.227877 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:27.227692 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4fc2e2-84855487c8-46cpz" event={"ID":"343dbb0c-f2d3-41cf-bc2c-283eba43e35d","Type":"ContainerDied","Data":"339e82d667875decc0cdf6280bbcbecb19a6c7f66e2ea83640557233bbbfef9e"} Oct 30 09:26:27.227877 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:27.227745 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4fc2e2-84855487c8-46cpz" event={"ID":"343dbb0c-f2d3-41cf-bc2c-283eba43e35d","Type":"ContainerDied","Data":"1d2e740bbddb557b366ef8a1ae13d682cca296e9379fcd4b7f3b7df0e25b47ec"} Oct 30 09:26:27.227877 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:27.227773 4562 scope.go:122] "RemoveContainer" containerID="339e82d667875decc0cdf6280bbcbecb19a6c7f66e2ea83640557233bbbfef9e" Oct 30 09:26:27.228097 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:27.227904 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4fc2e2-84855487c8-46cpz" Oct 30 09:26:27.247207 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:27.247174 4562 scope.go:122] "RemoveContainer" containerID="339e82d667875decc0cdf6280bbcbecb19a6c7f66e2ea83640557233bbbfef9e" Oct 30 09:26:27.248574 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:26:27.248501 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"339e82d667875decc0cdf6280bbcbecb19a6c7f66e2ea83640557233bbbfef9e\": container with ID starting with 339e82d667875decc0cdf6280bbcbecb19a6c7f66e2ea83640557233bbbfef9e not found: ID does not exist" containerID="339e82d667875decc0cdf6280bbcbecb19a6c7f66e2ea83640557233bbbfef9e" Oct 30 09:26:27.248574 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:27.248544 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"339e82d667875decc0cdf6280bbcbecb19a6c7f66e2ea83640557233bbbfef9e"} err="failed to get container status \"339e82d667875decc0cdf6280bbcbecb19a6c7f66e2ea83640557233bbbfef9e\": rpc error: code = NotFound desc = could not find container \"339e82d667875decc0cdf6280bbcbecb19a6c7f66e2ea83640557233bbbfef9e\": container with ID starting with 339e82d667875decc0cdf6280bbcbecb19a6c7f66e2ea83640557233bbbfef9e not found: ID does not exist" Oct 30 09:26:27.285392 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:27.285354 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4fc2e2-84855487c8-46cpz"] Oct 30 09:26:27.288322 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:27.288301 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4fc2e2-84855487c8-46cpz"] Oct 30 09:26:27.874989 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:27.874944 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="343dbb0c-f2d3-41cf-bc2c-283eba43e35d" path="/var/lib/kubelet/pods/343dbb0c-f2d3-41cf-bc2c-283eba43e35d/volumes" Oct 30 09:26:28.597317 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:26:28.597280 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:26:31.427017 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.426978 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-20f71c-645d97fb95-c5wrx"] Oct 30 09:26:31.427796 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.427773 4562 cpu_manager.go:414] "containerMap: removing container" podUID="a104dc46-cd60-483a-b159-4512b460125a" containerName="istio-proxy" Oct 30 09:26:31.427900 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.427797 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a104dc46-cd60-483a-b159-4512b460125a" containerName="istio-proxy" Oct 30 09:26:31.427900 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.427817 4562 cpu_manager.go:414] "containerMap: removing container" podUID="eada6a8f-af88-4279-ae23-1bab2a23be56" containerName="maas-api" Oct 30 09:26:31.427900 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.427826 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eada6a8f-af88-4279-ae23-1bab2a23be56" containerName="maas-api" Oct 30 09:26:31.427900 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.427836 4562 cpu_manager.go:414] "containerMap: removing container" podUID="286eb740-46f4-44ba-80f1-0de69bd89861" containerName="istio-proxy" Oct 30 09:26:31.427900 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.427845 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="286eb740-46f4-44ba-80f1-0de69bd89861" containerName="istio-proxy" Oct 30 09:26:31.427900 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.427863 4562 cpu_manager.go:414] "containerMap: removing container" podUID="343dbb0c-f2d3-41cf-bc2c-283eba43e35d" containerName="payload-processing" Oct 30 09:26:31.427900 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.427870 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="343dbb0c-f2d3-41cf-bc2c-283eba43e35d" containerName="payload-processing" Oct 30 09:26:31.427900 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.427881 4562 cpu_manager.go:414] "containerMap: removing container" podUID="1502d0e9-69e7-4c24-b69c-76ccc9d382f0" containerName="payload-processing" Oct 30 09:26:31.427900 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.427888 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1502d0e9-69e7-4c24-b69c-76ccc9d382f0" containerName="payload-processing" Oct 30 09:26:31.427900 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.427902 4562 cpu_manager.go:414] "containerMap: removing container" podUID="120fcc9d-1535-497e-94ca-827940c4323c" containerName="payload-pre-processing" Oct 30 09:26:31.428404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.427909 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="120fcc9d-1535-497e-94ca-827940c4323c" containerName="payload-pre-processing" Oct 30 09:26:31.428404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.427946 4562 cpu_manager.go:414] "containerMap: removing container" podUID="39375e4b-ce51-48bc-b63f-fee3fd03f3b7" containerName="maas-api" Oct 30 09:26:31.428404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.427955 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39375e4b-ce51-48bc-b63f-fee3fd03f3b7" containerName="maas-api" Oct 30 09:26:31.428404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.427966 4562 cpu_manager.go:414] "containerMap: removing container" podUID="c5aa403a-1de7-40d7-93ef-238dac140203" containerName="maas-api" Oct 30 09:26:31.428404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.427973 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5aa403a-1de7-40d7-93ef-238dac140203" containerName="maas-api" Oct 30 09:26:31.428404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.427986 4562 cpu_manager.go:414] "containerMap: removing container" podUID="7b433eaf-bc2e-4a08-8e7a-527e88ef4c59" containerName="istio-proxy" Oct 30 09:26:31.428404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.427993 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b433eaf-bc2e-4a08-8e7a-527e88ef4c59" containerName="istio-proxy" Oct 30 09:26:31.428404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.428010 4562 cpu_manager.go:414] "containerMap: removing container" podUID="bdfbe6ee-d5c2-48e6-a7d8-a05ea3bc8e01" containerName="payload-pre-processing" Oct 30 09:26:31.428404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.428017 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdfbe6ee-d5c2-48e6-a7d8-a05ea3bc8e01" containerName="payload-pre-processing" Oct 30 09:26:31.428404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.428028 4562 cpu_manager.go:414] "containerMap: removing container" podUID="b0cefc21-e1d4-45ed-b81a-68365ab8e716" containerName="payload-processing" Oct 30 09:26:31.428404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.428038 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0cefc21-e1d4-45ed-b81a-68365ab8e716" containerName="payload-processing" Oct 30 09:26:31.428404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.428051 4562 cpu_manager.go:414] "containerMap: removing container" podUID="7b433eaf-bc2e-4a08-8e7a-527e88ef4c59" containerName="istio-proxy" Oct 30 09:26:31.428404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.428058 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b433eaf-bc2e-4a08-8e7a-527e88ef4c59" containerName="istio-proxy" Oct 30 09:26:31.428404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.428068 4562 cpu_manager.go:414] "containerMap: removing container" podUID="7cb673aa-7113-4265-a9a4-e8e78651f02b" containerName="payload-pre-processing" Oct 30 09:26:31.428404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.428077 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cb673aa-7113-4265-a9a4-e8e78651f02b" containerName="payload-pre-processing" Oct 30 09:26:31.428404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.428095 4562 cpu_manager.go:414] "containerMap: removing container" podUID="f9dfb6a8-2d84-49d2-b5b3-3289259cc7bf" containerName="payload-pre-processing" Oct 30 09:26:31.428404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.428103 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9dfb6a8-2d84-49d2-b5b3-3289259cc7bf" containerName="payload-pre-processing" Oct 30 09:26:31.428404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.428117 4562 cpu_manager.go:414] "containerMap: removing container" podUID="286eb740-46f4-44ba-80f1-0de69bd89861" containerName="istio-proxy" Oct 30 09:26:31.428404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.428124 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="286eb740-46f4-44ba-80f1-0de69bd89861" containerName="istio-proxy" Oct 30 09:26:31.428404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.428144 4562 cpu_manager.go:414] "containerMap: removing container" podUID="f0b5325c-a526-4cbe-9d70-2dd858185eb5" containerName="maas-api" Oct 30 09:26:31.428404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.428151 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0b5325c-a526-4cbe-9d70-2dd858185eb5" containerName="maas-api" Oct 30 09:26:31.428404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.428165 4562 cpu_manager.go:414] "containerMap: removing container" podUID="f1aa98b2-58f0-442b-9591-0f00f8a9929c" containerName="maas-api" Oct 30 09:26:31.428404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.428172 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1aa98b2-58f0-442b-9591-0f00f8a9929c" containerName="maas-api" Oct 30 09:26:31.428404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.428187 4562 cpu_manager.go:414] "containerMap: removing container" podUID="dacc132b-fcda-49a6-a569-4696ea206638" containerName="payload-processing" Oct 30 09:26:31.428404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.428194 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dacc132b-fcda-49a6-a569-4696ea206638" containerName="payload-processing" Oct 30 09:26:31.428404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.428334 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9dfb6a8-2d84-49d2-b5b3-3289259cc7bf" containerName="payload-pre-processing" Oct 30 09:26:31.428404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.428348 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="286eb740-46f4-44ba-80f1-0de69bd89861" containerName="istio-proxy" Oct 30 09:26:31.428404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.428360 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="a104dc46-cd60-483a-b159-4512b460125a" containerName="istio-proxy" Oct 30 09:26:31.428404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.428369 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0b5325c-a526-4cbe-9d70-2dd858185eb5" containerName="maas-api" Oct 30 09:26:31.428404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.428393 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1aa98b2-58f0-442b-9591-0f00f8a9929c" containerName="maas-api" Oct 30 09:26:31.428404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.428402 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="dacc132b-fcda-49a6-a569-4696ea206638" containerName="payload-processing" Oct 30 09:26:31.428404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.428415 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="a104dc46-cd60-483a-b159-4512b460125a" containerName="istio-proxy" Oct 30 09:26:31.428404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.428424 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="eada6a8f-af88-4279-ae23-1bab2a23be56" containerName="maas-api" Oct 30 09:26:31.430303 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.428433 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="286eb740-46f4-44ba-80f1-0de69bd89861" containerName="istio-proxy" Oct 30 09:26:31.430303 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.428447 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="343dbb0c-f2d3-41cf-bc2c-283eba43e35d" containerName="payload-processing" Oct 30 09:26:31.430303 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.428459 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="1502d0e9-69e7-4c24-b69c-76ccc9d382f0" containerName="payload-processing" Oct 30 09:26:31.430303 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.428468 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="120fcc9d-1535-497e-94ca-827940c4323c" containerName="payload-pre-processing" Oct 30 09:26:31.430303 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.428478 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="39375e4b-ce51-48bc-b63f-fee3fd03f3b7" containerName="maas-api" Oct 30 09:26:31.430303 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.428488 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5aa403a-1de7-40d7-93ef-238dac140203" containerName="maas-api" Oct 30 09:26:31.430303 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.428498 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b433eaf-bc2e-4a08-8e7a-527e88ef4c59" containerName="istio-proxy" Oct 30 09:26:31.430303 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.428508 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdfbe6ee-d5c2-48e6-a7d8-a05ea3bc8e01" containerName="payload-pre-processing" Oct 30 09:26:31.430303 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.428517 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0cefc21-e1d4-45ed-b81a-68365ab8e716" containerName="payload-processing" Oct 30 09:26:31.430303 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.428527 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b433eaf-bc2e-4a08-8e7a-527e88ef4c59" containerName="istio-proxy" Oct 30 09:26:31.430303 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.428537 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cb673aa-7113-4265-a9a4-e8e78651f02b" containerName="payload-pre-processing" Oct 30 09:26:31.433616 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.433595 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-20f71c-645d97fb95-c5wrx" Oct 30 09:26:31.448651 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.448627 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-20f71c-645d97fb95-c5wrx"] Oct 30 09:26:31.458135 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.458111 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-20f71c\"" Oct 30 09:26:31.568405 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.551291 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd2hm\" (UniqueName: \"kubernetes.io/projected/3b314f31-c731-467a-bee2-79f08cbc0573-kube-api-access-sd2hm\") pod \"payload-pre-processing-e2e-praxis-a-20f71c-645d97fb95-c5wrx\" (UID: \"3b314f31-c731-467a-bee2-79f08cbc0573\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-20f71c-645d97fb95-c5wrx" Oct 30 09:26:31.570031 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.569996 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b314f31-c731-467a-bee2-79f08cbc0573-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-20f71c-645d97fb95-c5wrx\" (UID: \"3b314f31-c731-467a-bee2-79f08cbc0573\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-20f71c-645d97fb95-c5wrx" Oct 30 09:26:31.579091 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.579057 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-20f71c-dockercfg-fdll7\"" Oct 30 09:26:31.696466 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.690440 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-20f71c-6988847554-ld88l"] Oct 30 09:26:31.696466 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.691368 4562 cpu_manager.go:414] "containerMap: removing container" podUID="a104dc46-cd60-483a-b159-4512b460125a" containerName="istio-proxy" Oct 30 09:26:31.696466 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.691405 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a104dc46-cd60-483a-b159-4512b460125a" containerName="istio-proxy" Oct 30 09:26:31.724680 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.724659 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-20f71c-6988847554-ld88l"] Oct 30 09:26:31.724804 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.724785 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-20f71c-6988847554-ld88l" Oct 30 09:26:31.740430 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.740396 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd2hm\" (UniqueName: \"kubernetes.io/projected/3b314f31-c731-467a-bee2-79f08cbc0573-kube-api-access-sd2hm\") pod \"payload-pre-processing-e2e-praxis-a-20f71c-645d97fb95-c5wrx\" (UID: \"3b314f31-c731-467a-bee2-79f08cbc0573\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-20f71c-645d97fb95-c5wrx" Oct 30 09:26:31.741155 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.741131 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b314f31-c731-467a-bee2-79f08cbc0573-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-20f71c-645d97fb95-c5wrx\" (UID: \"3b314f31-c731-467a-bee2-79f08cbc0573\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-20f71c-645d97fb95-c5wrx" Oct 30 09:26:31.742562 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.742227 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b314f31-c731-467a-bee2-79f08cbc0573-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-20f71c-645d97fb95-c5wrx\" (UID: \"3b314f31-c731-467a-bee2-79f08cbc0573\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-20f71c-645d97fb95-c5wrx" Oct 30 09:26:31.761650 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.761620 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd2hm\" (UniqueName: \"kubernetes.io/projected/3b314f31-c731-467a-bee2-79f08cbc0573-kube-api-access-sd2hm\") pod \"payload-pre-processing-e2e-praxis-a-20f71c-645d97fb95-c5wrx\" (UID: \"3b314f31-c731-467a-bee2-79f08cbc0573\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-20f71c-645d97fb95-c5wrx" Oct 30 09:26:31.843364 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.843327 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e40a8439-563f-46dc-b272-17c688df5b6f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-20f71c-6988847554-ld88l\" (UID: \"e40a8439-563f-46dc-b272-17c688df5b6f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-20f71c-6988847554-ld88l" Oct 30 09:26:31.843546 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.843479 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25g5g\" (UniqueName: \"kubernetes.io/projected/e40a8439-563f-46dc-b272-17c688df5b6f-kube-api-access-25g5g\") pod \"payload-processing-e2e-praxis-a-20f71c-6988847554-ld88l\" (UID: \"e40a8439-563f-46dc-b272-17c688df5b6f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-20f71c-6988847554-ld88l" Oct 30 09:26:31.944671 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.944640 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25g5g\" (UniqueName: \"kubernetes.io/projected/e40a8439-563f-46dc-b272-17c688df5b6f-kube-api-access-25g5g\") pod \"payload-processing-e2e-praxis-a-20f71c-6988847554-ld88l\" (UID: \"e40a8439-563f-46dc-b272-17c688df5b6f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-20f71c-6988847554-ld88l" Oct 30 09:26:31.944843 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.944798 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e40a8439-563f-46dc-b272-17c688df5b6f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-20f71c-6988847554-ld88l\" (UID: \"e40a8439-563f-46dc-b272-17c688df5b6f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-20f71c-6988847554-ld88l" Oct 30 09:26:31.945537 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.945517 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e40a8439-563f-46dc-b272-17c688df5b6f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-20f71c-6988847554-ld88l\" (UID: \"e40a8439-563f-46dc-b272-17c688df5b6f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-20f71c-6988847554-ld88l" Oct 30 09:26:31.965097 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:31.965029 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25g5g\" (UniqueName: \"kubernetes.io/projected/e40a8439-563f-46dc-b272-17c688df5b6f-kube-api-access-25g5g\") pod \"payload-processing-e2e-praxis-a-20f71c-6988847554-ld88l\" (UID: \"e40a8439-563f-46dc-b272-17c688df5b6f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-20f71c-6988847554-ld88l" Oct 30 09:26:32.048098 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:32.048073 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-20f71c-645d97fb95-c5wrx" Oct 30 09:26:32.056353 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:32.056322 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-20f71c-6988847554-ld88l" Oct 30 09:26:32.383658 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:26:32.383627 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b314f31_c731_467a_bee2_79f08cbc0573.slice/crio-3599241cf86aa7965ab7fda17040d47e653dfefe9879522f4966677ed1f79314 WatchSource:0}: Error finding container 3599241cf86aa7965ab7fda17040d47e653dfefe9879522f4966677ed1f79314: Status 404 returned error can't find the container with id 3599241cf86aa7965ab7fda17040d47e653dfefe9879522f4966677ed1f79314 Oct 30 09:26:32.384596 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:32.384574 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-20f71c-645d97fb95-c5wrx"] Oct 30 09:26:32.407740 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:26:32.407715 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode40a8439_563f_46dc_b272_17c688df5b6f.slice/crio-62ecd3d47996d7fc058b222c6fb7b108ebb9e6fa517c28aa8d0654f81857fb84 WatchSource:0}: Error finding container 62ecd3d47996d7fc058b222c6fb7b108ebb9e6fa517c28aa8d0654f81857fb84: Status 404 returned error can't find the container with id 62ecd3d47996d7fc058b222c6fb7b108ebb9e6fa517c28aa8d0654f81857fb84 Oct 30 09:26:32.409834 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:32.409812 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-20f71c-6988847554-ld88l"] Oct 30 09:26:33.358319 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:33.358275 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-20f71c-645d97fb95-c5wrx" event={"ID":"3b314f31-c731-467a-bee2-79f08cbc0573","Type":"ContainerStarted","Data":"d50285693133789a0fbcacd42560181b256653e3e077924c8bfc2dab3c628a3f"} Oct 30 09:26:33.358792 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:33.358347 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-20f71c-645d97fb95-c5wrx" event={"ID":"3b314f31-c731-467a-bee2-79f08cbc0573","Type":"ContainerStarted","Data":"3599241cf86aa7965ab7fda17040d47e653dfefe9879522f4966677ed1f79314"} Oct 30 09:26:33.358792 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:33.358376 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-20f71c-645d97fb95-c5wrx" Oct 30 09:26:33.359964 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:33.359935 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-20f71c-6988847554-ld88l" event={"ID":"e40a8439-563f-46dc-b272-17c688df5b6f","Type":"ContainerStarted","Data":"d7ba8a16dfe84c056a59d5ff0a96d4a20965d819ea915153d09529c996e585c0"} Oct 30 09:26:33.360083 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:33.359981 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-20f71c-6988847554-ld88l" event={"ID":"e40a8439-563f-46dc-b272-17c688df5b6f","Type":"ContainerStarted","Data":"62ecd3d47996d7fc058b222c6fb7b108ebb9e6fa517c28aa8d0654f81857fb84"} Oct 30 09:26:33.360588 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:33.360562 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-20f71c-6988847554-ld88l" Oct 30 09:26:33.399208 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:33.399164 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-20f71c-645d97fb95-c5wrx" podStartSLOduration=2.399150378 podStartE2EDuration="2.399150378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:26:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:26:33.397299941 +0000 UTC m=+1953.658860929" watchObservedRunningTime="2026-10-30 09:26:33.399150378 +0000 UTC m=+1953.660711391" Oct 30 09:26:33.438471 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:33.438422 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-20f71c-6988847554-ld88l" podStartSLOduration=2.438408913 podStartE2EDuration="2.438408913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:26:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:26:33.436033569 +0000 UTC m=+1953.697594562" watchObservedRunningTime="2026-10-30 09:26:33.438408913 +0000 UTC m=+1953.699969917" Oct 30 09:26:36.506076 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:36.506036 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9d39c0-6f7c96b4cf-g5vh7"] Oct 30 09:26:36.510499 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:36.510477 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9d39c0-6f7c96b4cf-g5vh7" Oct 30 09:26:36.526402 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:36.526361 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9d39c0-6f7c96b4cf-g5vh7"] Oct 30 09:26:36.534938 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:36.534916 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-9d39c0\"" Oct 30 09:26:36.535214 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:36.535195 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-9d39c0-dockercfg-z57b9\"" Oct 30 09:26:36.612403 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:36.577967 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxnr6\" (UniqueName: \"kubernetes.io/projected/990bc616-4583-4bd0-b06b-e7f26e4cfa56-kube-api-access-lxnr6\") pod \"payload-pre-processing-e2e-ipp-a-9d39c0-6f7c96b4cf-g5vh7\" (UID: \"990bc616-4583-4bd0-b06b-e7f26e4cfa56\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9d39c0-6f7c96b4cf-g5vh7" Oct 30 09:26:36.669405 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:36.658808 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/990bc616-4583-4bd0-b06b-e7f26e4cfa56-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9d39c0-6f7c96b4cf-g5vh7\" (UID: \"990bc616-4583-4bd0-b06b-e7f26e4cfa56\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9d39c0-6f7c96b4cf-g5vh7" Oct 30 09:26:36.806613 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:36.802847 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/990bc616-4583-4bd0-b06b-e7f26e4cfa56-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9d39c0-6f7c96b4cf-g5vh7\" (UID: \"990bc616-4583-4bd0-b06b-e7f26e4cfa56\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9d39c0-6f7c96b4cf-g5vh7" Oct 30 09:26:36.806613 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:36.805750 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxnr6\" (UniqueName: \"kubernetes.io/projected/990bc616-4583-4bd0-b06b-e7f26e4cfa56-kube-api-access-lxnr6\") pod \"payload-pre-processing-e2e-ipp-a-9d39c0-6f7c96b4cf-g5vh7\" (UID: \"990bc616-4583-4bd0-b06b-e7f26e4cfa56\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9d39c0-6f7c96b4cf-g5vh7" Oct 30 09:26:36.806613 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:36.805500 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9d39c0-5df65f6dd4-t7xwh"] Oct 30 09:26:36.806613 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:36.804609 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/990bc616-4583-4bd0-b06b-e7f26e4cfa56-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9d39c0-6f7c96b4cf-g5vh7\" (UID: \"990bc616-4583-4bd0-b06b-e7f26e4cfa56\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9d39c0-6f7c96b4cf-g5vh7" Oct 30 09:26:36.812252 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:36.812228 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9d39c0-5df65f6dd4-t7xwh" Oct 30 09:26:36.815485 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:36.815331 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9d39c0-5df65f6dd4-t7xwh"] Oct 30 09:26:36.827000 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:36.826975 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxnr6\" (UniqueName: \"kubernetes.io/projected/990bc616-4583-4bd0-b06b-e7f26e4cfa56-kube-api-access-lxnr6\") pod \"payload-pre-processing-e2e-ipp-a-9d39c0-6f7c96b4cf-g5vh7\" (UID: \"990bc616-4583-4bd0-b06b-e7f26e4cfa56\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9d39c0-6f7c96b4cf-g5vh7" Oct 30 09:26:36.907188 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:36.907157 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdl5c\" (UniqueName: \"kubernetes.io/projected/30f0415e-6d4f-41e5-8cdd-3cebb8ef2c1a-kube-api-access-mdl5c\") pod \"payload-processing-e2e-ipp-a-9d39c0-5df65f6dd4-t7xwh\" (UID: \"30f0415e-6d4f-41e5-8cdd-3cebb8ef2c1a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9d39c0-5df65f6dd4-t7xwh" Oct 30 09:26:36.907666 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:36.907640 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30f0415e-6d4f-41e5-8cdd-3cebb8ef2c1a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9d39c0-5df65f6dd4-t7xwh\" (UID: \"30f0415e-6d4f-41e5-8cdd-3cebb8ef2c1a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9d39c0-5df65f6dd4-t7xwh" Oct 30 09:26:37.009070 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:37.009032 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30f0415e-6d4f-41e5-8cdd-3cebb8ef2c1a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9d39c0-5df65f6dd4-t7xwh\" (UID: \"30f0415e-6d4f-41e5-8cdd-3cebb8ef2c1a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9d39c0-5df65f6dd4-t7xwh" Oct 30 09:26:37.009235 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:37.009117 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdl5c\" (UniqueName: \"kubernetes.io/projected/30f0415e-6d4f-41e5-8cdd-3cebb8ef2c1a-kube-api-access-mdl5c\") pod \"payload-processing-e2e-ipp-a-9d39c0-5df65f6dd4-t7xwh\" (UID: \"30f0415e-6d4f-41e5-8cdd-3cebb8ef2c1a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9d39c0-5df65f6dd4-t7xwh" Oct 30 09:26:37.009815 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:37.009790 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30f0415e-6d4f-41e5-8cdd-3cebb8ef2c1a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9d39c0-5df65f6dd4-t7xwh\" (UID: \"30f0415e-6d4f-41e5-8cdd-3cebb8ef2c1a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9d39c0-5df65f6dd4-t7xwh" Oct 30 09:26:37.029829 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:37.029798 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdl5c\" (UniqueName: \"kubernetes.io/projected/30f0415e-6d4f-41e5-8cdd-3cebb8ef2c1a-kube-api-access-mdl5c\") pod \"payload-processing-e2e-ipp-a-9d39c0-5df65f6dd4-t7xwh\" (UID: \"30f0415e-6d4f-41e5-8cdd-3cebb8ef2c1a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9d39c0-5df65f6dd4-t7xwh" Oct 30 09:26:37.125696 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:37.125612 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9d39c0-6f7c96b4cf-g5vh7" Oct 30 09:26:37.154986 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:37.154917 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9d39c0-5df65f6dd4-t7xwh" Oct 30 09:26:37.430070 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:26:37.430028 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod990bc616_4583_4bd0_b06b_e7f26e4cfa56.slice/crio-3c2314caa55369e331f3afffac64750d3e7999e7ea3ea7f692721a0cb5e68d5b WatchSource:0}: Error finding container 3c2314caa55369e331f3afffac64750d3e7999e7ea3ea7f692721a0cb5e68d5b: Status 404 returned error can't find the container with id 3c2314caa55369e331f3afffac64750d3e7999e7ea3ea7f692721a0cb5e68d5b Oct 30 09:26:37.432412 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:37.432376 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9d39c0-6f7c96b4cf-g5vh7"] Oct 30 09:26:37.479285 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:26:37.479245 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30f0415e_6d4f_41e5_8cdd_3cebb8ef2c1a.slice/crio-3fa578dd2f3e362cf12923c741c33e967b2c4a8e0d37b9f7aa9d6721c1ba834a WatchSource:0}: Error finding container 3fa578dd2f3e362cf12923c741c33e967b2c4a8e0d37b9f7aa9d6721c1ba834a: Status 404 returned error can't find the container with id 3fa578dd2f3e362cf12923c741c33e967b2c4a8e0d37b9f7aa9d6721c1ba834a Oct 30 09:26:37.482955 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:37.482925 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9d39c0-5df65f6dd4-t7xwh"] Oct 30 09:26:38.391287 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:38.391247 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9d39c0-5df65f6dd4-t7xwh" event={"ID":"30f0415e-6d4f-41e5-8cdd-3cebb8ef2c1a","Type":"ContainerStarted","Data":"9daa984fbf64db83fbaa7fb5b954a4684c203937850866cf9f3e2bfa4767cead"} Oct 30 09:26:38.391753 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:38.391304 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9d39c0-5df65f6dd4-t7xwh" event={"ID":"30f0415e-6d4f-41e5-8cdd-3cebb8ef2c1a","Type":"ContainerStarted","Data":"3fa578dd2f3e362cf12923c741c33e967b2c4a8e0d37b9f7aa9d6721c1ba834a"} Oct 30 09:26:38.391753 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:38.391375 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-9d39c0-5df65f6dd4-t7xwh" Oct 30 09:26:38.392900 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:38.392877 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9d39c0-6f7c96b4cf-g5vh7" event={"ID":"990bc616-4583-4bd0-b06b-e7f26e4cfa56","Type":"ContainerStarted","Data":"89047819dee7ea7096403efaacef829e3f44ddb9f056ab68a24560e77b5a5bce"} Oct 30 09:26:38.393022 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:38.392916 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9d39c0-6f7c96b4cf-g5vh7" event={"ID":"990bc616-4583-4bd0-b06b-e7f26e4cfa56","Type":"ContainerStarted","Data":"3c2314caa55369e331f3afffac64750d3e7999e7ea3ea7f692721a0cb5e68d5b"} Oct 30 09:26:38.393022 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:38.392993 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9d39c0-6f7c96b4cf-g5vh7" Oct 30 09:26:38.441204 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:38.441156 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-9d39c0-5df65f6dd4-t7xwh" podStartSLOduration=2.441143107 podStartE2EDuration="2.441143107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:26:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:26:38.439029271 +0000 UTC m=+1958.700590282" watchObservedRunningTime="2026-10-30 09:26:38.441143107 +0000 UTC m=+1958.702704096" Oct 30 09:26:38.482885 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:38.482834 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9d39c0-6f7c96b4cf-g5vh7" podStartSLOduration=2.482822755 podStartE2EDuration="2.482822755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:26:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:26:38.479629092 +0000 UTC m=+1958.741190106" watchObservedRunningTime="2026-10-30 09:26:38.482822755 +0000 UTC m=+1958.744383769" Oct 30 09:26:44.369803 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:44.369775 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-20f71c-645d97fb95-c5wrx" Oct 30 09:26:45.428468 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:45.422603 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-20f71c-6988847554-ld88l" Oct 30 09:26:46.279913 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:46.279613 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4fc2e2-66476768fb-rzqx7"] Oct 30 09:26:46.279913 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:46.279832 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4fc2e2-66476768fb-rzqx7" podUID="b5a258ce-a81b-4605-8e83-ad8a63d4bc9c" containerName="maas-api" containerID="cri-o://baabe2cdd3b6860090cfe8e0fe1dd664dcd6423d0e2bedd41237e863c3e09f9d" gracePeriod=30 Oct 30 09:26:46.517110 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:46.517073 4562 generic.go:356] "Generic (PLEG): container finished" podID="b5a258ce-a81b-4605-8e83-ad8a63d4bc9c" containerID="baabe2cdd3b6860090cfe8e0fe1dd664dcd6423d0e2bedd41237e863c3e09f9d" exitCode=0 Oct 30 09:26:46.517518 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:46.517143 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4fc2e2-66476768fb-rzqx7" event={"ID":"b5a258ce-a81b-4605-8e83-ad8a63d4bc9c","Type":"ContainerDied","Data":"baabe2cdd3b6860090cfe8e0fe1dd664dcd6423d0e2bedd41237e863c3e09f9d"} Oct 30 09:26:46.830629 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:46.830593 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4fc2e2-66476768fb-rzqx7" Oct 30 09:26:46.893353 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:46.893312 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5a258ce-a81b-4605-8e83-ad8a63d4bc9c-kube-api-access-crkfh\" (UniqueName: \"kubernetes.io/projected/b5a258ce-a81b-4605-8e83-ad8a63d4bc9c-kube-api-access-crkfh\") pod \"b5a258ce-a81b-4605-8e83-ad8a63d4bc9c\" (UID: \"b5a258ce-a81b-4605-8e83-ad8a63d4bc9c\") " Oct 30 09:26:46.893667 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:46.893643 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5a258ce-a81b-4605-8e83-ad8a63d4bc9c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5a258ce-a81b-4605-8e83-ad8a63d4bc9c-openshift-service-ca\") pod \"b5a258ce-a81b-4605-8e83-ad8a63d4bc9c\" (UID: \"b5a258ce-a81b-4605-8e83-ad8a63d4bc9c\") " Oct 30 09:26:46.893935 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:46.893914 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5a258ce-a81b-4605-8e83-ad8a63d4bc9c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5a258ce-a81b-4605-8e83-ad8a63d4bc9c-metrics-certs\") pod \"b5a258ce-a81b-4605-8e83-ad8a63d4bc9c\" (UID: \"b5a258ce-a81b-4605-8e83-ad8a63d4bc9c\") " Oct 30 09:26:46.894087 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:46.894069 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5a258ce-a81b-4605-8e83-ad8a63d4bc9c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5a258ce-a81b-4605-8e83-ad8a63d4bc9c-maas-api-tls\") pod \"b5a258ce-a81b-4605-8e83-ad8a63d4bc9c\" (UID: \"b5a258ce-a81b-4605-8e83-ad8a63d4bc9c\") " Oct 30 09:26:46.896910 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:46.896034 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5a258ce-a81b-4605-8e83-ad8a63d4bc9c-openshift-service-ca" pod "b5a258ce-a81b-4605-8e83-ad8a63d4bc9c" (UID: "b5a258ce-a81b-4605-8e83-ad8a63d4bc9c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:26:46.919415 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:46.917295 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5a258ce-a81b-4605-8e83-ad8a63d4bc9c-kube-api-access-crkfh" pod "b5a258ce-a81b-4605-8e83-ad8a63d4bc9c" (UID: "b5a258ce-a81b-4605-8e83-ad8a63d4bc9c"). InnerVolumeSpecName "kube-api-access-crkfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:26:46.919415 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:46.918919 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5a258ce-a81b-4605-8e83-ad8a63d4bc9c-metrics-certs" pod "b5a258ce-a81b-4605-8e83-ad8a63d4bc9c" (UID: "b5a258ce-a81b-4605-8e83-ad8a63d4bc9c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:26:46.921147 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:46.920779 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5a258ce-a81b-4605-8e83-ad8a63d4bc9c-maas-api-tls" pod "b5a258ce-a81b-4605-8e83-ad8a63d4bc9c" (UID: "b5a258ce-a81b-4605-8e83-ad8a63d4bc9c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:26:47.001231 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:47.001124 4562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5a258ce-a81b-4605-8e83-ad8a63d4bc9c-metrics-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:26:47.001231 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:47.001165 4562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5a258ce-a81b-4605-8e83-ad8a63d4bc9c-maas-api-tls\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:26:47.001231 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:47.001179 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crkfh\" (UniqueName: \"kubernetes.io/projected/b5a258ce-a81b-4605-8e83-ad8a63d4bc9c-kube-api-access-crkfh\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:26:47.001231 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:47.001192 4562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5a258ce-a81b-4605-8e83-ad8a63d4bc9c-openshift-service-ca\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:26:47.535889 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:47.535750 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4fc2e2-66476768fb-rzqx7" event={"ID":"b5a258ce-a81b-4605-8e83-ad8a63d4bc9c","Type":"ContainerDied","Data":"00e2d38efa5218755bc4037daba6ee331ea91bb8303fafb2b80bae9c78b9b6d0"} Oct 30 09:26:47.535889 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:47.535875 4562 scope.go:122] "RemoveContainer" containerID="baabe2cdd3b6860090cfe8e0fe1dd664dcd6423d0e2bedd41237e863c3e09f9d" Oct 30 09:26:47.536548 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:47.536036 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4fc2e2-66476768fb-rzqx7" Oct 30 09:26:47.595927 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:47.595837 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4fc2e2-66476768fb-rzqx7"] Oct 30 09:26:47.608672 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:47.608640 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4fc2e2-66476768fb-rzqx7"] Oct 30 09:26:48.255297 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:48.254850 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5a258ce-a81b-4605-8e83-ad8a63d4bc9c" path="/var/lib/kubelet/pods/b5a258ce-a81b-4605-8e83-ad8a63d4bc9c/volumes" Oct 30 09:26:49.402367 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:49.402336 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9d39c0-6f7c96b4cf-g5vh7" Oct 30 09:26:49.402831 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:49.402465 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-9d39c0-5df65f6dd4-t7xwh" Oct 30 09:26:50.761205 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:50.761174 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-d95d35-f67b6f879-pdc6f"] Oct 30 09:26:50.761798 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:50.761784 4562 cpu_manager.go:414] "containerMap: removing container" podUID="b5a258ce-a81b-4605-8e83-ad8a63d4bc9c" containerName="maas-api" Oct 30 09:26:50.761862 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:50.761799 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5a258ce-a81b-4605-8e83-ad8a63d4bc9c" containerName="maas-api" Oct 30 09:26:50.761949 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:50.761939 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5a258ce-a81b-4605-8e83-ad8a63d4bc9c" containerName="maas-api" Oct 30 09:26:50.766930 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:50.766911 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d95d35-f67b6f879-pdc6f" Oct 30 09:26:50.781790 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:50.781760 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-d95d35-f67b6f879-pdc6f"] Oct 30 09:26:50.790043 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:50.790009 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-d95d35-dockercfg-vgntm\"" Oct 30 09:26:50.790043 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:50.790034 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-d95d35\"" Oct 30 09:26:50.790217 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:50.790069 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n57ll\" (UniqueName: \"kubernetes.io/projected/14458dfc-296c-47f3-adb5-1b28e26b4314-kube-api-access-n57ll\") pod \"payload-pre-processing-e2e-praxis-b-d95d35-f67b6f879-pdc6f\" (UID: \"14458dfc-296c-47f3-adb5-1b28e26b4314\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d95d35-f67b6f879-pdc6f" Oct 30 09:26:50.790217 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:50.790119 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14458dfc-296c-47f3-adb5-1b28e26b4314-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-d95d35-f67b6f879-pdc6f\" (UID: \"14458dfc-296c-47f3-adb5-1b28e26b4314\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d95d35-f67b6f879-pdc6f" Oct 30 09:26:50.903406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:50.896540 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n57ll\" (UniqueName: \"kubernetes.io/projected/14458dfc-296c-47f3-adb5-1b28e26b4314-kube-api-access-n57ll\") pod \"payload-pre-processing-e2e-praxis-b-d95d35-f67b6f879-pdc6f\" (UID: \"14458dfc-296c-47f3-adb5-1b28e26b4314\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d95d35-f67b6f879-pdc6f" Oct 30 09:26:50.903406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:50.896611 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14458dfc-296c-47f3-adb5-1b28e26b4314-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-d95d35-f67b6f879-pdc6f\" (UID: \"14458dfc-296c-47f3-adb5-1b28e26b4314\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d95d35-f67b6f879-pdc6f" Oct 30 09:26:50.940410 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:50.931698 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14458dfc-296c-47f3-adb5-1b28e26b4314-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-d95d35-f67b6f879-pdc6f\" (UID: \"14458dfc-296c-47f3-adb5-1b28e26b4314\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d95d35-f67b6f879-pdc6f" Oct 30 09:26:51.035568 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:51.027469 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-d95d35-88b87698-5msxx"] Oct 30 09:26:51.070840 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:51.070793 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-d95d35-88b87698-5msxx" Oct 30 09:26:51.087185 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:51.087153 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n57ll\" (UniqueName: \"kubernetes.io/projected/14458dfc-296c-47f3-adb5-1b28e26b4314-kube-api-access-n57ll\") pod \"payload-pre-processing-e2e-praxis-b-d95d35-f67b6f879-pdc6f\" (UID: \"14458dfc-296c-47f3-adb5-1b28e26b4314\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d95d35-f67b6f879-pdc6f" Oct 30 09:26:51.089287 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:51.087794 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-d95d35-88b87698-5msxx"] Oct 30 09:26:51.111208 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:51.091732 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d95d35-f67b6f879-pdc6f" Oct 30 09:26:51.130812 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:51.130776 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49e0fd93-7200-46ea-bcf7-28d1df31d16f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-d95d35-88b87698-5msxx\" (UID: \"49e0fd93-7200-46ea-bcf7-28d1df31d16f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-d95d35-88b87698-5msxx" Oct 30 09:26:51.130929 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:51.130834 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wccwh\" (UniqueName: \"kubernetes.io/projected/49e0fd93-7200-46ea-bcf7-28d1df31d16f-kube-api-access-wccwh\") pod \"payload-processing-e2e-praxis-b-d95d35-88b87698-5msxx\" (UID: \"49e0fd93-7200-46ea-bcf7-28d1df31d16f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-d95d35-88b87698-5msxx" Oct 30 09:26:51.232074 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:51.232034 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49e0fd93-7200-46ea-bcf7-28d1df31d16f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-d95d35-88b87698-5msxx\" (UID: \"49e0fd93-7200-46ea-bcf7-28d1df31d16f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-d95d35-88b87698-5msxx" Oct 30 09:26:51.232259 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:51.232092 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wccwh\" (UniqueName: \"kubernetes.io/projected/49e0fd93-7200-46ea-bcf7-28d1df31d16f-kube-api-access-wccwh\") pod \"payload-processing-e2e-praxis-b-d95d35-88b87698-5msxx\" (UID: \"49e0fd93-7200-46ea-bcf7-28d1df31d16f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-d95d35-88b87698-5msxx" Oct 30 09:26:51.233104 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:51.233075 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49e0fd93-7200-46ea-bcf7-28d1df31d16f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-d95d35-88b87698-5msxx\" (UID: \"49e0fd93-7200-46ea-bcf7-28d1df31d16f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-d95d35-88b87698-5msxx" Oct 30 09:26:51.254610 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:51.254578 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wccwh\" (UniqueName: \"kubernetes.io/projected/49e0fd93-7200-46ea-bcf7-28d1df31d16f-kube-api-access-wccwh\") pod \"payload-processing-e2e-praxis-b-d95d35-88b87698-5msxx\" (UID: \"49e0fd93-7200-46ea-bcf7-28d1df31d16f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-d95d35-88b87698-5msxx" Oct 30 09:26:51.415314 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:26:51.415273 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14458dfc_296c_47f3_adb5_1b28e26b4314.slice/crio-9c0c8070ead25476b8d5095fab9e87ce179d317e983c65906b23e1a29aed1a87 WatchSource:0}: Error finding container 9c0c8070ead25476b8d5095fab9e87ce179d317e983c65906b23e1a29aed1a87: Status 404 returned error can't find the container with id 9c0c8070ead25476b8d5095fab9e87ce179d317e983c65906b23e1a29aed1a87 Oct 30 09:26:51.417040 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:51.417017 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-d95d35-f67b6f879-pdc6f"] Oct 30 09:26:51.418026 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:51.418007 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-d95d35-88b87698-5msxx" Oct 30 09:26:51.568492 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:51.568451 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d95d35-f67b6f879-pdc6f" event={"ID":"14458dfc-296c-47f3-adb5-1b28e26b4314","Type":"ContainerStarted","Data":"5d37a4051953b1894154c20beaa97193242252cae2df88fa2fe95cf2bdd5277a"} Oct 30 09:26:51.568681 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:51.568521 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d95d35-f67b6f879-pdc6f" event={"ID":"14458dfc-296c-47f3-adb5-1b28e26b4314","Type":"ContainerStarted","Data":"9c0c8070ead25476b8d5095fab9e87ce179d317e983c65906b23e1a29aed1a87"} Oct 30 09:26:51.568681 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:51.568589 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d95d35-f67b6f879-pdc6f" Oct 30 09:26:51.629796 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:51.616089 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d95d35-f67b6f879-pdc6f" podStartSLOduration=1.616066895 podStartE2EDuration="1.616066895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:26:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:26:51.61565156 +0000 UTC m=+1971.877212575" watchObservedRunningTime="2026-10-30 09:26:51.616066895 +0000 UTC m=+1971.877627883" Oct 30 09:26:51.779232 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:51.779191 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-d95d35-88b87698-5msxx"] Oct 30 09:26:52.578842 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:52.578805 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-d95d35-88b87698-5msxx" event={"ID":"49e0fd93-7200-46ea-bcf7-28d1df31d16f","Type":"ContainerStarted","Data":"fd9bec4f6b5de7778bbf8df83e5fe7925f4fa7a47539cf14708f5fd81eae04c4"} Oct 30 09:26:52.579153 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:52.578902 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-d95d35-88b87698-5msxx" event={"ID":"49e0fd93-7200-46ea-bcf7-28d1df31d16f","Type":"ContainerStarted","Data":"fa4929553356dc966c383fcade4e080065b1fab98157e606907901c02af0601e"} Oct 30 09:26:52.579153 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:52.579059 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-d95d35-88b87698-5msxx" Oct 30 09:26:52.622515 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:26:52.622460 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-d95d35-88b87698-5msxx" podStartSLOduration=2.622445087 podStartE2EDuration="2.622445087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:26:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:26:52.618900617 +0000 UTC m=+1972.880461605" watchObservedRunningTime="2026-10-30 09:26:52.622445087 +0000 UTC m=+1972.884006074" Oct 30 09:26:58.597890 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:26:58.597845 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:27:02.582784 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:02.582747 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d95d35-f67b6f879-pdc6f" Oct 30 09:27:03.589244 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:03.589210 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-d95d35-88b87698-5msxx" Oct 30 09:27:07.418041 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:07.418002 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-d3de9d-798f5684bd-t5k56"] Oct 30 09:27:07.423878 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:07.423851 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-d3de9d-798f5684bd-t5k56" Oct 30 09:27:07.444288 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:07.442751 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-d3de9d-798f5684bd-t5k56"] Oct 30 09:27:07.488348 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:07.488315 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-d3de9d-6c6b4db5d7-jt9hq"] Oct 30 09:27:07.501145 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:07.501119 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d3de9d-6c6b4db5d7-jt9hq" Oct 30 09:27:07.508044 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:07.507852 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-d3de9d\"" Oct 30 09:27:07.510833 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:07.510811 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-d3de9d-6c6b4db5d7-jt9hq"] Oct 30 09:27:07.561403 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:07.551677 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk5cg\" (UniqueName: \"kubernetes.io/projected/c80a8dfe-e8a6-4adc-bdc6-2019c66bce9a-kube-api-access-xk5cg\") pod \"payload-pre-processing-e2e-praxis-b-d3de9d-6c6b4db5d7-jt9hq\" (UID: \"c80a8dfe-e8a6-4adc-bdc6-2019c66bce9a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d3de9d-6c6b4db5d7-jt9hq" Oct 30 09:27:07.561403 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:07.551794 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2z8r\" (UniqueName: \"kubernetes.io/projected/0d356da3-0606-4dd7-940c-f1ef62ba69f0-kube-api-access-h2z8r\") pod \"payload-processing-e2e-praxis-b-d3de9d-798f5684bd-t5k56\" (UID: \"0d356da3-0606-4dd7-940c-f1ef62ba69f0\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-d3de9d-798f5684bd-t5k56" Oct 30 09:27:07.561403 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:07.551896 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d356da3-0606-4dd7-940c-f1ef62ba69f0-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-d3de9d-798f5684bd-t5k56\" (UID: \"0d356da3-0606-4dd7-940c-f1ef62ba69f0\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-d3de9d-798f5684bd-t5k56" Oct 30 09:27:07.561403 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:07.552124 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c80a8dfe-e8a6-4adc-bdc6-2019c66bce9a-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-d3de9d-6c6b4db5d7-jt9hq\" (UID: \"c80a8dfe-e8a6-4adc-bdc6-2019c66bce9a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d3de9d-6c6b4db5d7-jt9hq" Oct 30 09:27:07.610410 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:07.600713 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-d3de9d-dockercfg-j6lwt\"" Oct 30 09:27:07.693477 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:07.680431 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c80a8dfe-e8a6-4adc-bdc6-2019c66bce9a-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-d3de9d-6c6b4db5d7-jt9hq\" (UID: \"c80a8dfe-e8a6-4adc-bdc6-2019c66bce9a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d3de9d-6c6b4db5d7-jt9hq" Oct 30 09:27:07.693477 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:07.680688 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk5cg\" (UniqueName: \"kubernetes.io/projected/c80a8dfe-e8a6-4adc-bdc6-2019c66bce9a-kube-api-access-xk5cg\") pod \"payload-pre-processing-e2e-praxis-b-d3de9d-6c6b4db5d7-jt9hq\" (UID: \"c80a8dfe-e8a6-4adc-bdc6-2019c66bce9a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d3de9d-6c6b4db5d7-jt9hq" Oct 30 09:27:07.693477 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:07.680775 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2z8r\" (UniqueName: \"kubernetes.io/projected/0d356da3-0606-4dd7-940c-f1ef62ba69f0-kube-api-access-h2z8r\") pod \"payload-processing-e2e-praxis-b-d3de9d-798f5684bd-t5k56\" (UID: \"0d356da3-0606-4dd7-940c-f1ef62ba69f0\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-d3de9d-798f5684bd-t5k56" Oct 30 09:27:07.693477 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:07.680884 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d356da3-0606-4dd7-940c-f1ef62ba69f0-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-d3de9d-798f5684bd-t5k56\" (UID: \"0d356da3-0606-4dd7-940c-f1ef62ba69f0\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-d3de9d-798f5684bd-t5k56" Oct 30 09:27:07.693477 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:07.681906 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d356da3-0606-4dd7-940c-f1ef62ba69f0-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-d3de9d-798f5684bd-t5k56\" (UID: \"0d356da3-0606-4dd7-940c-f1ef62ba69f0\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-d3de9d-798f5684bd-t5k56" Oct 30 09:27:07.744413 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:07.739883 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c80a8dfe-e8a6-4adc-bdc6-2019c66bce9a-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-d3de9d-6c6b4db5d7-jt9hq\" (UID: \"c80a8dfe-e8a6-4adc-bdc6-2019c66bce9a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d3de9d-6c6b4db5d7-jt9hq" Oct 30 09:27:07.827136 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:07.827063 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk5cg\" (UniqueName: \"kubernetes.io/projected/c80a8dfe-e8a6-4adc-bdc6-2019c66bce9a-kube-api-access-xk5cg\") pod \"payload-pre-processing-e2e-praxis-b-d3de9d-6c6b4db5d7-jt9hq\" (UID: \"c80a8dfe-e8a6-4adc-bdc6-2019c66bce9a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d3de9d-6c6b4db5d7-jt9hq" Oct 30 09:27:07.828545 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:07.828518 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2z8r\" (UniqueName: \"kubernetes.io/projected/0d356da3-0606-4dd7-940c-f1ef62ba69f0-kube-api-access-h2z8r\") pod \"payload-processing-e2e-praxis-b-d3de9d-798f5684bd-t5k56\" (UID: \"0d356da3-0606-4dd7-940c-f1ef62ba69f0\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-d3de9d-798f5684bd-t5k56" Oct 30 09:27:08.038287 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:08.038205 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-d3de9d-798f5684bd-t5k56" Oct 30 09:27:08.122863 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:08.122829 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d3de9d-6c6b4db5d7-jt9hq" Oct 30 09:27:08.338303 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:27:08.338273 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d356da3_0606_4dd7_940c_f1ef62ba69f0.slice/crio-7b980fd44c6b64ced85920ec5e04ecf9a1eb2f191eb7c494010865f78aa8ec49 WatchSource:0}: Error finding container 7b980fd44c6b64ced85920ec5e04ecf9a1eb2f191eb7c494010865f78aa8ec49: Status 404 returned error can't find the container with id 7b980fd44c6b64ced85920ec5e04ecf9a1eb2f191eb7c494010865f78aa8ec49 Oct 30 09:27:08.339981 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:08.339958 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-d3de9d-798f5684bd-t5k56"] Oct 30 09:27:08.438818 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:27:08.438774 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc80a8dfe_e8a6_4adc_bdc6_2019c66bce9a.slice/crio-544536cf7bbfc4a1df5c076c7fe6d3331944805155a02b4e0e003e8ee04d4e8b WatchSource:0}: Error finding container 544536cf7bbfc4a1df5c076c7fe6d3331944805155a02b4e0e003e8ee04d4e8b: Status 404 returned error can't find the container with id 544536cf7bbfc4a1df5c076c7fe6d3331944805155a02b4e0e003e8ee04d4e8b Oct 30 09:27:08.440603 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:08.440574 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-d3de9d-6c6b4db5d7-jt9hq"] Oct 30 09:27:08.818891 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:08.818857 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d3de9d-6c6b4db5d7-jt9hq" event={"ID":"c80a8dfe-e8a6-4adc-bdc6-2019c66bce9a","Type":"ContainerStarted","Data":"2bafadeb7f5b8b0c8b2b1bd6dbf2552430a207385e80c64d03a0956159b9eefa"} Oct 30 09:27:08.819081 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:08.818923 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d3de9d-6c6b4db5d7-jt9hq" event={"ID":"c80a8dfe-e8a6-4adc-bdc6-2019c66bce9a","Type":"ContainerStarted","Data":"544536cf7bbfc4a1df5c076c7fe6d3331944805155a02b4e0e003e8ee04d4e8b"} Oct 30 09:27:08.819143 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:08.819079 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d3de9d-6c6b4db5d7-jt9hq" Oct 30 09:27:08.820701 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:08.820672 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-d3de9d-798f5684bd-t5k56" event={"ID":"0d356da3-0606-4dd7-940c-f1ef62ba69f0","Type":"ContainerStarted","Data":"73e35a1e40521bf62ed68e565d4d1ee124b206cb7feaf2ef36dfa59a0b273a8e"} Oct 30 09:27:08.820844 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:08.820713 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-d3de9d-798f5684bd-t5k56" event={"ID":"0d356da3-0606-4dd7-940c-f1ef62ba69f0","Type":"ContainerStarted","Data":"7b980fd44c6b64ced85920ec5e04ecf9a1eb2f191eb7c494010865f78aa8ec49"} Oct 30 09:27:08.820844 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:08.820768 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-d3de9d-798f5684bd-t5k56" Oct 30 09:27:08.870745 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:08.870693 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d3de9d-6c6b4db5d7-jt9hq" podStartSLOduration=1.870680854 podStartE2EDuration="1.870680854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:27:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:27:08.867449284 +0000 UTC m=+1989.129010291" watchObservedRunningTime="2026-10-30 09:27:08.870680854 +0000 UTC m=+1989.132241870" Oct 30 09:27:08.911105 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:08.911060 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-d3de9d-798f5684bd-t5k56" podStartSLOduration=1.911047914 podStartE2EDuration="1.911047914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:27:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:27:08.908566313 +0000 UTC m=+1989.170127312" watchObservedRunningTime="2026-10-30 09:27:08.911047914 +0000 UTC m=+1989.172608901" Oct 30 09:27:09.350500 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:09.350467 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-d95d35-799494fd9b-rxn8v"] Oct 30 09:27:09.354870 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:09.354851 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d95d35-799494fd9b-rxn8v" Oct 30 09:27:09.370488 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:09.370452 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-d95d35-serving-cert\"" Oct 30 09:27:09.374559 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:09.374532 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-d95d35-799494fd9b-rxn8v"] Oct 30 09:27:09.418867 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:09.418830 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n78fl\" (UniqueName: \"kubernetes.io/projected/d2a58d8b-b84a-403f-bd46-5dfe7d3cadec-kube-api-access-n78fl\") pod \"maas-api-e2e-praxis-b-d95d35-799494fd9b-rxn8v\" (UID: \"d2a58d8b-b84a-403f-bd46-5dfe7d3cadec\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d95d35-799494fd9b-rxn8v" Oct 30 09:27:09.419037 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:09.418914 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2a58d8b-b84a-403f-bd46-5dfe7d3cadec-metrics-certs\") pod \"maas-api-e2e-praxis-b-d95d35-799494fd9b-rxn8v\" (UID: \"d2a58d8b-b84a-403f-bd46-5dfe7d3cadec\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d95d35-799494fd9b-rxn8v" Oct 30 09:27:09.419037 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:09.418992 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2a58d8b-b84a-403f-bd46-5dfe7d3cadec-maas-api-tls\") pod \"maas-api-e2e-praxis-b-d95d35-799494fd9b-rxn8v\" (UID: \"d2a58d8b-b84a-403f-bd46-5dfe7d3cadec\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d95d35-799494fd9b-rxn8v" Oct 30 09:27:09.419263 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:09.419242 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2a58d8b-b84a-403f-bd46-5dfe7d3cadec-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-d95d35-799494fd9b-rxn8v\" (UID: \"d2a58d8b-b84a-403f-bd46-5dfe7d3cadec\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d95d35-799494fd9b-rxn8v" Oct 30 09:27:09.520369 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:09.520330 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n78fl\" (UniqueName: \"kubernetes.io/projected/d2a58d8b-b84a-403f-bd46-5dfe7d3cadec-kube-api-access-n78fl\") pod \"maas-api-e2e-praxis-b-d95d35-799494fd9b-rxn8v\" (UID: \"d2a58d8b-b84a-403f-bd46-5dfe7d3cadec\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d95d35-799494fd9b-rxn8v" Oct 30 09:27:09.520764 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:09.520431 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2a58d8b-b84a-403f-bd46-5dfe7d3cadec-metrics-certs\") pod \"maas-api-e2e-praxis-b-d95d35-799494fd9b-rxn8v\" (UID: \"d2a58d8b-b84a-403f-bd46-5dfe7d3cadec\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d95d35-799494fd9b-rxn8v" Oct 30 09:27:09.520764 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:09.520458 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2a58d8b-b84a-403f-bd46-5dfe7d3cadec-maas-api-tls\") pod \"maas-api-e2e-praxis-b-d95d35-799494fd9b-rxn8v\" (UID: \"d2a58d8b-b84a-403f-bd46-5dfe7d3cadec\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d95d35-799494fd9b-rxn8v" Oct 30 09:27:09.520764 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:09.520668 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2a58d8b-b84a-403f-bd46-5dfe7d3cadec-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-d95d35-799494fd9b-rxn8v\" (UID: \"d2a58d8b-b84a-403f-bd46-5dfe7d3cadec\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d95d35-799494fd9b-rxn8v" Oct 30 09:27:09.521297 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:09.521267 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2a58d8b-b84a-403f-bd46-5dfe7d3cadec-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-d95d35-799494fd9b-rxn8v\" (UID: \"d2a58d8b-b84a-403f-bd46-5dfe7d3cadec\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d95d35-799494fd9b-rxn8v" Oct 30 09:27:09.523171 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:09.523146 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2a58d8b-b84a-403f-bd46-5dfe7d3cadec-metrics-certs\") pod \"maas-api-e2e-praxis-b-d95d35-799494fd9b-rxn8v\" (UID: \"d2a58d8b-b84a-403f-bd46-5dfe7d3cadec\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d95d35-799494fd9b-rxn8v" Oct 30 09:27:09.523256 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:09.523148 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2a58d8b-b84a-403f-bd46-5dfe7d3cadec-maas-api-tls\") pod \"maas-api-e2e-praxis-b-d95d35-799494fd9b-rxn8v\" (UID: \"d2a58d8b-b84a-403f-bd46-5dfe7d3cadec\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d95d35-799494fd9b-rxn8v" Oct 30 09:27:09.542397 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:09.542358 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n78fl\" (UniqueName: \"kubernetes.io/projected/d2a58d8b-b84a-403f-bd46-5dfe7d3cadec-kube-api-access-n78fl\") pod \"maas-api-e2e-praxis-b-d95d35-799494fd9b-rxn8v\" (UID: \"d2a58d8b-b84a-403f-bd46-5dfe7d3cadec\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d95d35-799494fd9b-rxn8v" Oct 30 09:27:09.669162 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:09.668984 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d95d35-799494fd9b-rxn8v" Oct 30 09:27:09.984356 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:27:09.984328 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2a58d8b_b84a_403f_bd46_5dfe7d3cadec.slice/crio-b64b313656c063f814b78b5f8c46a0407e81739a4fa79daccd82c994a0b5a171 WatchSource:0}: Error finding container b64b313656c063f814b78b5f8c46a0407e81739a4fa79daccd82c994a0b5a171: Status 404 returned error can't find the container with id b64b313656c063f814b78b5f8c46a0407e81739a4fa79daccd82c994a0b5a171 Oct 30 09:27:09.986021 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:09.985998 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-d95d35-799494fd9b-rxn8v"] Oct 30 09:27:09.986113 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:09.986035 4562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 09:27:10.839306 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:10.839268 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d95d35-799494fd9b-rxn8v" event={"ID":"d2a58d8b-b84a-403f-bd46-5dfe7d3cadec","Type":"ContainerStarted","Data":"6781ba2699c5a77587074d740f1f42847f81c217a42803e1c63b10b4e8890a7e"} Oct 30 09:27:10.839740 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:10.839335 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d95d35-799494fd9b-rxn8v" event={"ID":"d2a58d8b-b84a-403f-bd46-5dfe7d3cadec","Type":"ContainerStarted","Data":"b64b313656c063f814b78b5f8c46a0407e81739a4fa79daccd82c994a0b5a171"} Oct 30 09:27:10.839740 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:10.839407 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d95d35-799494fd9b-rxn8v" Oct 30 09:27:10.886746 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:10.886686 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d95d35-799494fd9b-rxn8v" podStartSLOduration=1.536536999 podStartE2EDuration="1.886667809s" totalImagesPullingTime="350.13081ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:27:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:27:10.885724367 +0000 UTC m=+1991.147285373" watchObservedRunningTime="2026-10-30 09:27:10.886667809 +0000 UTC m=+1991.148228799" Oct 30 09:27:16.879701 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:16.879670 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d95d35-799494fd9b-rxn8v" Oct 30 09:27:19.834261 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:19.834229 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-d3de9d-798f5684bd-t5k56" Oct 30 09:27:19.834701 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:19.834308 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d3de9d-6c6b4db5d7-jt9hq" Oct 30 09:27:21.148278 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:21.148239 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9d39c0-5df65f6dd4-t7xwh"] Oct 30 09:27:21.181872 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:21.181838 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9d39c0-6f7c96b4cf-g5vh7"] Oct 30 09:27:26.149461 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:26.149328 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-9d39c0-5df65f6dd4-t7xwh" podUID="30f0415e-6d4f-41e5-8cdd-3cebb8ef2c1a" containerName="payload-processing" containerID="cri-o://9daa984fbf64db83fbaa7fb5b954a4684c203937850866cf9f3e2bfa4767cead" gracePeriod=25 Oct 30 09:27:26.182428 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:26.182349 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9d39c0-6f7c96b4cf-g5vh7" podUID="990bc616-4583-4bd0-b06b-e7f26e4cfa56" containerName="payload-pre-processing" containerID="cri-o://89047819dee7ea7096403efaacef829e3f44ddb9f056ab68a24560e77b5a5bce" gracePeriod=25 Oct 30 09:27:26.505891 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:26.505862 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9d39c0-6f7c96b4cf-g5vh7" Oct 30 09:27:26.511031 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:26.511010 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9d39c0-5df65f6dd4-t7xwh" Oct 30 09:27:26.621145 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:26.621108 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/990bc616-4583-4bd0-b06b-e7f26e4cfa56-kube-api-access-lxnr6\" (UniqueName: \"kubernetes.io/projected/990bc616-4583-4bd0-b06b-e7f26e4cfa56-kube-api-access-lxnr6\") pod \"990bc616-4583-4bd0-b06b-e7f26e4cfa56\" (UID: \"990bc616-4583-4bd0-b06b-e7f26e4cfa56\") " Oct 30 09:27:26.621449 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:26.621317 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30f0415e-6d4f-41e5-8cdd-3cebb8ef2c1a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30f0415e-6d4f-41e5-8cdd-3cebb8ef2c1a-plugins-config-volume\") pod \"30f0415e-6d4f-41e5-8cdd-3cebb8ef2c1a\" (UID: \"30f0415e-6d4f-41e5-8cdd-3cebb8ef2c1a\") " Oct 30 09:27:26.621568 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:26.621479 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30f0415e-6d4f-41e5-8cdd-3cebb8ef2c1a-kube-api-access-mdl5c\" (UniqueName: \"kubernetes.io/projected/30f0415e-6d4f-41e5-8cdd-3cebb8ef2c1a-kube-api-access-mdl5c\") pod \"30f0415e-6d4f-41e5-8cdd-3cebb8ef2c1a\" (UID: \"30f0415e-6d4f-41e5-8cdd-3cebb8ef2c1a\") " Oct 30 09:27:26.621568 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:26.621521 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/990bc616-4583-4bd0-b06b-e7f26e4cfa56-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/990bc616-4583-4bd0-b06b-e7f26e4cfa56-plugins-config-volume\") pod \"990bc616-4583-4bd0-b06b-e7f26e4cfa56\" (UID: \"990bc616-4583-4bd0-b06b-e7f26e4cfa56\") " Oct 30 09:27:26.622053 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:26.622026 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/990bc616-4583-4bd0-b06b-e7f26e4cfa56-plugins-config-volume" pod "990bc616-4583-4bd0-b06b-e7f26e4cfa56" (UID: "990bc616-4583-4bd0-b06b-e7f26e4cfa56"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:27:26.622516 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:26.622490 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30f0415e-6d4f-41e5-8cdd-3cebb8ef2c1a-plugins-config-volume" pod "30f0415e-6d4f-41e5-8cdd-3cebb8ef2c1a" (UID: "30f0415e-6d4f-41e5-8cdd-3cebb8ef2c1a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:27:26.624240 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:26.624213 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/990bc616-4583-4bd0-b06b-e7f26e4cfa56-kube-api-access-lxnr6" pod "990bc616-4583-4bd0-b06b-e7f26e4cfa56" (UID: "990bc616-4583-4bd0-b06b-e7f26e4cfa56"). InnerVolumeSpecName "kube-api-access-lxnr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:27:26.624524 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:26.624500 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30f0415e-6d4f-41e5-8cdd-3cebb8ef2c1a-kube-api-access-mdl5c" pod "30f0415e-6d4f-41e5-8cdd-3cebb8ef2c1a" (UID: "30f0415e-6d4f-41e5-8cdd-3cebb8ef2c1a"). InnerVolumeSpecName "kube-api-access-mdl5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:27:26.725111 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:26.724954 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdl5c\" (UniqueName: \"kubernetes.io/projected/30f0415e-6d4f-41e5-8cdd-3cebb8ef2c1a-kube-api-access-mdl5c\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:27:26.725111 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:26.724996 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/990bc616-4583-4bd0-b06b-e7f26e4cfa56-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:27:26.725111 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:26.725010 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxnr6\" (UniqueName: \"kubernetes.io/projected/990bc616-4583-4bd0-b06b-e7f26e4cfa56-kube-api-access-lxnr6\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:27:26.725111 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:26.725024 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30f0415e-6d4f-41e5-8cdd-3cebb8ef2c1a-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:27:27.004306 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:27.004222 4562 generic.go:356] "Generic (PLEG): container finished" podID="30f0415e-6d4f-41e5-8cdd-3cebb8ef2c1a" containerID="9daa984fbf64db83fbaa7fb5b954a4684c203937850866cf9f3e2bfa4767cead" exitCode=0 Oct 30 09:27:27.004306 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:27.004268 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9d39c0-5df65f6dd4-t7xwh" Oct 30 09:27:27.004306 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:27.004294 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9d39c0-5df65f6dd4-t7xwh" event={"ID":"30f0415e-6d4f-41e5-8cdd-3cebb8ef2c1a","Type":"ContainerDied","Data":"9daa984fbf64db83fbaa7fb5b954a4684c203937850866cf9f3e2bfa4767cead"} Oct 30 09:27:27.004578 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:27.004336 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9d39c0-5df65f6dd4-t7xwh" event={"ID":"30f0415e-6d4f-41e5-8cdd-3cebb8ef2c1a","Type":"ContainerDied","Data":"3fa578dd2f3e362cf12923c741c33e967b2c4a8e0d37b9f7aa9d6721c1ba834a"} Oct 30 09:27:27.004578 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:27.004358 4562 scope.go:122] "RemoveContainer" containerID="9daa984fbf64db83fbaa7fb5b954a4684c203937850866cf9f3e2bfa4767cead" Oct 30 09:27:27.005824 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:27.005799 4562 generic.go:356] "Generic (PLEG): container finished" podID="990bc616-4583-4bd0-b06b-e7f26e4cfa56" containerID="89047819dee7ea7096403efaacef829e3f44ddb9f056ab68a24560e77b5a5bce" exitCode=0 Oct 30 09:27:27.005957 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:27.005841 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9d39c0-6f7c96b4cf-g5vh7" Oct 30 09:27:27.005957 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:27.005887 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9d39c0-6f7c96b4cf-g5vh7" event={"ID":"990bc616-4583-4bd0-b06b-e7f26e4cfa56","Type":"ContainerDied","Data":"89047819dee7ea7096403efaacef829e3f44ddb9f056ab68a24560e77b5a5bce"} Oct 30 09:27:27.005957 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:27.005930 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9d39c0-6f7c96b4cf-g5vh7" event={"ID":"990bc616-4583-4bd0-b06b-e7f26e4cfa56","Type":"ContainerDied","Data":"3c2314caa55369e331f3afffac64750d3e7999e7ea3ea7f692721a0cb5e68d5b"} Oct 30 09:27:27.015374 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:27.015354 4562 scope.go:122] "RemoveContainer" containerID="9daa984fbf64db83fbaa7fb5b954a4684c203937850866cf9f3e2bfa4767cead" Oct 30 09:27:27.015858 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:27:27.015700 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9daa984fbf64db83fbaa7fb5b954a4684c203937850866cf9f3e2bfa4767cead\": container with ID starting with 9daa984fbf64db83fbaa7fb5b954a4684c203937850866cf9f3e2bfa4767cead not found: ID does not exist" containerID="9daa984fbf64db83fbaa7fb5b954a4684c203937850866cf9f3e2bfa4767cead" Oct 30 09:27:27.015858 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:27.015727 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9daa984fbf64db83fbaa7fb5b954a4684c203937850866cf9f3e2bfa4767cead"} err="failed to get container status \"9daa984fbf64db83fbaa7fb5b954a4684c203937850866cf9f3e2bfa4767cead\": rpc error: code = NotFound desc = could not find container \"9daa984fbf64db83fbaa7fb5b954a4684c203937850866cf9f3e2bfa4767cead\": container with ID starting with 9daa984fbf64db83fbaa7fb5b954a4684c203937850866cf9f3e2bfa4767cead not found: ID does not exist" Oct 30 09:27:27.015858 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:27.015746 4562 scope.go:122] "RemoveContainer" containerID="89047819dee7ea7096403efaacef829e3f44ddb9f056ab68a24560e77b5a5bce" Oct 30 09:27:27.026235 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:27.026216 4562 scope.go:122] "RemoveContainer" containerID="89047819dee7ea7096403efaacef829e3f44ddb9f056ab68a24560e77b5a5bce" Oct 30 09:27:27.026511 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:27:27.026492 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89047819dee7ea7096403efaacef829e3f44ddb9f056ab68a24560e77b5a5bce\": container with ID starting with 89047819dee7ea7096403efaacef829e3f44ddb9f056ab68a24560e77b5a5bce not found: ID does not exist" containerID="89047819dee7ea7096403efaacef829e3f44ddb9f056ab68a24560e77b5a5bce" Oct 30 09:27:27.026585 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:27.026518 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89047819dee7ea7096403efaacef829e3f44ddb9f056ab68a24560e77b5a5bce"} err="failed to get container status \"89047819dee7ea7096403efaacef829e3f44ddb9f056ab68a24560e77b5a5bce\": rpc error: code = NotFound desc = could not find container \"89047819dee7ea7096403efaacef829e3f44ddb9f056ab68a24560e77b5a5bce\": container with ID starting with 89047819dee7ea7096403efaacef829e3f44ddb9f056ab68a24560e77b5a5bce not found: ID does not exist" Oct 30 09:27:27.064999 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:27.064964 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9d39c0-5df65f6dd4-t7xwh"] Oct 30 09:27:27.103017 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:27.102981 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9d39c0-5df65f6dd4-t7xwh"] Oct 30 09:27:27.112912 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:27.112879 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9d39c0-6f7c96b4cf-g5vh7"] Oct 30 09:27:27.118511 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:27.118487 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9d39c0-6f7c96b4cf-g5vh7"] Oct 30 09:27:27.875667 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:27.875633 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30f0415e-6d4f-41e5-8cdd-3cebb8ef2c1a" path="/var/lib/kubelet/pods/30f0415e-6d4f-41e5-8cdd-3cebb8ef2c1a/volumes" Oct 30 09:27:27.876171 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:27.876154 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="990bc616-4583-4bd0-b06b-e7f26e4cfa56" path="/var/lib/kubelet/pods/990bc616-4583-4bd0-b06b-e7f26e4cfa56/volumes" Oct 30 09:27:28.597073 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:27:28.597034 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:27:28.652879 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:28.652842 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-51e091-5f4b488499-clrjg"] Oct 30 09:27:28.653617 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:28.653596 4562 cpu_manager.go:414] "containerMap: removing container" podUID="990bc616-4583-4bd0-b06b-e7f26e4cfa56" containerName="payload-pre-processing" Oct 30 09:27:28.653617 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:28.653617 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="990bc616-4583-4bd0-b06b-e7f26e4cfa56" containerName="payload-pre-processing" Oct 30 09:27:28.653807 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:28.653669 4562 cpu_manager.go:414] "containerMap: removing container" podUID="30f0415e-6d4f-41e5-8cdd-3cebb8ef2c1a" containerName="payload-processing" Oct 30 09:27:28.653807 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:28.653677 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30f0415e-6d4f-41e5-8cdd-3cebb8ef2c1a" containerName="payload-processing" Oct 30 09:27:28.653911 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:28.653823 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="30f0415e-6d4f-41e5-8cdd-3cebb8ef2c1a" containerName="payload-processing" Oct 30 09:27:28.653911 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:28.653847 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="990bc616-4583-4bd0-b06b-e7f26e4cfa56" containerName="payload-pre-processing" Oct 30 09:27:28.659806 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:28.659776 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-51e091-5f4b488499-clrjg" Oct 30 09:27:28.677099 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:28.677068 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-51e091-5f4b488499-clrjg"] Oct 30 09:27:28.686114 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:28.686052 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-51e091-dockercfg-9mhwx\"" Oct 30 09:27:28.808475 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:28.799440 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-51e091\"" Oct 30 09:27:28.808475 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:28.801961 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/633dc628-d461-4b26-874d-1bead2cea5b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-51e091-5f4b488499-clrjg\" (UID: \"633dc628-d461-4b26-874d-1bead2cea5b9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-51e091-5f4b488499-clrjg" Oct 30 09:27:28.808475 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:28.802121 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvsdq\" (UniqueName: \"kubernetes.io/projected/633dc628-d461-4b26-874d-1bead2cea5b9-kube-api-access-tvsdq\") pod \"payload-pre-processing-e2e-ipp-b-51e091-5f4b488499-clrjg\" (UID: \"633dc628-d461-4b26-874d-1bead2cea5b9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-51e091-5f4b488499-clrjg" Oct 30 09:27:28.890456 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:28.887769 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-51e091-7956b856cb-qfl8z"] Oct 30 09:27:28.930412 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:28.900862 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-51e091-7956b856cb-qfl8z" Oct 30 09:27:28.930412 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:28.916477 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/633dc628-d461-4b26-874d-1bead2cea5b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-51e091-5f4b488499-clrjg\" (UID: \"633dc628-d461-4b26-874d-1bead2cea5b9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-51e091-5f4b488499-clrjg" Oct 30 09:27:28.930412 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:28.916728 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvsdq\" (UniqueName: \"kubernetes.io/projected/633dc628-d461-4b26-874d-1bead2cea5b9-kube-api-access-tvsdq\") pod \"payload-pre-processing-e2e-ipp-b-51e091-5f4b488499-clrjg\" (UID: \"633dc628-d461-4b26-874d-1bead2cea5b9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-51e091-5f4b488499-clrjg" Oct 30 09:27:28.930412 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:28.918555 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/633dc628-d461-4b26-874d-1bead2cea5b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-51e091-5f4b488499-clrjg\" (UID: \"633dc628-d461-4b26-874d-1bead2cea5b9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-51e091-5f4b488499-clrjg" Oct 30 09:27:28.971494 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:28.955422 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvsdq\" (UniqueName: \"kubernetes.io/projected/633dc628-d461-4b26-874d-1bead2cea5b9-kube-api-access-tvsdq\") pod \"payload-pre-processing-e2e-ipp-b-51e091-5f4b488499-clrjg\" (UID: \"633dc628-d461-4b26-874d-1bead2cea5b9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-51e091-5f4b488499-clrjg" Oct 30 09:27:28.974256 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:28.974168 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-51e091-5f4b488499-clrjg" Oct 30 09:27:28.992706 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:28.992581 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-51e091-7956b856cb-qfl8z"] Oct 30 09:27:29.019412 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:29.018553 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmm9s\" (UniqueName: \"kubernetes.io/projected/c1adfa80-f01a-40cb-bd1c-d1e98873d077-kube-api-access-wmm9s\") pod \"payload-processing-e2e-ipp-b-51e091-7956b856cb-qfl8z\" (UID: \"c1adfa80-f01a-40cb-bd1c-d1e98873d077\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-51e091-7956b856cb-qfl8z" Oct 30 09:27:29.019412 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:29.018606 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1adfa80-f01a-40cb-bd1c-d1e98873d077-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-51e091-7956b856cb-qfl8z\" (UID: \"c1adfa80-f01a-40cb-bd1c-d1e98873d077\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-51e091-7956b856cb-qfl8z" Oct 30 09:27:29.119636 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:29.119606 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmm9s\" (UniqueName: \"kubernetes.io/projected/c1adfa80-f01a-40cb-bd1c-d1e98873d077-kube-api-access-wmm9s\") pod \"payload-processing-e2e-ipp-b-51e091-7956b856cb-qfl8z\" (UID: \"c1adfa80-f01a-40cb-bd1c-d1e98873d077\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-51e091-7956b856cb-qfl8z" Oct 30 09:27:29.119789 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:29.119650 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1adfa80-f01a-40cb-bd1c-d1e98873d077-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-51e091-7956b856cb-qfl8z\" (UID: \"c1adfa80-f01a-40cb-bd1c-d1e98873d077\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-51e091-7956b856cb-qfl8z" Oct 30 09:27:29.120588 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:29.120563 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1adfa80-f01a-40cb-bd1c-d1e98873d077-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-51e091-7956b856cb-qfl8z\" (UID: \"c1adfa80-f01a-40cb-bd1c-d1e98873d077\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-51e091-7956b856cb-qfl8z" Oct 30 09:27:29.141630 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:29.141604 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmm9s\" (UniqueName: \"kubernetes.io/projected/c1adfa80-f01a-40cb-bd1c-d1e98873d077-kube-api-access-wmm9s\") pod \"payload-processing-e2e-ipp-b-51e091-7956b856cb-qfl8z\" (UID: \"c1adfa80-f01a-40cb-bd1c-d1e98873d077\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-51e091-7956b856cb-qfl8z" Oct 30 09:27:29.277789 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:29.277755 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-51e091-7956b856cb-qfl8z" Oct 30 09:27:29.300080 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:27:29.300042 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod633dc628_d461_4b26_874d_1bead2cea5b9.slice/crio-dbca6c2c664a47c129312439404a9016a6d69f7972f71b011c9a7a6c465eb229 WatchSource:0}: Error finding container dbca6c2c664a47c129312439404a9016a6d69f7972f71b011c9a7a6c465eb229: Status 404 returned error can't find the container with id dbca6c2c664a47c129312439404a9016a6d69f7972f71b011c9a7a6c465eb229 Oct 30 09:27:29.301010 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:29.300983 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-51e091-5f4b488499-clrjg"] Oct 30 09:27:29.649575 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:27:29.649544 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1adfa80_f01a_40cb_bd1c_d1e98873d077.slice/crio-33c342e3f978d08acf3faf3fd839b2993ac43dd4abf26f8ec0971a61a3f263b2 WatchSource:0}: Error finding container 33c342e3f978d08acf3faf3fd839b2993ac43dd4abf26f8ec0971a61a3f263b2: Status 404 returned error can't find the container with id 33c342e3f978d08acf3faf3fd839b2993ac43dd4abf26f8ec0971a61a3f263b2 Oct 30 09:27:29.650911 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:29.650883 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-51e091-7956b856cb-qfl8z"] Oct 30 09:27:30.034585 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:30.034553 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-51e091-7956b856cb-qfl8z" event={"ID":"c1adfa80-f01a-40cb-bd1c-d1e98873d077","Type":"ContainerStarted","Data":"20ced906e968a7f1bae8a63909b984d1b31fe8027bc207ed4b2d1340787306ca"} Oct 30 09:27:30.035007 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:30.034604 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-51e091-7956b856cb-qfl8z" event={"ID":"c1adfa80-f01a-40cb-bd1c-d1e98873d077","Type":"ContainerStarted","Data":"33c342e3f978d08acf3faf3fd839b2993ac43dd4abf26f8ec0971a61a3f263b2"} Oct 30 09:27:30.035007 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:30.034707 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-51e091-7956b856cb-qfl8z" Oct 30 09:27:30.036246 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:30.036214 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-51e091-5f4b488499-clrjg" event={"ID":"633dc628-d461-4b26-874d-1bead2cea5b9","Type":"ContainerStarted","Data":"d79daf111a47049e629ed1fc4c34a2dccd334e7cd396721c1955877370345b04"} Oct 30 09:27:30.036358 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:30.036262 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-51e091-5f4b488499-clrjg" event={"ID":"633dc628-d461-4b26-874d-1bead2cea5b9","Type":"ContainerStarted","Data":"dbca6c2c664a47c129312439404a9016a6d69f7972f71b011c9a7a6c465eb229"} Oct 30 09:27:30.036358 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:30.036288 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-51e091-5f4b488499-clrjg" Oct 30 09:27:30.118058 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:30.117995 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-51e091-5f4b488499-clrjg" podStartSLOduration=2.117979235 podStartE2EDuration="2.117979235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:27:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:27:30.117253043 +0000 UTC m=+2010.378814033" watchObservedRunningTime="2026-10-30 09:27:30.117979235 +0000 UTC m=+2010.379540224" Oct 30 09:27:30.120224 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:30.120187 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-51e091-7956b856cb-qfl8z" podStartSLOduration=2.12018023 podStartE2EDuration="2.12018023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:27:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:27:30.077571094 +0000 UTC m=+2010.339132083" watchObservedRunningTime="2026-10-30 09:27:30.12018023 +0000 UTC m=+2010.381741217" Oct 30 09:27:35.877311 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:35.877278 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh"] Oct 30 09:27:35.877745 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:35.877482 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh" podUID="99f76ea3-743e-4536-9443-2454af9db849" containerName="istio-proxy" containerID="cri-o://95bd10bbf27ff2e9d65865d0fef7b08e5d732c5544ed5b48fc36a8f5743a50bd" gracePeriod=30 Oct 30 09:27:36.048622 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:36.048586 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cbd96e-58fb495dc7-m7gn6"] Oct 30 09:27:36.053612 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:36.053585 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cbd96e-58fb495dc7-m7gn6" Oct 30 09:27:36.078399 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:36.077598 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-cbd96e-serving-cert\"" Oct 30 09:27:36.096708 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:36.096618 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cbd96e-58fb495dc7-m7gn6"] Oct 30 09:27:36.208130 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:36.208097 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56e6a564-249b-463f-a531-e1177d1c8150-metrics-certs\") pod \"maas-api-e2e-praxis-b-cbd96e-58fb495dc7-m7gn6\" (UID: \"56e6a564-249b-463f-a531-e1177d1c8150\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cbd96e-58fb495dc7-m7gn6" Oct 30 09:27:36.208326 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:36.208231 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56e6a564-249b-463f-a531-e1177d1c8150-maas-api-tls\") pod \"maas-api-e2e-praxis-b-cbd96e-58fb495dc7-m7gn6\" (UID: \"56e6a564-249b-463f-a531-e1177d1c8150\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cbd96e-58fb495dc7-m7gn6" Oct 30 09:27:36.208326 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:36.208299 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzh2z\" (UniqueName: \"kubernetes.io/projected/56e6a564-249b-463f-a531-e1177d1c8150-kube-api-access-nzh2z\") pod \"maas-api-e2e-praxis-b-cbd96e-58fb495dc7-m7gn6\" (UID: \"56e6a564-249b-463f-a531-e1177d1c8150\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cbd96e-58fb495dc7-m7gn6" Oct 30 09:27:36.208490 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:36.208336 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56e6a564-249b-463f-a531-e1177d1c8150-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-cbd96e-58fb495dc7-m7gn6\" (UID: \"56e6a564-249b-463f-a531-e1177d1c8150\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cbd96e-58fb495dc7-m7gn6" Oct 30 09:27:36.318408 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:36.310814 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56e6a564-249b-463f-a531-e1177d1c8150-metrics-certs\") pod \"maas-api-e2e-praxis-b-cbd96e-58fb495dc7-m7gn6\" (UID: \"56e6a564-249b-463f-a531-e1177d1c8150\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cbd96e-58fb495dc7-m7gn6" Oct 30 09:27:36.318829 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:36.318788 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56e6a564-249b-463f-a531-e1177d1c8150-maas-api-tls\") pod \"maas-api-e2e-praxis-b-cbd96e-58fb495dc7-m7gn6\" (UID: \"56e6a564-249b-463f-a531-e1177d1c8150\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cbd96e-58fb495dc7-m7gn6" Oct 30 09:27:36.319039 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:36.319019 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzh2z\" (UniqueName: \"kubernetes.io/projected/56e6a564-249b-463f-a531-e1177d1c8150-kube-api-access-nzh2z\") pod \"maas-api-e2e-praxis-b-cbd96e-58fb495dc7-m7gn6\" (UID: \"56e6a564-249b-463f-a531-e1177d1c8150\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cbd96e-58fb495dc7-m7gn6" Oct 30 09:27:36.319189 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:36.319169 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56e6a564-249b-463f-a531-e1177d1c8150-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-cbd96e-58fb495dc7-m7gn6\" (UID: \"56e6a564-249b-463f-a531-e1177d1c8150\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cbd96e-58fb495dc7-m7gn6" Oct 30 09:27:36.329541 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:36.320161 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56e6a564-249b-463f-a531-e1177d1c8150-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-cbd96e-58fb495dc7-m7gn6\" (UID: \"56e6a564-249b-463f-a531-e1177d1c8150\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cbd96e-58fb495dc7-m7gn6" Oct 30 09:27:36.351439 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:36.342818 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56e6a564-249b-463f-a531-e1177d1c8150-metrics-certs\") pod \"maas-api-e2e-praxis-b-cbd96e-58fb495dc7-m7gn6\" (UID: \"56e6a564-249b-463f-a531-e1177d1c8150\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cbd96e-58fb495dc7-m7gn6" Oct 30 09:27:36.366184 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:36.365554 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56e6a564-249b-463f-a531-e1177d1c8150-maas-api-tls\") pod \"maas-api-e2e-praxis-b-cbd96e-58fb495dc7-m7gn6\" (UID: \"56e6a564-249b-463f-a531-e1177d1c8150\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cbd96e-58fb495dc7-m7gn6" Oct 30 09:27:36.367941 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:36.367905 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzh2z\" (UniqueName: \"kubernetes.io/projected/56e6a564-249b-463f-a531-e1177d1c8150-kube-api-access-nzh2z\") pod \"maas-api-e2e-praxis-b-cbd96e-58fb495dc7-m7gn6\" (UID: \"56e6a564-249b-463f-a531-e1177d1c8150\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cbd96e-58fb495dc7-m7gn6" Oct 30 09:27:36.369655 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:36.369629 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cbd96e-58fb495dc7-m7gn6" Oct 30 09:27:36.670304 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:27:36.670269 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56e6a564_249b_463f_a531_e1177d1c8150.slice/crio-3062bc38ff6a52a0b99a4f3deae683c3f36696d8361be7c23128289dd8e2e730 WatchSource:0}: Error finding container 3062bc38ff6a52a0b99a4f3deae683c3f36696d8361be7c23128289dd8e2e730: Status 404 returned error can't find the container with id 3062bc38ff6a52a0b99a4f3deae683c3f36696d8361be7c23128289dd8e2e730 Oct 30 09:27:36.671974 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:36.671950 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cbd96e-58fb495dc7-m7gn6"] Oct 30 09:27:37.090574 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:37.090541 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cbd96e-58fb495dc7-m7gn6" event={"ID":"56e6a564-249b-463f-a531-e1177d1c8150","Type":"ContainerStarted","Data":"26b08e4b1a3b566e9a56caf92a3768b162939267d15af15d5540b35fce2d1b3b"} Oct 30 09:27:37.090955 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:37.090603 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cbd96e-58fb495dc7-m7gn6" event={"ID":"56e6a564-249b-463f-a531-e1177d1c8150","Type":"ContainerStarted","Data":"3062bc38ff6a52a0b99a4f3deae683c3f36696d8361be7c23128289dd8e2e730"} Oct 30 09:27:37.090955 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:37.090634 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cbd96e-58fb495dc7-m7gn6" Oct 30 09:27:37.564287 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:37.564219 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cbd96e-58fb495dc7-m7gn6" podStartSLOduration=1.261378876 podStartE2EDuration="1.564199236s" totalImagesPullingTime="302.82036ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:27:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:27:37.168512272 +0000 UTC m=+2017.430073285" watchObservedRunningTime="2026-10-30 09:27:37.564199236 +0000 UTC m=+2017.825760225" Oct 30 09:27:37.566775 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:37.566752 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2pgpc"] Oct 30 09:27:37.567004 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:37.566981 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2pgpc" podUID="12bbbb56-5c34-4297-8ab0-2d25958adcf5" containerName="istio-proxy" containerID="cri-o://d2637bf8160a36f897b1c8a96ae391f7bd86a920bdea62aa2565264dcae218ea" gracePeriod=30 Oct 30 09:27:37.572138 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:37.572111 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fh6qf"] Oct 30 09:27:37.572488 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:37.572459 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fh6qf" podUID="beaa426c-dd69-4ab1-8776-c2512f0ff19d" containerName="istio-proxy" containerID="cri-o://e50a38edde0f298df418766d799f723e9eb98535b06411fd585ac0b63762564c" gracePeriod=30 Oct 30 09:27:37.584245 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:37.584202 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k7rc9"] Oct 30 09:27:37.584478 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:37.584449 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k7rc9" podUID="866a5fb8-5939-4f6a-b01f-060e32222484" containerName="istio-proxy" containerID="cri-o://b132c538dc26d7d540925a98019ccb6a34c340d12a8fae92c383925d25de4064" gracePeriod=30 Oct 30 09:27:39.810179 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:39.810134 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9d39c0-648d978b78-446d4"] Oct 30 09:27:39.824732 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:39.824699 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9d39c0-648d978b78-446d4"] Oct 30 09:27:39.824973 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:39.824832 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9d39c0-648d978b78-446d4" Oct 30 09:27:39.830629 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:39.830586 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqjfw\" (UniqueName: \"kubernetes.io/projected/0141a7df-acc1-4bb4-9cd8-c66e44ee4eb4-kube-api-access-xqjfw\") pod \"payload-processing-e2e-ipp-a-9d39c0-648d978b78-446d4\" (UID: \"0141a7df-acc1-4bb4-9cd8-c66e44ee4eb4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9d39c0-648d978b78-446d4" Oct 30 09:27:39.830770 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:39.830745 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0141a7df-acc1-4bb4-9cd8-c66e44ee4eb4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9d39c0-648d978b78-446d4\" (UID: \"0141a7df-acc1-4bb4-9cd8-c66e44ee4eb4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9d39c0-648d978b78-446d4" Oct 30 09:27:39.842439 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:39.842368 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-9d39c0\"" Oct 30 09:27:39.860432 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:39.854601 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-9d39c0-dockercfg-5jmxw\"" Oct 30 09:27:39.964484 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:39.964442 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0141a7df-acc1-4bb4-9cd8-c66e44ee4eb4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9d39c0-648d978b78-446d4\" (UID: \"0141a7df-acc1-4bb4-9cd8-c66e44ee4eb4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9d39c0-648d978b78-446d4" Oct 30 09:27:39.977441 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:39.972759 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqjfw\" (UniqueName: \"kubernetes.io/projected/0141a7df-acc1-4bb4-9cd8-c66e44ee4eb4-kube-api-access-xqjfw\") pod \"payload-processing-e2e-ipp-a-9d39c0-648d978b78-446d4\" (UID: \"0141a7df-acc1-4bb4-9cd8-c66e44ee4eb4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9d39c0-648d978b78-446d4" Oct 30 09:27:39.978587 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:39.978555 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0141a7df-acc1-4bb4-9cd8-c66e44ee4eb4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9d39c0-648d978b78-446d4\" (UID: \"0141a7df-acc1-4bb4-9cd8-c66e44ee4eb4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9d39c0-648d978b78-446d4" Oct 30 09:27:40.005412 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:39.997326 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9d39c0-7fc7b8887f-rfr9x"] Oct 30 09:27:40.036402 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:40.017291 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqjfw\" (UniqueName: \"kubernetes.io/projected/0141a7df-acc1-4bb4-9cd8-c66e44ee4eb4-kube-api-access-xqjfw\") pod \"payload-processing-e2e-ipp-a-9d39c0-648d978b78-446d4\" (UID: \"0141a7df-acc1-4bb4-9cd8-c66e44ee4eb4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9d39c0-648d978b78-446d4" Oct 30 09:27:40.047406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:40.040814 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9d39c0-7fc7b8887f-rfr9x" Oct 30 09:27:40.165797 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:40.119629 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxhj8\" (UniqueName: \"kubernetes.io/projected/80a75bd3-f6d2-4a1b-ab28-d7111f61f5e0-kube-api-access-bxhj8\") pod \"payload-pre-processing-e2e-ipp-a-9d39c0-7fc7b8887f-rfr9x\" (UID: \"80a75bd3-f6d2-4a1b-ab28-d7111f61f5e0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9d39c0-7fc7b8887f-rfr9x" Oct 30 09:27:40.165797 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:40.119757 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80a75bd3-f6d2-4a1b-ab28-d7111f61f5e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9d39c0-7fc7b8887f-rfr9x\" (UID: \"80a75bd3-f6d2-4a1b-ab28-d7111f61f5e0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9d39c0-7fc7b8887f-rfr9x" Oct 30 09:27:40.165797 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:40.150907 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9d39c0-648d978b78-446d4" Oct 30 09:27:40.177151 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:40.177122 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9d39c0-7fc7b8887f-rfr9x"] Oct 30 09:27:40.220886 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:40.220847 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxhj8\" (UniqueName: \"kubernetes.io/projected/80a75bd3-f6d2-4a1b-ab28-d7111f61f5e0-kube-api-access-bxhj8\") pod \"payload-pre-processing-e2e-ipp-a-9d39c0-7fc7b8887f-rfr9x\" (UID: \"80a75bd3-f6d2-4a1b-ab28-d7111f61f5e0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9d39c0-7fc7b8887f-rfr9x" Oct 30 09:27:40.221066 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:40.220905 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80a75bd3-f6d2-4a1b-ab28-d7111f61f5e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9d39c0-7fc7b8887f-rfr9x\" (UID: \"80a75bd3-f6d2-4a1b-ab28-d7111f61f5e0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9d39c0-7fc7b8887f-rfr9x" Oct 30 09:27:40.221833 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:40.221808 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80a75bd3-f6d2-4a1b-ab28-d7111f61f5e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9d39c0-7fc7b8887f-rfr9x\" (UID: \"80a75bd3-f6d2-4a1b-ab28-d7111f61f5e0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9d39c0-7fc7b8887f-rfr9x" Oct 30 09:27:40.252569 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:40.252532 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxhj8\" (UniqueName: \"kubernetes.io/projected/80a75bd3-f6d2-4a1b-ab28-d7111f61f5e0-kube-api-access-bxhj8\") pod \"payload-pre-processing-e2e-ipp-a-9d39c0-7fc7b8887f-rfr9x\" (UID: \"80a75bd3-f6d2-4a1b-ab28-d7111f61f5e0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9d39c0-7fc7b8887f-rfr9x" Oct 30 09:27:40.471710 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:40.467755 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9d39c0-7fc7b8887f-rfr9x" Oct 30 09:27:40.546935 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:27:40.546900 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0141a7df_acc1_4bb4_9cd8_c66e44ee4eb4.slice/crio-01a970641c4b413e821dffe4c76f8676dc7194a89bd9f732912a976279a0a226 WatchSource:0}: Error finding container 01a970641c4b413e821dffe4c76f8676dc7194a89bd9f732912a976279a0a226: Status 404 returned error can't find the container with id 01a970641c4b413e821dffe4c76f8676dc7194a89bd9f732912a976279a0a226 Oct 30 09:27:40.548242 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:40.548213 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9d39c0-648d978b78-446d4"] Oct 30 09:27:40.828831 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:40.828795 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9d39c0-7fc7b8887f-rfr9x"] Oct 30 09:27:40.832576 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:27:40.832544 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80a75bd3_f6d2_4a1b_ab28_d7111f61f5e0.slice/crio-828305c94c3d566ae9c4f6b8260667e25b6af88537edf16ecd022933283e7a70 WatchSource:0}: Error finding container 828305c94c3d566ae9c4f6b8260667e25b6af88537edf16ecd022933283e7a70: Status 404 returned error can't find the container with id 828305c94c3d566ae9c4f6b8260667e25b6af88537edf16ecd022933283e7a70 Oct 30 09:27:41.082583 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:41.082554 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-51e091-5f4b488499-clrjg" Oct 30 09:27:41.085949 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:41.084888 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-51e091-7956b856cb-qfl8z" Oct 30 09:27:41.234566 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:41.234509 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9d39c0-648d978b78-446d4" event={"ID":"0141a7df-acc1-4bb4-9cd8-c66e44ee4eb4","Type":"ContainerStarted","Data":"2bd5c139ae0e3da61ce049fc14e5f017cd3f6c6a29419358d8f13f173a64e21f"} Oct 30 09:27:41.234741 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:41.234579 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9d39c0-648d978b78-446d4" event={"ID":"0141a7df-acc1-4bb4-9cd8-c66e44ee4eb4","Type":"ContainerStarted","Data":"01a970641c4b413e821dffe4c76f8676dc7194a89bd9f732912a976279a0a226"} Oct 30 09:27:41.234981 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:41.234958 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-9d39c0-648d978b78-446d4" Oct 30 09:27:41.243319 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:41.242423 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9d39c0-7fc7b8887f-rfr9x" event={"ID":"80a75bd3-f6d2-4a1b-ab28-d7111f61f5e0","Type":"ContainerStarted","Data":"7cd7c914d6399085825ad11c40b0c2373320b53de6f3ea1c59a3f63b78ebe37a"} Oct 30 09:27:41.243319 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:41.242486 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9d39c0-7fc7b8887f-rfr9x" event={"ID":"80a75bd3-f6d2-4a1b-ab28-d7111f61f5e0","Type":"ContainerStarted","Data":"828305c94c3d566ae9c4f6b8260667e25b6af88537edf16ecd022933283e7a70"} Oct 30 09:27:41.243319 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:41.243253 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9d39c0-7fc7b8887f-rfr9x" Oct 30 09:27:41.245855 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:41.245827 4562 generic.go:356] "Generic (PLEG): container finished" podID="99f76ea3-743e-4536-9443-2454af9db849" containerID="95bd10bbf27ff2e9d65865d0fef7b08e5d732c5544ed5b48fc36a8f5743a50bd" exitCode=0 Oct 30 09:27:41.246054 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:41.246035 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh" event={"ID":"99f76ea3-743e-4536-9443-2454af9db849","Type":"ContainerDied","Data":"95bd10bbf27ff2e9d65865d0fef7b08e5d732c5544ed5b48fc36a8f5743a50bd"} Oct 30 09:27:41.278026 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:41.277958 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-9d39c0-648d978b78-446d4" podStartSLOduration=2.277940076 podStartE2EDuration="2.277940076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:27:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:27:41.276910661 +0000 UTC m=+2021.538471649" watchObservedRunningTime="2026-10-30 09:27:41.277940076 +0000 UTC m=+2021.539501064" Oct 30 09:27:41.331158 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:41.329472 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9d39c0-7fc7b8887f-rfr9x" podStartSLOduration=2.3294522029999998 podStartE2EDuration="2.329452203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:27:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:27:41.323830555 +0000 UTC m=+2021.585391545" watchObservedRunningTime="2026-10-30 09:27:41.329452203 +0000 UTC m=+2021.591013191" Oct 30 09:27:41.431259 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:41.431163 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh" Oct 30 09:27:41.528361 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:41.523219 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99f76ea3-743e-4536-9443-2454af9db849-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99f76ea3-743e-4536-9443-2454af9db849-istio-envoy\") pod \"99f76ea3-743e-4536-9443-2454af9db849\" (UID: \"99f76ea3-743e-4536-9443-2454af9db849\") " Oct 30 09:27:41.528361 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:41.523281 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99f76ea3-743e-4536-9443-2454af9db849-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99f76ea3-743e-4536-9443-2454af9db849-workload-socket\") pod \"99f76ea3-743e-4536-9443-2454af9db849\" (UID: \"99f76ea3-743e-4536-9443-2454af9db849\") " Oct 30 09:27:41.528361 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:41.523325 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99f76ea3-743e-4536-9443-2454af9db849-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99f76ea3-743e-4536-9443-2454af9db849-istiod-ca-cert\") pod \"99f76ea3-743e-4536-9443-2454af9db849\" (UID: \"99f76ea3-743e-4536-9443-2454af9db849\") " Oct 30 09:27:41.528361 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:41.523398 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/99f76ea3-743e-4536-9443-2454af9db849-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99f76ea3-743e-4536-9443-2454af9db849-istio-podinfo\") pod \"99f76ea3-743e-4536-9443-2454af9db849\" (UID: \"99f76ea3-743e-4536-9443-2454af9db849\") " Oct 30 09:27:41.528361 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:41.523484 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99f76ea3-743e-4536-9443-2454af9db849-kube-api-access-5gjjn\" (UniqueName: \"kubernetes.io/projected/99f76ea3-743e-4536-9443-2454af9db849-kube-api-access-5gjjn\") pod \"99f76ea3-743e-4536-9443-2454af9db849\" (UID: \"99f76ea3-743e-4536-9443-2454af9db849\") " Oct 30 09:27:41.528361 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:41.523559 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99f76ea3-743e-4536-9443-2454af9db849-istio-token\" (UniqueName: \"kubernetes.io/projected/99f76ea3-743e-4536-9443-2454af9db849-istio-token\") pod \"99f76ea3-743e-4536-9443-2454af9db849\" (UID: \"99f76ea3-743e-4536-9443-2454af9db849\") " Oct 30 09:27:41.528361 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:41.523648 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99f76ea3-743e-4536-9443-2454af9db849-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99f76ea3-743e-4536-9443-2454af9db849-workload-certs\") pod \"99f76ea3-743e-4536-9443-2454af9db849\" (UID: \"99f76ea3-743e-4536-9443-2454af9db849\") " Oct 30 09:27:41.528361 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:41.523676 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99f76ea3-743e-4536-9443-2454af9db849-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99f76ea3-743e-4536-9443-2454af9db849-credential-socket\") pod \"99f76ea3-743e-4536-9443-2454af9db849\" (UID: \"99f76ea3-743e-4536-9443-2454af9db849\") " Oct 30 09:27:41.528361 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:41.523779 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99f76ea3-743e-4536-9443-2454af9db849-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99f76ea3-743e-4536-9443-2454af9db849-istio-data\") pod \"99f76ea3-743e-4536-9443-2454af9db849\" (UID: \"99f76ea3-743e-4536-9443-2454af9db849\") " Oct 30 09:27:41.528361 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:41.525168 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99f76ea3-743e-4536-9443-2454af9db849-istio-data" pod "99f76ea3-743e-4536-9443-2454af9db849" (UID: "99f76ea3-743e-4536-9443-2454af9db849"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:27:41.529313 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:41.529286 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99f76ea3-743e-4536-9443-2454af9db849-workload-socket" pod "99f76ea3-743e-4536-9443-2454af9db849" (UID: "99f76ea3-743e-4536-9443-2454af9db849"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:27:41.530590 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:41.529965 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99f76ea3-743e-4536-9443-2454af9db849-istio-envoy" pod "99f76ea3-743e-4536-9443-2454af9db849" (UID: "99f76ea3-743e-4536-9443-2454af9db849"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:27:41.530590 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:41.530553 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99f76ea3-743e-4536-9443-2454af9db849-workload-certs" pod "99f76ea3-743e-4536-9443-2454af9db849" (UID: "99f76ea3-743e-4536-9443-2454af9db849"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:27:41.530845 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:41.530827 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99f76ea3-743e-4536-9443-2454af9db849-istiod-ca-cert" pod "99f76ea3-743e-4536-9443-2454af9db849" (UID: "99f76ea3-743e-4536-9443-2454af9db849"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:27:41.531717 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:41.531695 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99f76ea3-743e-4536-9443-2454af9db849-credential-socket" pod "99f76ea3-743e-4536-9443-2454af9db849" (UID: "99f76ea3-743e-4536-9443-2454af9db849"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:27:41.540961 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:41.540937 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99f76ea3-743e-4536-9443-2454af9db849-kube-api-access-5gjjn" pod "99f76ea3-743e-4536-9443-2454af9db849" (UID: "99f76ea3-743e-4536-9443-2454af9db849"). InnerVolumeSpecName "kube-api-access-5gjjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:27:41.541273 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:41.541245 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/99f76ea3-743e-4536-9443-2454af9db849-istio-podinfo" pod "99f76ea3-743e-4536-9443-2454af9db849" (UID: "99f76ea3-743e-4536-9443-2454af9db849"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:27:41.541471 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:41.541447 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99f76ea3-743e-4536-9443-2454af9db849-istio-token" pod "99f76ea3-743e-4536-9443-2454af9db849" (UID: "99f76ea3-743e-4536-9443-2454af9db849"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:27:41.625950 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:41.625835 4562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99f76ea3-743e-4536-9443-2454af9db849-istio-token\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:27:41.625950 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:41.625864 4562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99f76ea3-743e-4536-9443-2454af9db849-workload-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:27:41.625950 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:41.625875 4562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99f76ea3-743e-4536-9443-2454af9db849-credential-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:27:41.625950 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:41.625885 4562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99f76ea3-743e-4536-9443-2454af9db849-istio-data\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:27:41.625950 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:41.625894 4562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99f76ea3-743e-4536-9443-2454af9db849-istio-envoy\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:27:41.625950 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:41.625902 4562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99f76ea3-743e-4536-9443-2454af9db849-workload-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:27:41.625950 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:41.625911 4562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99f76ea3-743e-4536-9443-2454af9db849-istiod-ca-cert\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:27:41.625950 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:41.625920 4562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99f76ea3-743e-4536-9443-2454af9db849-istio-podinfo\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:27:41.625950 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:41.625930 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gjjn\" (UniqueName: \"kubernetes.io/projected/99f76ea3-743e-4536-9443-2454af9db849-kube-api-access-5gjjn\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:27:42.254451 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:42.254421 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh" Oct 30 09:27:42.254948 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:42.254449 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh" event={"ID":"99f76ea3-743e-4536-9443-2454af9db849","Type":"ContainerDied","Data":"b9460fe4ebcd89c4dc8259a803c452dc52d0ae8760d81d20639a2bad222e3a45"} Oct 30 09:27:42.254948 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:42.254514 4562 scope.go:122] "RemoveContainer" containerID="95bd10bbf27ff2e9d65865d0fef7b08e5d732c5544ed5b48fc36a8f5743a50bd" Oct 30 09:27:42.331646 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:42.331619 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh"] Oct 30 09:27:42.396405 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:42.396016 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-2jsfh"] Oct 30 09:27:42.901410 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:42.900605 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2pgpc" Oct 30 09:27:42.942827 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:42.942793 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fh6qf" Oct 30 09:27:42.950822 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:42.950793 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k7rc9" Oct 30 09:27:42.979509 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:42.979478 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/beaa426c-dd69-4ab1-8776-c2512f0ff19d-kube-api-access-j6gf2\" (UniqueName: \"kubernetes.io/projected/beaa426c-dd69-4ab1-8776-c2512f0ff19d-kube-api-access-j6gf2\") pod \"beaa426c-dd69-4ab1-8776-c2512f0ff19d\" (UID: \"beaa426c-dd69-4ab1-8776-c2512f0ff19d\") " Oct 30 09:27:42.979713 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:42.979536 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beaa426c-dd69-4ab1-8776-c2512f0ff19d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beaa426c-dd69-4ab1-8776-c2512f0ff19d-workload-certs\") pod \"beaa426c-dd69-4ab1-8776-c2512f0ff19d\" (UID: \"beaa426c-dd69-4ab1-8776-c2512f0ff19d\") " Oct 30 09:27:42.979713 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:42.979576 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12bbbb56-5c34-4297-8ab0-2d25958adcf5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12bbbb56-5c34-4297-8ab0-2d25958adcf5-workload-socket\") pod \"12bbbb56-5c34-4297-8ab0-2d25958adcf5\" (UID: \"12bbbb56-5c34-4297-8ab0-2d25958adcf5\") " Oct 30 09:27:42.979713 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:42.979628 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12bbbb56-5c34-4297-8ab0-2d25958adcf5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12bbbb56-5c34-4297-8ab0-2d25958adcf5-workload-certs\") pod \"12bbbb56-5c34-4297-8ab0-2d25958adcf5\" (UID: \"12bbbb56-5c34-4297-8ab0-2d25958adcf5\") " Oct 30 09:27:42.979713 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:42.979654 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/866a5fb8-5939-4f6a-b01f-060e32222484-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/866a5fb8-5939-4f6a-b01f-060e32222484-workload-socket\") pod \"866a5fb8-5939-4f6a-b01f-060e32222484\" (UID: \"866a5fb8-5939-4f6a-b01f-060e32222484\") " Oct 30 09:27:42.979713 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:42.979691 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beaa426c-dd69-4ab1-8776-c2512f0ff19d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beaa426c-dd69-4ab1-8776-c2512f0ff19d-workload-socket\") pod \"beaa426c-dd69-4ab1-8776-c2512f0ff19d\" (UID: \"beaa426c-dd69-4ab1-8776-c2512f0ff19d\") " Oct 30 09:27:42.980097 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:42.979739 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beaa426c-dd69-4ab1-8776-c2512f0ff19d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beaa426c-dd69-4ab1-8776-c2512f0ff19d-credential-socket\") pod \"beaa426c-dd69-4ab1-8776-c2512f0ff19d\" (UID: \"beaa426c-dd69-4ab1-8776-c2512f0ff19d\") " Oct 30 09:27:42.980097 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:42.979797 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/beaa426c-dd69-4ab1-8776-c2512f0ff19d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beaa426c-dd69-4ab1-8776-c2512f0ff19d-istiod-ca-cert\") pod \"beaa426c-dd69-4ab1-8776-c2512f0ff19d\" (UID: \"beaa426c-dd69-4ab1-8776-c2512f0ff19d\") " Oct 30 09:27:42.980097 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:42.979828 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12bbbb56-5c34-4297-8ab0-2d25958adcf5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12bbbb56-5c34-4297-8ab0-2d25958adcf5-istio-envoy\") pod \"12bbbb56-5c34-4297-8ab0-2d25958adcf5\" (UID: \"12bbbb56-5c34-4297-8ab0-2d25958adcf5\") " Oct 30 09:27:42.980097 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:42.979867 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/866a5fb8-5939-4f6a-b01f-060e32222484-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/866a5fb8-5939-4f6a-b01f-060e32222484-workload-certs\") pod \"866a5fb8-5939-4f6a-b01f-060e32222484\" (UID: \"866a5fb8-5939-4f6a-b01f-060e32222484\") " Oct 30 09:27:42.980097 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:42.979893 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12bbbb56-5c34-4297-8ab0-2d25958adcf5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12bbbb56-5c34-4297-8ab0-2d25958adcf5-credential-socket\") pod \"12bbbb56-5c34-4297-8ab0-2d25958adcf5\" (UID: \"12bbbb56-5c34-4297-8ab0-2d25958adcf5\") " Oct 30 09:27:42.980097 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:42.979925 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/866a5fb8-5939-4f6a-b01f-060e32222484-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/866a5fb8-5939-4f6a-b01f-060e32222484-istio-envoy\") pod \"866a5fb8-5939-4f6a-b01f-060e32222484\" (UID: \"866a5fb8-5939-4f6a-b01f-060e32222484\") " Oct 30 09:27:42.980097 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:42.979957 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/beaa426c-dd69-4ab1-8776-c2512f0ff19d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beaa426c-dd69-4ab1-8776-c2512f0ff19d-istio-podinfo\") pod \"beaa426c-dd69-4ab1-8776-c2512f0ff19d\" (UID: \"beaa426c-dd69-4ab1-8776-c2512f0ff19d\") " Oct 30 09:27:42.980097 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:42.980008 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12bbbb56-5c34-4297-8ab0-2d25958adcf5-kube-api-access-vqv82\" (UniqueName: \"kubernetes.io/projected/12bbbb56-5c34-4297-8ab0-2d25958adcf5-kube-api-access-vqv82\") pod \"12bbbb56-5c34-4297-8ab0-2d25958adcf5\" (UID: \"12bbbb56-5c34-4297-8ab0-2d25958adcf5\") " Oct 30 09:27:42.980097 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:42.980040 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/beaa426c-dd69-4ab1-8776-c2512f0ff19d-istio-token\" (UniqueName: \"kubernetes.io/projected/beaa426c-dd69-4ab1-8776-c2512f0ff19d-istio-token\") pod \"beaa426c-dd69-4ab1-8776-c2512f0ff19d\" (UID: \"beaa426c-dd69-4ab1-8776-c2512f0ff19d\") " Oct 30 09:27:42.980097 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:42.980083 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12bbbb56-5c34-4297-8ab0-2d25958adcf5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12bbbb56-5c34-4297-8ab0-2d25958adcf5-istiod-ca-cert\") pod \"12bbbb56-5c34-4297-8ab0-2d25958adcf5\" (UID: \"12bbbb56-5c34-4297-8ab0-2d25958adcf5\") " Oct 30 09:27:42.980648 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:42.980120 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/866a5fb8-5939-4f6a-b01f-060e32222484-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/866a5fb8-5939-4f6a-b01f-060e32222484-istio-data\") pod \"866a5fb8-5939-4f6a-b01f-060e32222484\" (UID: \"866a5fb8-5939-4f6a-b01f-060e32222484\") " Oct 30 09:27:42.980648 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:42.980153 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/12bbbb56-5c34-4297-8ab0-2d25958adcf5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12bbbb56-5c34-4297-8ab0-2d25958adcf5-istio-podinfo\") pod \"12bbbb56-5c34-4297-8ab0-2d25958adcf5\" (UID: \"12bbbb56-5c34-4297-8ab0-2d25958adcf5\") " Oct 30 09:27:42.980648 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:42.980185 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/866a5fb8-5939-4f6a-b01f-060e32222484-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/866a5fb8-5939-4f6a-b01f-060e32222484-istio-podinfo\") pod \"866a5fb8-5939-4f6a-b01f-060e32222484\" (UID: \"866a5fb8-5939-4f6a-b01f-060e32222484\") " Oct 30 09:27:42.980648 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:42.980244 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12bbbb56-5c34-4297-8ab0-2d25958adcf5-istio-token\" (UniqueName: \"kubernetes.io/projected/12bbbb56-5c34-4297-8ab0-2d25958adcf5-istio-token\") pod \"12bbbb56-5c34-4297-8ab0-2d25958adcf5\" (UID: \"12bbbb56-5c34-4297-8ab0-2d25958adcf5\") " Oct 30 09:27:42.980648 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:42.980276 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/866a5fb8-5939-4f6a-b01f-060e32222484-kube-api-access-79mjt\" (UniqueName: \"kubernetes.io/projected/866a5fb8-5939-4f6a-b01f-060e32222484-kube-api-access-79mjt\") pod \"866a5fb8-5939-4f6a-b01f-060e32222484\" (UID: \"866a5fb8-5939-4f6a-b01f-060e32222484\") " Oct 30 09:27:42.980648 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:42.980306 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beaa426c-dd69-4ab1-8776-c2512f0ff19d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beaa426c-dd69-4ab1-8776-c2512f0ff19d-istio-envoy\") pod \"beaa426c-dd69-4ab1-8776-c2512f0ff19d\" (UID: \"beaa426c-dd69-4ab1-8776-c2512f0ff19d\") " Oct 30 09:27:42.980648 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:42.980339 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beaa426c-dd69-4ab1-8776-c2512f0ff19d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beaa426c-dd69-4ab1-8776-c2512f0ff19d-istio-data\") pod \"beaa426c-dd69-4ab1-8776-c2512f0ff19d\" (UID: \"beaa426c-dd69-4ab1-8776-c2512f0ff19d\") " Oct 30 09:27:42.980648 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:42.980414 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/866a5fb8-5939-4f6a-b01f-060e32222484-istio-token\" (UniqueName: \"kubernetes.io/projected/866a5fb8-5939-4f6a-b01f-060e32222484-istio-token\") pod \"866a5fb8-5939-4f6a-b01f-060e32222484\" (UID: \"866a5fb8-5939-4f6a-b01f-060e32222484\") " Oct 30 09:27:42.980648 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:42.980577 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/866a5fb8-5939-4f6a-b01f-060e32222484-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/866a5fb8-5939-4f6a-b01f-060e32222484-credential-socket\") pod \"866a5fb8-5939-4f6a-b01f-060e32222484\" (UID: \"866a5fb8-5939-4f6a-b01f-060e32222484\") " Oct 30 09:27:42.980648 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:42.980616 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12bbbb56-5c34-4297-8ab0-2d25958adcf5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12bbbb56-5c34-4297-8ab0-2d25958adcf5-istio-data\") pod \"12bbbb56-5c34-4297-8ab0-2d25958adcf5\" (UID: \"12bbbb56-5c34-4297-8ab0-2d25958adcf5\") " Oct 30 09:27:42.981155 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:42.980709 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/866a5fb8-5939-4f6a-b01f-060e32222484-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/866a5fb8-5939-4f6a-b01f-060e32222484-istiod-ca-cert\") pod \"866a5fb8-5939-4f6a-b01f-060e32222484\" (UID: \"866a5fb8-5939-4f6a-b01f-060e32222484\") " Oct 30 09:27:42.981871 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:42.981795 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/866a5fb8-5939-4f6a-b01f-060e32222484-istiod-ca-cert" pod "866a5fb8-5939-4f6a-b01f-060e32222484" (UID: "866a5fb8-5939-4f6a-b01f-060e32222484"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:27:42.982308 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:42.982028 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beaa426c-dd69-4ab1-8776-c2512f0ff19d-workload-socket" pod "beaa426c-dd69-4ab1-8776-c2512f0ff19d" (UID: "beaa426c-dd69-4ab1-8776-c2512f0ff19d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:27:42.983856 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:42.982684 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12bbbb56-5c34-4297-8ab0-2d25958adcf5-workload-socket" pod "12bbbb56-5c34-4297-8ab0-2d25958adcf5" (UID: "12bbbb56-5c34-4297-8ab0-2d25958adcf5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:27:42.983856 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:42.982796 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/866a5fb8-5939-4f6a-b01f-060e32222484-workload-socket" pod "866a5fb8-5939-4f6a-b01f-060e32222484" (UID: "866a5fb8-5939-4f6a-b01f-060e32222484"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:27:42.983856 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:42.982909 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beaa426c-dd69-4ab1-8776-c2512f0ff19d-workload-certs" pod "beaa426c-dd69-4ab1-8776-c2512f0ff19d" (UID: "beaa426c-dd69-4ab1-8776-c2512f0ff19d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:27:42.983856 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:42.982932 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12bbbb56-5c34-4297-8ab0-2d25958adcf5-workload-certs" pod "12bbbb56-5c34-4297-8ab0-2d25958adcf5" (UID: "12bbbb56-5c34-4297-8ab0-2d25958adcf5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:27:42.983856 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:42.983096 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beaa426c-dd69-4ab1-8776-c2512f0ff19d-istio-data" pod "beaa426c-dd69-4ab1-8776-c2512f0ff19d" (UID: "beaa426c-dd69-4ab1-8776-c2512f0ff19d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:27:42.983856 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:42.983287 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beaa426c-dd69-4ab1-8776-c2512f0ff19d-credential-socket" pod "beaa426c-dd69-4ab1-8776-c2512f0ff19d" (UID: "beaa426c-dd69-4ab1-8776-c2512f0ff19d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:27:42.983856 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:42.983585 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12bbbb56-5c34-4297-8ab0-2d25958adcf5-istiod-ca-cert" pod "12bbbb56-5c34-4297-8ab0-2d25958adcf5" (UID: "12bbbb56-5c34-4297-8ab0-2d25958adcf5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:27:42.984575 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:42.984532 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/866a5fb8-5939-4f6a-b01f-060e32222484-kube-api-access-79mjt" pod "866a5fb8-5939-4f6a-b01f-060e32222484" (UID: "866a5fb8-5939-4f6a-b01f-060e32222484"). InnerVolumeSpecName "kube-api-access-79mjt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:27:42.984808 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:42.984757 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/866a5fb8-5939-4f6a-b01f-060e32222484-credential-socket" pod "866a5fb8-5939-4f6a-b01f-060e32222484" (UID: "866a5fb8-5939-4f6a-b01f-060e32222484"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:27:42.985122 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:42.984933 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12bbbb56-5c34-4297-8ab0-2d25958adcf5-istio-data" pod "12bbbb56-5c34-4297-8ab0-2d25958adcf5" (UID: "12bbbb56-5c34-4297-8ab0-2d25958adcf5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:27:42.987725 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:42.987565 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12bbbb56-5c34-4297-8ab0-2d25958adcf5-credential-socket" pod "12bbbb56-5c34-4297-8ab0-2d25958adcf5" (UID: "12bbbb56-5c34-4297-8ab0-2d25958adcf5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:27:42.988131 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:42.987887 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/866a5fb8-5939-4f6a-b01f-060e32222484-workload-certs" pod "866a5fb8-5939-4f6a-b01f-060e32222484" (UID: "866a5fb8-5939-4f6a-b01f-060e32222484"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:27:42.989424 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:42.989335 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/beaa426c-dd69-4ab1-8776-c2512f0ff19d-istiod-ca-cert" pod "beaa426c-dd69-4ab1-8776-c2512f0ff19d" (UID: "beaa426c-dd69-4ab1-8776-c2512f0ff19d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:27:42.991053 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:42.991025 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/866a5fb8-5939-4f6a-b01f-060e32222484-istio-token" pod "866a5fb8-5939-4f6a-b01f-060e32222484" (UID: "866a5fb8-5939-4f6a-b01f-060e32222484"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:27:42.991161 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:42.991067 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12bbbb56-5c34-4297-8ab0-2d25958adcf5-istio-token" pod "12bbbb56-5c34-4297-8ab0-2d25958adcf5" (UID: "12bbbb56-5c34-4297-8ab0-2d25958adcf5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:27:42.991413 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:42.991268 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/866a5fb8-5939-4f6a-b01f-060e32222484-istio-data" pod "866a5fb8-5939-4f6a-b01f-060e32222484" (UID: "866a5fb8-5939-4f6a-b01f-060e32222484"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:27:42.991413 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:42.991300 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/866a5fb8-5939-4f6a-b01f-060e32222484-istio-envoy" pod "866a5fb8-5939-4f6a-b01f-060e32222484" (UID: "866a5fb8-5939-4f6a-b01f-060e32222484"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:27:42.991538 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:42.991427 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beaa426c-dd69-4ab1-8776-c2512f0ff19d-kube-api-access-j6gf2" pod "beaa426c-dd69-4ab1-8776-c2512f0ff19d" (UID: "beaa426c-dd69-4ab1-8776-c2512f0ff19d"). InnerVolumeSpecName "kube-api-access-j6gf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:27:42.991538 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:42.991436 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12bbbb56-5c34-4297-8ab0-2d25958adcf5-istio-envoy" pod "12bbbb56-5c34-4297-8ab0-2d25958adcf5" (UID: "12bbbb56-5c34-4297-8ab0-2d25958adcf5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:27:42.991538 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:42.991471 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12bbbb56-5c34-4297-8ab0-2d25958adcf5-kube-api-access-vqv82" pod "12bbbb56-5c34-4297-8ab0-2d25958adcf5" (UID: "12bbbb56-5c34-4297-8ab0-2d25958adcf5"). InnerVolumeSpecName "kube-api-access-vqv82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:27:42.991538 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:42.991517 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beaa426c-dd69-4ab1-8776-c2512f0ff19d-istio-envoy" pod "beaa426c-dd69-4ab1-8776-c2512f0ff19d" (UID: "beaa426c-dd69-4ab1-8776-c2512f0ff19d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:27:42.991749 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:42.991653 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/12bbbb56-5c34-4297-8ab0-2d25958adcf5-istio-podinfo" pod "12bbbb56-5c34-4297-8ab0-2d25958adcf5" (UID: "12bbbb56-5c34-4297-8ab0-2d25958adcf5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:27:42.991749 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:42.991720 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/beaa426c-dd69-4ab1-8776-c2512f0ff19d-istio-podinfo" pod "beaa426c-dd69-4ab1-8776-c2512f0ff19d" (UID: "beaa426c-dd69-4ab1-8776-c2512f0ff19d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:27:42.993310 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:42.993285 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beaa426c-dd69-4ab1-8776-c2512f0ff19d-istio-token" pod "beaa426c-dd69-4ab1-8776-c2512f0ff19d" (UID: "beaa426c-dd69-4ab1-8776-c2512f0ff19d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:27:42.993400 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:42.993310 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/866a5fb8-5939-4f6a-b01f-060e32222484-istio-podinfo" pod "866a5fb8-5939-4f6a-b01f-060e32222484" (UID: "866a5fb8-5939-4f6a-b01f-060e32222484"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:27:43.083439 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:43.083404 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6gf2\" (UniqueName: \"kubernetes.io/projected/beaa426c-dd69-4ab1-8776-c2512f0ff19d-kube-api-access-j6gf2\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:27:43.083439 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:43.083441 4562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beaa426c-dd69-4ab1-8776-c2512f0ff19d-workload-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:27:43.083698 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:43.083456 4562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12bbbb56-5c34-4297-8ab0-2d25958adcf5-workload-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:27:43.083698 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:43.083470 4562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12bbbb56-5c34-4297-8ab0-2d25958adcf5-workload-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:27:43.083698 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:43.083484 4562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/866a5fb8-5939-4f6a-b01f-060e32222484-workload-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:27:43.083698 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:43.083497 4562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beaa426c-dd69-4ab1-8776-c2512f0ff19d-workload-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:27:43.083698 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:43.083510 4562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beaa426c-dd69-4ab1-8776-c2512f0ff19d-credential-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:27:43.083698 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:43.083524 4562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beaa426c-dd69-4ab1-8776-c2512f0ff19d-istiod-ca-cert\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:27:43.083698 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:43.083537 4562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12bbbb56-5c34-4297-8ab0-2d25958adcf5-istio-envoy\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:27:43.083698 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:43.083549 4562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/866a5fb8-5939-4f6a-b01f-060e32222484-workload-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:27:43.083698 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:43.083562 4562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12bbbb56-5c34-4297-8ab0-2d25958adcf5-credential-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:27:43.083698 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:43.083574 4562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/866a5fb8-5939-4f6a-b01f-060e32222484-istio-envoy\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:27:43.083698 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:43.083587 4562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beaa426c-dd69-4ab1-8776-c2512f0ff19d-istio-podinfo\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:27:43.083698 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:43.083599 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqv82\" (UniqueName: \"kubernetes.io/projected/12bbbb56-5c34-4297-8ab0-2d25958adcf5-kube-api-access-vqv82\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:27:43.083698 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:43.083613 4562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beaa426c-dd69-4ab1-8776-c2512f0ff19d-istio-token\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:27:43.083698 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:43.083626 4562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12bbbb56-5c34-4297-8ab0-2d25958adcf5-istiod-ca-cert\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:27:43.083698 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:43.083638 4562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/866a5fb8-5939-4f6a-b01f-060e32222484-istio-data\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:27:43.083698 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:43.083650 4562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12bbbb56-5c34-4297-8ab0-2d25958adcf5-istio-podinfo\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:27:43.083698 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:43.083663 4562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/866a5fb8-5939-4f6a-b01f-060e32222484-istio-podinfo\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:27:43.083698 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:43.083676 4562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12bbbb56-5c34-4297-8ab0-2d25958adcf5-istio-token\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:27:43.083698 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:43.083690 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79mjt\" (UniqueName: \"kubernetes.io/projected/866a5fb8-5939-4f6a-b01f-060e32222484-kube-api-access-79mjt\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:27:43.084693 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:43.083725 4562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beaa426c-dd69-4ab1-8776-c2512f0ff19d-istio-envoy\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:27:43.084693 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:43.083740 4562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beaa426c-dd69-4ab1-8776-c2512f0ff19d-istio-data\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:27:43.084693 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:43.083753 4562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/866a5fb8-5939-4f6a-b01f-060e32222484-istio-token\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:27:43.084693 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:43.083766 4562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/866a5fb8-5939-4f6a-b01f-060e32222484-credential-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:27:43.084693 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:43.083779 4562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12bbbb56-5c34-4297-8ab0-2d25958adcf5-istio-data\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:27:43.084693 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:43.083794 4562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/866a5fb8-5939-4f6a-b01f-060e32222484-istiod-ca-cert\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:27:43.106853 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:43.106771 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cbd96e-58fb495dc7-m7gn6" Oct 30 09:27:43.264542 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:43.264447 4562 generic.go:356] "Generic (PLEG): container finished" podID="866a5fb8-5939-4f6a-b01f-060e32222484" containerID="b132c538dc26d7d540925a98019ccb6a34c340d12a8fae92c383925d25de4064" exitCode=0 Oct 30 09:27:43.264542 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:43.264491 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k7rc9" Oct 30 09:27:43.265031 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:43.264505 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k7rc9" event={"ID":"866a5fb8-5939-4f6a-b01f-060e32222484","Type":"ContainerDied","Data":"b132c538dc26d7d540925a98019ccb6a34c340d12a8fae92c383925d25de4064"} Oct 30 09:27:43.265031 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:43.264646 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k7rc9" event={"ID":"866a5fb8-5939-4f6a-b01f-060e32222484","Type":"ContainerDied","Data":"d097f8d4f8f5f5e18b6e48a383a69ba4dc46e19e275caf3a66f28aa83a4e9ff1"} Oct 30 09:27:43.265031 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:43.264691 4562 scope.go:122] "RemoveContainer" containerID="b132c538dc26d7d540925a98019ccb6a34c340d12a8fae92c383925d25de4064" Oct 30 09:27:43.267624 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:43.267600 4562 generic.go:356] "Generic (PLEG): container finished" podID="beaa426c-dd69-4ab1-8776-c2512f0ff19d" containerID="e50a38edde0f298df418766d799f723e9eb98535b06411fd585ac0b63762564c" exitCode=0 Oct 30 09:27:43.267743 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:43.267644 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fh6qf" Oct 30 09:27:43.267743 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:43.267679 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fh6qf" event={"ID":"beaa426c-dd69-4ab1-8776-c2512f0ff19d","Type":"ContainerDied","Data":"e50a38edde0f298df418766d799f723e9eb98535b06411fd585ac0b63762564c"} Oct 30 09:27:43.267743 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:43.267734 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fh6qf" event={"ID":"beaa426c-dd69-4ab1-8776-c2512f0ff19d","Type":"ContainerDied","Data":"ed1595ab6867c252981a1b83900495f2ad1508a090f5c61720a06f6b9e51f7ba"} Oct 30 09:27:43.271968 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:43.271944 4562 generic.go:356] "Generic (PLEG): container finished" podID="12bbbb56-5c34-4297-8ab0-2d25958adcf5" containerID="d2637bf8160a36f897b1c8a96ae391f7bd86a920bdea62aa2565264dcae218ea" exitCode=0 Oct 30 09:27:43.272095 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:43.271984 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2pgpc" Oct 30 09:27:43.272095 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:43.272023 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2pgpc" event={"ID":"12bbbb56-5c34-4297-8ab0-2d25958adcf5","Type":"ContainerDied","Data":"d2637bf8160a36f897b1c8a96ae391f7bd86a920bdea62aa2565264dcae218ea"} Oct 30 09:27:43.272095 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:43.272061 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2pgpc" event={"ID":"12bbbb56-5c34-4297-8ab0-2d25958adcf5","Type":"ContainerDied","Data":"503d158f6f6fff14178423f32e63f1fdc3cc49794df8eaa8fc6eed5b27ac03ef"} Oct 30 09:27:43.301590 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:43.301567 4562 scope.go:122] "RemoveContainer" containerID="b132c538dc26d7d540925a98019ccb6a34c340d12a8fae92c383925d25de4064" Oct 30 09:27:43.301908 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:27:43.301889 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b132c538dc26d7d540925a98019ccb6a34c340d12a8fae92c383925d25de4064\": container with ID starting with b132c538dc26d7d540925a98019ccb6a34c340d12a8fae92c383925d25de4064 not found: ID does not exist" containerID="b132c538dc26d7d540925a98019ccb6a34c340d12a8fae92c383925d25de4064" Oct 30 09:27:43.301969 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:43.301919 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b132c538dc26d7d540925a98019ccb6a34c340d12a8fae92c383925d25de4064"} err="failed to get container status \"b132c538dc26d7d540925a98019ccb6a34c340d12a8fae92c383925d25de4064\": rpc error: code = NotFound desc = could not find container \"b132c538dc26d7d540925a98019ccb6a34c340d12a8fae92c383925d25de4064\": container with ID starting with b132c538dc26d7d540925a98019ccb6a34c340d12a8fae92c383925d25de4064 not found: ID does not exist" Oct 30 09:27:43.301969 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:43.301937 4562 scope.go:122] "RemoveContainer" containerID="e50a38edde0f298df418766d799f723e9eb98535b06411fd585ac0b63762564c" Oct 30 09:27:43.313042 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:43.313022 4562 scope.go:122] "RemoveContainer" containerID="e50a38edde0f298df418766d799f723e9eb98535b06411fd585ac0b63762564c" Oct 30 09:27:43.313339 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:27:43.313313 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e50a38edde0f298df418766d799f723e9eb98535b06411fd585ac0b63762564c\": container with ID starting with e50a38edde0f298df418766d799f723e9eb98535b06411fd585ac0b63762564c not found: ID does not exist" containerID="e50a38edde0f298df418766d799f723e9eb98535b06411fd585ac0b63762564c" Oct 30 09:27:43.313433 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:43.313349 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e50a38edde0f298df418766d799f723e9eb98535b06411fd585ac0b63762564c"} err="failed to get container status \"e50a38edde0f298df418766d799f723e9eb98535b06411fd585ac0b63762564c\": rpc error: code = NotFound desc = could not find container \"e50a38edde0f298df418766d799f723e9eb98535b06411fd585ac0b63762564c\": container with ID starting with e50a38edde0f298df418766d799f723e9eb98535b06411fd585ac0b63762564c not found: ID does not exist" Oct 30 09:27:43.313433 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:43.313373 4562 scope.go:122] "RemoveContainer" containerID="d2637bf8160a36f897b1c8a96ae391f7bd86a920bdea62aa2565264dcae218ea" Oct 30 09:27:43.323805 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:43.323785 4562 scope.go:122] "RemoveContainer" containerID="d2637bf8160a36f897b1c8a96ae391f7bd86a920bdea62aa2565264dcae218ea" Oct 30 09:27:43.324084 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:27:43.324065 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2637bf8160a36f897b1c8a96ae391f7bd86a920bdea62aa2565264dcae218ea\": container with ID starting with d2637bf8160a36f897b1c8a96ae391f7bd86a920bdea62aa2565264dcae218ea not found: ID does not exist" containerID="d2637bf8160a36f897b1c8a96ae391f7bd86a920bdea62aa2565264dcae218ea" Oct 30 09:27:43.324149 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:43.324093 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2637bf8160a36f897b1c8a96ae391f7bd86a920bdea62aa2565264dcae218ea"} err="failed to get container status \"d2637bf8160a36f897b1c8a96ae391f7bd86a920bdea62aa2565264dcae218ea\": rpc error: code = NotFound desc = could not find container \"d2637bf8160a36f897b1c8a96ae391f7bd86a920bdea62aa2565264dcae218ea\": container with ID starting with d2637bf8160a36f897b1c8a96ae391f7bd86a920bdea62aa2565264dcae218ea not found: ID does not exist" Oct 30 09:27:43.358642 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:43.358531 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k7rc9"] Oct 30 09:27:43.365613 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:43.365592 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k7rc9"] Oct 30 09:27:43.403844 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:43.403822 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fh6qf"] Oct 30 09:27:43.411887 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:43.411864 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fh6qf"] Oct 30 09:27:43.448817 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:43.448796 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2pgpc"] Oct 30 09:27:43.455467 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:43.455443 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2pgpc"] Oct 30 09:27:43.876716 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:43.876683 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12bbbb56-5c34-4297-8ab0-2d25958adcf5" path="/var/lib/kubelet/pods/12bbbb56-5c34-4297-8ab0-2d25958adcf5/volumes" Oct 30 09:27:43.877177 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:43.877162 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="866a5fb8-5939-4f6a-b01f-060e32222484" path="/var/lib/kubelet/pods/866a5fb8-5939-4f6a-b01f-060e32222484/volumes" Oct 30 09:27:43.877607 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:43.877593 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99f76ea3-743e-4536-9443-2454af9db849" path="/var/lib/kubelet/pods/99f76ea3-743e-4536-9443-2454af9db849/volumes" Oct 30 09:27:43.878346 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:43.878330 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="beaa426c-dd69-4ab1-8776-c2512f0ff19d" path="/var/lib/kubelet/pods/beaa426c-dd69-4ab1-8776-c2512f0ff19d/volumes" Oct 30 09:27:46.647312 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:46.647271 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp"] Oct 30 09:27:46.648191 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:46.648164 4562 cpu_manager.go:414] "containerMap: removing container" podUID="beaa426c-dd69-4ab1-8776-c2512f0ff19d" containerName="istio-proxy" Oct 30 09:27:46.648349 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:46.648327 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="beaa426c-dd69-4ab1-8776-c2512f0ff19d" containerName="istio-proxy" Oct 30 09:27:46.648496 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:46.648375 4562 cpu_manager.go:414] "containerMap: removing container" podUID="99f76ea3-743e-4536-9443-2454af9db849" containerName="istio-proxy" Oct 30 09:27:46.648496 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:46.648395 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99f76ea3-743e-4536-9443-2454af9db849" containerName="istio-proxy" Oct 30 09:27:46.648496 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:46.648409 4562 cpu_manager.go:414] "containerMap: removing container" podUID="12bbbb56-5c34-4297-8ab0-2d25958adcf5" containerName="istio-proxy" Oct 30 09:27:46.648496 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:46.648415 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12bbbb56-5c34-4297-8ab0-2d25958adcf5" containerName="istio-proxy" Oct 30 09:27:46.648496 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:46.648425 4562 cpu_manager.go:414] "containerMap: removing container" podUID="866a5fb8-5939-4f6a-b01f-060e32222484" containerName="istio-proxy" Oct 30 09:27:46.648496 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:46.648430 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="866a5fb8-5939-4f6a-b01f-060e32222484" containerName="istio-proxy" Oct 30 09:27:46.648808 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:46.648562 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="beaa426c-dd69-4ab1-8776-c2512f0ff19d" containerName="istio-proxy" Oct 30 09:27:46.648808 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:46.648571 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="99f76ea3-743e-4536-9443-2454af9db849" containerName="istio-proxy" Oct 30 09:27:46.648808 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:46.648577 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="12bbbb56-5c34-4297-8ab0-2d25958adcf5" containerName="istio-proxy" Oct 30 09:27:46.648808 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:46.648585 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="866a5fb8-5939-4f6a-b01f-060e32222484" containerName="istio-proxy" Oct 30 09:27:46.653860 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:46.653843 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp" Oct 30 09:27:46.670945 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:46.670921 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-d3de9d-openshift-default-dockercfg-6cq7q\"" Oct 30 09:27:46.675526 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:46.675502 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp"] Oct 30 09:27:46.737045 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:46.737013 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c482f1ed-23a5-4cbb-a077-c907faac860a-istio-podinfo\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp\" (UID: \"c482f1ed-23a5-4cbb-a077-c907faac860a\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp" Oct 30 09:27:46.737202 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:46.737067 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c482f1ed-23a5-4cbb-a077-c907faac860a-istio-data\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp\" (UID: \"c482f1ed-23a5-4cbb-a077-c907faac860a\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp" Oct 30 09:27:46.737202 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:46.737164 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mg7m\" (UniqueName: \"kubernetes.io/projected/c482f1ed-23a5-4cbb-a077-c907faac860a-kube-api-access-6mg7m\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp\" (UID: \"c482f1ed-23a5-4cbb-a077-c907faac860a\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp" Oct 30 09:27:46.737276 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:46.737215 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c482f1ed-23a5-4cbb-a077-c907faac860a-istiod-ca-cert\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp\" (UID: \"c482f1ed-23a5-4cbb-a077-c907faac860a\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp" Oct 30 09:27:46.737312 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:46.737294 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c482f1ed-23a5-4cbb-a077-c907faac860a-credential-socket\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp\" (UID: \"c482f1ed-23a5-4cbb-a077-c907faac860a\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp" Oct 30 09:27:46.737350 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:46.737321 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c482f1ed-23a5-4cbb-a077-c907faac860a-istio-token\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp\" (UID: \"c482f1ed-23a5-4cbb-a077-c907faac860a\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp" Oct 30 09:27:46.737441 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:46.737427 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c482f1ed-23a5-4cbb-a077-c907faac860a-istio-envoy\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp\" (UID: \"c482f1ed-23a5-4cbb-a077-c907faac860a\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp" Oct 30 09:27:46.737476 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:46.737468 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c482f1ed-23a5-4cbb-a077-c907faac860a-workload-socket\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp\" (UID: \"c482f1ed-23a5-4cbb-a077-c907faac860a\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp" Oct 30 09:27:46.737563 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:46.737552 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c482f1ed-23a5-4cbb-a077-c907faac860a-workload-certs\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp\" (UID: \"c482f1ed-23a5-4cbb-a077-c907faac860a\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp" Oct 30 09:27:46.838558 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:46.838528 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c482f1ed-23a5-4cbb-a077-c907faac860a-workload-socket\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp\" (UID: \"c482f1ed-23a5-4cbb-a077-c907faac860a\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp" Oct 30 09:27:46.838723 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:46.838616 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c482f1ed-23a5-4cbb-a077-c907faac860a-workload-certs\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp\" (UID: \"c482f1ed-23a5-4cbb-a077-c907faac860a\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp" Oct 30 09:27:46.838723 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:46.838645 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c482f1ed-23a5-4cbb-a077-c907faac860a-istio-podinfo\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp\" (UID: \"c482f1ed-23a5-4cbb-a077-c907faac860a\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp" Oct 30 09:27:46.838723 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:46.838679 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c482f1ed-23a5-4cbb-a077-c907faac860a-istio-data\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp\" (UID: \"c482f1ed-23a5-4cbb-a077-c907faac860a\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp" Oct 30 09:27:46.838838 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:46.838734 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mg7m\" (UniqueName: \"kubernetes.io/projected/c482f1ed-23a5-4cbb-a077-c907faac860a-kube-api-access-6mg7m\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp\" (UID: \"c482f1ed-23a5-4cbb-a077-c907faac860a\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp" Oct 30 09:27:46.838838 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:46.838794 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c482f1ed-23a5-4cbb-a077-c907faac860a-istiod-ca-cert\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp\" (UID: \"c482f1ed-23a5-4cbb-a077-c907faac860a\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp" Oct 30 09:27:46.838948 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:46.838842 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c482f1ed-23a5-4cbb-a077-c907faac860a-credential-socket\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp\" (UID: \"c482f1ed-23a5-4cbb-a077-c907faac860a\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp" Oct 30 09:27:46.838948 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:46.838874 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c482f1ed-23a5-4cbb-a077-c907faac860a-istio-token\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp\" (UID: \"c482f1ed-23a5-4cbb-a077-c907faac860a\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp" Oct 30 09:27:46.839057 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:46.838974 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c482f1ed-23a5-4cbb-a077-c907faac860a-istio-envoy\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp\" (UID: \"c482f1ed-23a5-4cbb-a077-c907faac860a\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp" Oct 30 09:27:46.839057 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:46.838989 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c482f1ed-23a5-4cbb-a077-c907faac860a-workload-socket\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp\" (UID: \"c482f1ed-23a5-4cbb-a077-c907faac860a\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp" Oct 30 09:27:46.839165 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:46.839077 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c482f1ed-23a5-4cbb-a077-c907faac860a-istio-data\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp\" (UID: \"c482f1ed-23a5-4cbb-a077-c907faac860a\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp" Oct 30 09:27:46.839227 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:46.839200 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c482f1ed-23a5-4cbb-a077-c907faac860a-credential-socket\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp\" (UID: \"c482f1ed-23a5-4cbb-a077-c907faac860a\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp" Oct 30 09:27:46.839328 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:46.839307 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c482f1ed-23a5-4cbb-a077-c907faac860a-workload-certs\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp\" (UID: \"c482f1ed-23a5-4cbb-a077-c907faac860a\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp" Oct 30 09:27:46.839721 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:46.839698 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c482f1ed-23a5-4cbb-a077-c907faac860a-istiod-ca-cert\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp\" (UID: \"c482f1ed-23a5-4cbb-a077-c907faac860a\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp" Oct 30 09:27:46.841164 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:46.841140 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c482f1ed-23a5-4cbb-a077-c907faac860a-istio-envoy\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp\" (UID: \"c482f1ed-23a5-4cbb-a077-c907faac860a\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp" Oct 30 09:27:46.841253 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:46.841176 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c482f1ed-23a5-4cbb-a077-c907faac860a-istio-podinfo\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp\" (UID: \"c482f1ed-23a5-4cbb-a077-c907faac860a\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp" Oct 30 09:27:46.858674 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:46.858651 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c482f1ed-23a5-4cbb-a077-c907faac860a-istio-token\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp\" (UID: \"c482f1ed-23a5-4cbb-a077-c907faac860a\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp" Oct 30 09:27:46.858943 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:46.858922 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mg7m\" (UniqueName: \"kubernetes.io/projected/c482f1ed-23a5-4cbb-a077-c907faac860a-kube-api-access-6mg7m\") pod \"e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp\" (UID: \"c482f1ed-23a5-4cbb-a077-c907faac860a\") " pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp" Oct 30 09:27:46.967622 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:46.967598 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp" Oct 30 09:27:47.338423 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:47.320107 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:27:47.338423 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:47.320265 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:27:47.338423 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:47.320314 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:27:47.338423 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:47.331284 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp"] Oct 30 09:27:47.438411 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:47.417118 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp" event={"ID":"c482f1ed-23a5-4cbb-a077-c907faac860a","Type":"ContainerStarted","Data":"613fc9f2946802836accaa60df6826bf19fdc301b1b29d692104e4120bd61368"} Oct 30 09:27:48.437741 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:48.437699 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp" event={"ID":"c482f1ed-23a5-4cbb-a077-c907faac860a","Type":"ContainerStarted","Data":"5d7f89f3f4d2edba5bee12b91918478c155b8634625967fbc57b3372b1de93d8"} Oct 30 09:27:48.968762 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:48.968725 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp" Oct 30 09:27:48.970099 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:48.970068 4562 patch_prober.go:29] interesting pod/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 09:27:48.970219 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:48.970124 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp" podUID="c482f1ed-23a5-4cbb-a077-c907faac860a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 09:27:49.661405 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:49.660839 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp" podStartSLOduration=3.6608168169999997 podStartE2EDuration="3.660816817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:27:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:27:48.512797437 +0000 UTC m=+2028.774358425" watchObservedRunningTime="2026-10-30 09:27:49.660816817 +0000 UTC m=+2029.922377807" Oct 30 09:27:49.665611 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:49.665570 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-51e091-5ddbc647c9-rpb26"] Oct 30 09:27:49.670117 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:49.670099 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-51e091-5ddbc647c9-rpb26" Oct 30 09:27:49.685236 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:49.685207 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-51e091-5ddbc647c9-rpb26"] Oct 30 09:27:49.694274 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:49.694248 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-51e091-serving-cert\"" Oct 30 09:27:49.715648 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:49.715617 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5149468c-48b5-43ee-9eba-b3b43f41af6d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-51e091-5ddbc647c9-rpb26\" (UID: \"5149468c-48b5-43ee-9eba-b3b43f41af6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-51e091-5ddbc647c9-rpb26" Oct 30 09:27:49.715854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:49.715827 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8lmj\" (UniqueName: \"kubernetes.io/projected/5149468c-48b5-43ee-9eba-b3b43f41af6d-kube-api-access-d8lmj\") pod \"maas-api-e2e-ipp-b-51e091-5ddbc647c9-rpb26\" (UID: \"5149468c-48b5-43ee-9eba-b3b43f41af6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-51e091-5ddbc647c9-rpb26" Oct 30 09:27:49.716017 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:49.716001 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5149468c-48b5-43ee-9eba-b3b43f41af6d-metrics-certs\") pod \"maas-api-e2e-ipp-b-51e091-5ddbc647c9-rpb26\" (UID: \"5149468c-48b5-43ee-9eba-b3b43f41af6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-51e091-5ddbc647c9-rpb26" Oct 30 09:27:49.716082 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:49.716069 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5149468c-48b5-43ee-9eba-b3b43f41af6d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-51e091-5ddbc647c9-rpb26\" (UID: \"5149468c-48b5-43ee-9eba-b3b43f41af6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-51e091-5ddbc647c9-rpb26" Oct 30 09:27:49.817908 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:49.817873 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5149468c-48b5-43ee-9eba-b3b43f41af6d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-51e091-5ddbc647c9-rpb26\" (UID: \"5149468c-48b5-43ee-9eba-b3b43f41af6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-51e091-5ddbc647c9-rpb26" Oct 30 09:27:49.817908 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:49.817915 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8lmj\" (UniqueName: \"kubernetes.io/projected/5149468c-48b5-43ee-9eba-b3b43f41af6d-kube-api-access-d8lmj\") pod \"maas-api-e2e-ipp-b-51e091-5ddbc647c9-rpb26\" (UID: \"5149468c-48b5-43ee-9eba-b3b43f41af6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-51e091-5ddbc647c9-rpb26" Oct 30 09:27:49.818111 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:49.818085 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5149468c-48b5-43ee-9eba-b3b43f41af6d-metrics-certs\") pod \"maas-api-e2e-ipp-b-51e091-5ddbc647c9-rpb26\" (UID: \"5149468c-48b5-43ee-9eba-b3b43f41af6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-51e091-5ddbc647c9-rpb26" Oct 30 09:27:49.818250 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:49.818230 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5149468c-48b5-43ee-9eba-b3b43f41af6d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-51e091-5ddbc647c9-rpb26\" (UID: \"5149468c-48b5-43ee-9eba-b3b43f41af6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-51e091-5ddbc647c9-rpb26" Oct 30 09:27:49.818402 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:27:49.818368 4562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-51e091-serving-cert: secret "maas-api-e2e-ipp-b-51e091-serving-cert" not found Oct 30 09:27:49.818491 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:27:49.818453 4562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5149468c-48b5-43ee-9eba-b3b43f41af6d-maas-api-tls podName:5149468c-48b5-43ee-9eba-b3b43f41af6d nodeName:}" failed. No retries permitted until 2026-10-30 09:27:50.318432542 +0000 UTC m=+2030.579993520 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5149468c-48b5-43ee-9eba-b3b43f41af6d-maas-api-tls") pod "maas-api-e2e-ipp-b-51e091-5ddbc647c9-rpb26" (UID: "5149468c-48b5-43ee-9eba-b3b43f41af6d") : secret "maas-api-e2e-ipp-b-51e091-serving-cert" not found Oct 30 09:27:49.818698 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:49.818673 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5149468c-48b5-43ee-9eba-b3b43f41af6d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-51e091-5ddbc647c9-rpb26\" (UID: \"5149468c-48b5-43ee-9eba-b3b43f41af6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-51e091-5ddbc647c9-rpb26" Oct 30 09:27:49.820873 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:49.820850 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5149468c-48b5-43ee-9eba-b3b43f41af6d-metrics-certs\") pod \"maas-api-e2e-ipp-b-51e091-5ddbc647c9-rpb26\" (UID: \"5149468c-48b5-43ee-9eba-b3b43f41af6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-51e091-5ddbc647c9-rpb26" Oct 30 09:27:49.871465 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:49.871427 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8lmj\" (UniqueName: \"kubernetes.io/projected/5149468c-48b5-43ee-9eba-b3b43f41af6d-kube-api-access-d8lmj\") pod \"maas-api-e2e-ipp-b-51e091-5ddbc647c9-rpb26\" (UID: \"5149468c-48b5-43ee-9eba-b3b43f41af6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-51e091-5ddbc647c9-rpb26" Oct 30 09:27:49.976033 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:49.975983 4562 patch_prober.go:29] interesting pod/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 09:27:49.976347 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:49.976303 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp" podUID="c482f1ed-23a5-4cbb-a077-c907faac860a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 09:27:50.329930 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:50.329841 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5149468c-48b5-43ee-9eba-b3b43f41af6d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-51e091-5ddbc647c9-rpb26\" (UID: \"5149468c-48b5-43ee-9eba-b3b43f41af6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-51e091-5ddbc647c9-rpb26" Oct 30 09:27:50.332332 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:50.332306 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5149468c-48b5-43ee-9eba-b3b43f41af6d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-51e091-5ddbc647c9-rpb26\" (UID: \"5149468c-48b5-43ee-9eba-b3b43f41af6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-51e091-5ddbc647c9-rpb26" Oct 30 09:27:50.594458 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:50.590929 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-51e091-5ddbc647c9-rpb26" Oct 30 09:27:51.009210 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:50.995854 4562 patch_prober.go:29] interesting pod/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 09:27:51.009210 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:50.995914 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp" podUID="c482f1ed-23a5-4cbb-a077-c907faac860a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 09:27:51.042852 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:51.033398 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-51e091-5ddbc647c9-rpb26"] Oct 30 09:27:51.042852 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:27:51.033989 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5149468c_48b5_43ee_9eba_b3b43f41af6d.slice/crio-fd39808cc8001c772c88155a52bc2e95567c171ff534af56b7fec18f6cb74fc9 WatchSource:0}: Error finding container fd39808cc8001c772c88155a52bc2e95567c171ff534af56b7fec18f6cb74fc9: Status 404 returned error can't find the container with id fd39808cc8001c772c88155a52bc2e95567c171ff534af56b7fec18f6cb74fc9 Oct 30 09:27:51.480296 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:51.480259 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-51e091-5ddbc647c9-rpb26" event={"ID":"5149468c-48b5-43ee-9eba-b3b43f41af6d","Type":"ContainerStarted","Data":"fd39808cc8001c772c88155a52bc2e95567c171ff534af56b7fec18f6cb74fc9"} Oct 30 09:27:51.968497 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:51.968053 4562 patch_prober.go:29] interesting pod/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 09:27:51.968497 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:51.968120 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp" podUID="c482f1ed-23a5-4cbb-a077-c907faac860a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 09:27:52.271955 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:52.271864 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-9d39c0-648d978b78-446d4" Oct 30 09:27:52.488762 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:52.488726 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-51e091-5ddbc647c9-rpb26" event={"ID":"5149468c-48b5-43ee-9eba-b3b43f41af6d","Type":"ContainerStarted","Data":"799122ba9c236f204c0702b715ddc327b30d2516497c4a2ad8d12666e8a66d82"} Oct 30 09:27:52.488944 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:52.488790 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-51e091-5ddbc647c9-rpb26" Oct 30 09:27:52.531182 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:52.531087 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-51e091-5ddbc647c9-rpb26" podStartSLOduration=3.0707293509999998 podStartE2EDuration="3.531074959s" totalImagesPullingTime="460.345608ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:27:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:27:52.530133163 +0000 UTC m=+2032.791694152" watchObservedRunningTime="2026-10-30 09:27:52.531074959 +0000 UTC m=+2032.792635946" Oct 30 09:27:52.968302 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:52.968264 4562 patch_prober.go:29] interesting pod/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 09:27:52.968488 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:52.968333 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp" podUID="c482f1ed-23a5-4cbb-a077-c907faac860a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 09:27:53.297474 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:53.292226 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9d39c0-7fc7b8887f-rfr9x" Oct 30 09:27:53.990403 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:53.990086 4562 patch_prober.go:29] interesting pod/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 09:27:53.990403 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:53.990203 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp" podUID="c482f1ed-23a5-4cbb-a077-c907faac860a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 09:27:54.973957 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:54.973927 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp" Oct 30 09:27:54.974472 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:54.974450 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp" Oct 30 09:27:54.975247 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:54.975216 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp" Oct 30 09:27:58.510122 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:27:58.510081 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-51e091-5ddbc647c9-rpb26" Oct 30 09:27:58.605614 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:27:58.605553 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:28:02.635451 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:02.635409 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d87c85-5d8c96c5-fm4dg"] Oct 30 09:28:02.642889 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:02.642862 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d87c85-5d8c96c5-fm4dg" Oct 30 09:28:02.661493 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:02.661456 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d87c85-5d8c96c5-fm4dg"] Oct 30 09:28:02.676069 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:02.676030 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-d87c85\"" Oct 30 09:28:02.678941 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:02.678918 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-d87c85-dockercfg-ll6st\"" Oct 30 09:28:02.800574 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:02.800528 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vss87\" (UniqueName: \"kubernetes.io/projected/7110103e-fba8-4baf-a62e-9942f43f0f71-kube-api-access-vss87\") pod \"payload-pre-processing-e2e-ipp-a-d87c85-5d8c96c5-fm4dg\" (UID: \"7110103e-fba8-4baf-a62e-9942f43f0f71\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d87c85-5d8c96c5-fm4dg" Oct 30 09:28:02.827823 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:02.827784 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7110103e-fba8-4baf-a62e-9942f43f0f71-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d87c85-5d8c96c5-fm4dg\" (UID: \"7110103e-fba8-4baf-a62e-9942f43f0f71\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d87c85-5d8c96c5-fm4dg" Oct 30 09:28:02.984404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:02.974162 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vss87\" (UniqueName: \"kubernetes.io/projected/7110103e-fba8-4baf-a62e-9942f43f0f71-kube-api-access-vss87\") pod \"payload-pre-processing-e2e-ipp-a-d87c85-5d8c96c5-fm4dg\" (UID: \"7110103e-fba8-4baf-a62e-9942f43f0f71\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d87c85-5d8c96c5-fm4dg" Oct 30 09:28:03.042261 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:02.982772 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d87c85-75b5984778-bpxt6"] Oct 30 09:28:03.048809 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:03.048730 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vss87\" (UniqueName: \"kubernetes.io/projected/7110103e-fba8-4baf-a62e-9942f43f0f71-kube-api-access-vss87\") pod \"payload-pre-processing-e2e-ipp-a-d87c85-5d8c96c5-fm4dg\" (UID: \"7110103e-fba8-4baf-a62e-9942f43f0f71\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d87c85-5d8c96c5-fm4dg" Oct 30 09:28:03.049005 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:03.048884 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d87c85-75b5984778-bpxt6"] Oct 30 09:28:03.049005 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:03.048998 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d87c85-75b5984778-bpxt6" Oct 30 09:28:03.070421 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:03.067241 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7110103e-fba8-4baf-a62e-9942f43f0f71-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d87c85-5d8c96c5-fm4dg\" (UID: \"7110103e-fba8-4baf-a62e-9942f43f0f71\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d87c85-5d8c96c5-fm4dg" Oct 30 09:28:03.070421 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:03.068545 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7110103e-fba8-4baf-a62e-9942f43f0f71-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d87c85-5d8c96c5-fm4dg\" (UID: \"7110103e-fba8-4baf-a62e-9942f43f0f71\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d87c85-5d8c96c5-fm4dg" Oct 30 09:28:03.168649 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:03.168614 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4fea576-dd2f-4ab6-bcaa-ed3169d2f4dc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d87c85-75b5984778-bpxt6\" (UID: \"c4fea576-dd2f-4ab6-bcaa-ed3169d2f4dc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d87c85-75b5984778-bpxt6" Oct 30 09:28:03.169160 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:03.169133 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9wgm\" (UniqueName: \"kubernetes.io/projected/c4fea576-dd2f-4ab6-bcaa-ed3169d2f4dc-kube-api-access-v9wgm\") pod \"payload-processing-e2e-ipp-a-d87c85-75b5984778-bpxt6\" (UID: \"c4fea576-dd2f-4ab6-bcaa-ed3169d2f4dc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d87c85-75b5984778-bpxt6" Oct 30 09:28:03.264184 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:03.264092 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d87c85-5d8c96c5-fm4dg" Oct 30 09:28:03.271008 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:03.270977 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9wgm\" (UniqueName: \"kubernetes.io/projected/c4fea576-dd2f-4ab6-bcaa-ed3169d2f4dc-kube-api-access-v9wgm\") pod \"payload-processing-e2e-ipp-a-d87c85-75b5984778-bpxt6\" (UID: \"c4fea576-dd2f-4ab6-bcaa-ed3169d2f4dc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d87c85-75b5984778-bpxt6" Oct 30 09:28:03.271198 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:03.271183 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4fea576-dd2f-4ab6-bcaa-ed3169d2f4dc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d87c85-75b5984778-bpxt6\" (UID: \"c4fea576-dd2f-4ab6-bcaa-ed3169d2f4dc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d87c85-75b5984778-bpxt6" Oct 30 09:28:03.272063 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:03.272038 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4fea576-dd2f-4ab6-bcaa-ed3169d2f4dc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d87c85-75b5984778-bpxt6\" (UID: \"c4fea576-dd2f-4ab6-bcaa-ed3169d2f4dc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d87c85-75b5984778-bpxt6" Oct 30 09:28:03.293369 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:03.293341 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9wgm\" (UniqueName: \"kubernetes.io/projected/c4fea576-dd2f-4ab6-bcaa-ed3169d2f4dc-kube-api-access-v9wgm\") pod \"payload-processing-e2e-ipp-a-d87c85-75b5984778-bpxt6\" (UID: \"c4fea576-dd2f-4ab6-bcaa-ed3169d2f4dc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d87c85-75b5984778-bpxt6" Oct 30 09:28:03.398626 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:03.398585 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d87c85-75b5984778-bpxt6" Oct 30 09:28:03.572897 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:28:03.572860 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7110103e_fba8_4baf_a62e_9942f43f0f71.slice/crio-87862e32c09abce2efa92696f0e97390f144b33150b5845989b6805b5b93af0c WatchSource:0}: Error finding container 87862e32c09abce2efa92696f0e97390f144b33150b5845989b6805b5b93af0c: Status 404 returned error can't find the container with id 87862e32c09abce2efa92696f0e97390f144b33150b5845989b6805b5b93af0c Oct 30 09:28:03.573829 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:03.573802 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d87c85-5d8c96c5-fm4dg"] Oct 30 09:28:03.709255 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:28:03.709218 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4fea576_dd2f_4ab6_bcaa_ed3169d2f4dc.slice/crio-cddcbf66f9fa9a0bed792d28a875bd7d297fc87632fe12ae997f3b71ca88d652 WatchSource:0}: Error finding container cddcbf66f9fa9a0bed792d28a875bd7d297fc87632fe12ae997f3b71ca88d652: Status 404 returned error can't find the container with id cddcbf66f9fa9a0bed792d28a875bd7d297fc87632fe12ae997f3b71ca88d652 Oct 30 09:28:03.711115 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:03.711089 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d87c85-75b5984778-bpxt6"] Oct 30 09:28:04.588276 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:04.588239 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d87c85-5d8c96c5-fm4dg" event={"ID":"7110103e-fba8-4baf-a62e-9942f43f0f71","Type":"ContainerStarted","Data":"924b566505798ec5081d8638c3529ade417c2d4e2d126a710993732e23a32796"} Oct 30 09:28:04.588521 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:04.588299 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d87c85-5d8c96c5-fm4dg" event={"ID":"7110103e-fba8-4baf-a62e-9942f43f0f71","Type":"ContainerStarted","Data":"87862e32c09abce2efa92696f0e97390f144b33150b5845989b6805b5b93af0c"} Oct 30 09:28:04.588521 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:04.588423 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d87c85-5d8c96c5-fm4dg" Oct 30 09:28:04.590255 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:04.590219 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d87c85-75b5984778-bpxt6" event={"ID":"c4fea576-dd2f-4ab6-bcaa-ed3169d2f4dc","Type":"ContainerStarted","Data":"97cd135e19a53bd5a3683f9cd092e1cf72cb567d92e46b002de3bac62e2399e7"} Oct 30 09:28:04.590375 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:04.590274 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d87c85-75b5984778-bpxt6" event={"ID":"c4fea576-dd2f-4ab6-bcaa-ed3169d2f4dc","Type":"ContainerStarted","Data":"cddcbf66f9fa9a0bed792d28a875bd7d297fc87632fe12ae997f3b71ca88d652"} Oct 30 09:28:04.590375 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:04.590297 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d87c85-75b5984778-bpxt6" Oct 30 09:28:04.632801 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:04.632754 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d87c85-5d8c96c5-fm4dg" podStartSLOduration=2.632739726 podStartE2EDuration="2.632739726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:28:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:28:04.631058848 +0000 UTC m=+2044.892619837" watchObservedRunningTime="2026-10-30 09:28:04.632739726 +0000 UTC m=+2044.894300714" Oct 30 09:28:04.674659 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:04.674614 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-d87c85-75b5984778-bpxt6" podStartSLOduration=2.674599391 podStartE2EDuration="2.674599391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:28:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:28:04.671282961 +0000 UTC m=+2044.932843949" watchObservedRunningTime="2026-10-30 09:28:04.674599391 +0000 UTC m=+2044.936160397" Oct 30 09:28:15.600200 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:15.600170 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d87c85-5d8c96c5-fm4dg" Oct 30 09:28:15.600600 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:15.600239 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d87c85-75b5984778-bpxt6" Oct 30 09:28:19.110796 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:19.110753 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-20f71c-6988847554-ld88l"] Oct 30 09:28:19.149145 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:19.149114 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-20f71c-645d97fb95-c5wrx"] Oct 30 09:28:22.001882 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:22.001849 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-51e091-7956b856cb-qfl8z"] Oct 30 09:28:22.072623 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:22.067430 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-51e091-5f4b488499-clrjg"] Oct 30 09:28:24.111153 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:24.111111 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-20f71c-6988847554-ld88l" podUID="e40a8439-563f-46dc-b272-17c688df5b6f" containerName="payload-processing" containerID="cri-o://d7ba8a16dfe84c056a59d5ff0a96d4a20965d819ea915153d09529c996e585c0" gracePeriod=25 Oct 30 09:28:24.149487 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:24.149455 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-20f71c-645d97fb95-c5wrx" podUID="3b314f31-c731-467a-bee2-79f08cbc0573" containerName="payload-pre-processing" containerID="cri-o://d50285693133789a0fbcacd42560181b256653e3e077924c8bfc2dab3c628a3f" gracePeriod=25 Oct 30 09:28:24.367339 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:28:24.367232 4562 logging.go:55] [core] [Channel #2955 SubChannel #2956]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.135:50052", ServerName: "10.133.0.135:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.135:50052: connect: connection refused" Oct 30 09:28:24.424770 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:24.424741 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-20f71c-6988847554-ld88l" Oct 30 09:28:24.480029 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:24.480005 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-20f71c-645d97fb95-c5wrx" Oct 30 09:28:24.564375 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:24.564337 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e40a8439-563f-46dc-b272-17c688df5b6f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e40a8439-563f-46dc-b272-17c688df5b6f-plugins-config-volume\") pod \"e40a8439-563f-46dc-b272-17c688df5b6f\" (UID: \"e40a8439-563f-46dc-b272-17c688df5b6f\") " Oct 30 09:28:24.564598 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:24.564563 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e40a8439-563f-46dc-b272-17c688df5b6f-kube-api-access-25g5g\" (UniqueName: \"kubernetes.io/projected/e40a8439-563f-46dc-b272-17c688df5b6f-kube-api-access-25g5g\") pod \"e40a8439-563f-46dc-b272-17c688df5b6f\" (UID: \"e40a8439-563f-46dc-b272-17c688df5b6f\") " Oct 30 09:28:24.564762 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:24.564740 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e40a8439-563f-46dc-b272-17c688df5b6f-plugins-config-volume" pod "e40a8439-563f-46dc-b272-17c688df5b6f" (UID: "e40a8439-563f-46dc-b272-17c688df5b6f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:28:24.565240 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:24.565207 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e40a8439-563f-46dc-b272-17c688df5b6f-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:28:24.566879 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:24.566856 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e40a8439-563f-46dc-b272-17c688df5b6f-kube-api-access-25g5g" pod "e40a8439-563f-46dc-b272-17c688df5b6f" (UID: "e40a8439-563f-46dc-b272-17c688df5b6f"). InnerVolumeSpecName "kube-api-access-25g5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:28:24.666230 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:24.666157 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b314f31-c731-467a-bee2-79f08cbc0573-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b314f31-c731-467a-bee2-79f08cbc0573-plugins-config-volume\") pod \"3b314f31-c731-467a-bee2-79f08cbc0573\" (UID: \"3b314f31-c731-467a-bee2-79f08cbc0573\") " Oct 30 09:28:24.666365 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:24.666322 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b314f31-c731-467a-bee2-79f08cbc0573-kube-api-access-sd2hm\" (UniqueName: \"kubernetes.io/projected/3b314f31-c731-467a-bee2-79f08cbc0573-kube-api-access-sd2hm\") pod \"3b314f31-c731-467a-bee2-79f08cbc0573\" (UID: \"3b314f31-c731-467a-bee2-79f08cbc0573\") " Oct 30 09:28:24.666630 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:24.666607 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b314f31-c731-467a-bee2-79f08cbc0573-plugins-config-volume" pod "3b314f31-c731-467a-bee2-79f08cbc0573" (UID: "3b314f31-c731-467a-bee2-79f08cbc0573"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:28:24.666913 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:24.666897 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25g5g\" (UniqueName: \"kubernetes.io/projected/e40a8439-563f-46dc-b272-17c688df5b6f-kube-api-access-25g5g\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:28:24.666948 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:24.666918 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b314f31-c731-467a-bee2-79f08cbc0573-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:28:24.668423 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:24.668399 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b314f31-c731-467a-bee2-79f08cbc0573-kube-api-access-sd2hm" pod "3b314f31-c731-467a-bee2-79f08cbc0573" (UID: "3b314f31-c731-467a-bee2-79f08cbc0573"). InnerVolumeSpecName "kube-api-access-sd2hm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:28:24.768316 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:24.768288 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sd2hm\" (UniqueName: \"kubernetes.io/projected/3b314f31-c731-467a-bee2-79f08cbc0573-kube-api-access-sd2hm\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:28:24.842931 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:24.842901 4562 generic.go:356] "Generic (PLEG): container finished" podID="e40a8439-563f-46dc-b272-17c688df5b6f" containerID="d7ba8a16dfe84c056a59d5ff0a96d4a20965d819ea915153d09529c996e585c0" exitCode=0 Oct 30 09:28:24.843091 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:24.842945 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-20f71c-6988847554-ld88l" Oct 30 09:28:24.843091 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:24.842989 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-20f71c-6988847554-ld88l" event={"ID":"e40a8439-563f-46dc-b272-17c688df5b6f","Type":"ContainerDied","Data":"d7ba8a16dfe84c056a59d5ff0a96d4a20965d819ea915153d09529c996e585c0"} Oct 30 09:28:24.843091 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:24.843075 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-20f71c-6988847554-ld88l" event={"ID":"e40a8439-563f-46dc-b272-17c688df5b6f","Type":"ContainerDied","Data":"62ecd3d47996d7fc058b222c6fb7b108ebb9e6fa517c28aa8d0654f81857fb84"} Oct 30 09:28:24.843262 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:24.843101 4562 scope.go:122] "RemoveContainer" containerID="d7ba8a16dfe84c056a59d5ff0a96d4a20965d819ea915153d09529c996e585c0" Oct 30 09:28:24.844760 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:24.844733 4562 generic.go:356] "Generic (PLEG): container finished" podID="3b314f31-c731-467a-bee2-79f08cbc0573" containerID="d50285693133789a0fbcacd42560181b256653e3e077924c8bfc2dab3c628a3f" exitCode=0 Oct 30 09:28:24.844889 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:24.844780 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-20f71c-645d97fb95-c5wrx" Oct 30 09:28:24.844889 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:24.844829 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-20f71c-645d97fb95-c5wrx" event={"ID":"3b314f31-c731-467a-bee2-79f08cbc0573","Type":"ContainerDied","Data":"d50285693133789a0fbcacd42560181b256653e3e077924c8bfc2dab3c628a3f"} Oct 30 09:28:24.844889 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:24.844878 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-20f71c-645d97fb95-c5wrx" event={"ID":"3b314f31-c731-467a-bee2-79f08cbc0573","Type":"ContainerDied","Data":"3599241cf86aa7965ab7fda17040d47e653dfefe9879522f4966677ed1f79314"} Oct 30 09:28:24.857585 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:24.856217 4562 scope.go:122] "RemoveContainer" containerID="d7ba8a16dfe84c056a59d5ff0a96d4a20965d819ea915153d09529c996e585c0" Oct 30 09:28:24.857585 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:28:24.856608 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7ba8a16dfe84c056a59d5ff0a96d4a20965d819ea915153d09529c996e585c0\": container with ID starting with d7ba8a16dfe84c056a59d5ff0a96d4a20965d819ea915153d09529c996e585c0 not found: ID does not exist" containerID="d7ba8a16dfe84c056a59d5ff0a96d4a20965d819ea915153d09529c996e585c0" Oct 30 09:28:24.857585 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:24.856650 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7ba8a16dfe84c056a59d5ff0a96d4a20965d819ea915153d09529c996e585c0"} err="failed to get container status \"d7ba8a16dfe84c056a59d5ff0a96d4a20965d819ea915153d09529c996e585c0\": rpc error: code = NotFound desc = could not find container \"d7ba8a16dfe84c056a59d5ff0a96d4a20965d819ea915153d09529c996e585c0\": container with ID starting with d7ba8a16dfe84c056a59d5ff0a96d4a20965d819ea915153d09529c996e585c0 not found: ID does not exist" Oct 30 09:28:24.857585 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:24.856673 4562 scope.go:122] "RemoveContainer" containerID="d50285693133789a0fbcacd42560181b256653e3e077924c8bfc2dab3c628a3f" Oct 30 09:28:24.865829 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:24.865810 4562 scope.go:122] "RemoveContainer" containerID="d50285693133789a0fbcacd42560181b256653e3e077924c8bfc2dab3c628a3f" Oct 30 09:28:24.866084 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:28:24.866067 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d50285693133789a0fbcacd42560181b256653e3e077924c8bfc2dab3c628a3f\": container with ID starting with d50285693133789a0fbcacd42560181b256653e3e077924c8bfc2dab3c628a3f not found: ID does not exist" containerID="d50285693133789a0fbcacd42560181b256653e3e077924c8bfc2dab3c628a3f" Oct 30 09:28:24.866141 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:24.866090 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d50285693133789a0fbcacd42560181b256653e3e077924c8bfc2dab3c628a3f"} err="failed to get container status \"d50285693133789a0fbcacd42560181b256653e3e077924c8bfc2dab3c628a3f\": rpc error: code = NotFound desc = could not find container \"d50285693133789a0fbcacd42560181b256653e3e077924c8bfc2dab3c628a3f\": container with ID starting with d50285693133789a0fbcacd42560181b256653e3e077924c8bfc2dab3c628a3f not found: ID does not exist" Oct 30 09:28:24.898251 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:24.898224 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-20f71c-645d97fb95-c5wrx"] Oct 30 09:28:24.911434 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:24.911409 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-20f71c-645d97fb95-c5wrx"] Oct 30 09:28:24.939275 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:24.939254 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-20f71c-6988847554-ld88l"] Oct 30 09:28:24.944599 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:24.944573 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-20f71c-6988847554-ld88l"] Oct 30 09:28:25.376901 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:25.376327 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-20f71c-645d97fb95-c5wrx" podUID="3b314f31-c731-467a-bee2-79f08cbc0573" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.135:50052\" within 1s: context deadline exceeded" Oct 30 09:28:25.376901 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:28:25.376782 4562 logging.go:55] [core] [Channel #2955 SubChannel #2956]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.135:50052", ServerName: "10.133.0.135:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.135:50052: operation was canceled" Oct 30 09:28:25.875427 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:25.875375 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b314f31-c731-467a-bee2-79f08cbc0573" path="/var/lib/kubelet/pods/3b314f31-c731-467a-bee2-79f08cbc0573/volumes" Oct 30 09:28:25.875885 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:25.875871 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e40a8439-563f-46dc-b272-17c688df5b6f" path="/var/lib/kubelet/pods/e40a8439-563f-46dc-b272-17c688df5b6f/volumes" Oct 30 09:28:27.003064 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:27.003021 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-51e091-7956b856cb-qfl8z" podUID="c1adfa80-f01a-40cb-bd1c-d1e98873d077" containerName="payload-processing" containerID="cri-o://20ced906e968a7f1bae8a63909b984d1b31fe8027bc207ed4b2d1340787306ca" gracePeriod=25 Oct 30 09:28:27.068562 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:27.068509 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-51e091-5f4b488499-clrjg" podUID="633dc628-d461-4b26-874d-1bead2cea5b9" containerName="payload-pre-processing" containerID="cri-o://d79daf111a47049e629ed1fc4c34a2dccd334e7cd396721c1955877370345b04" gracePeriod=25 Oct 30 09:28:27.354805 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:27.354768 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-51e091-7956b856cb-qfl8z" Oct 30 09:28:27.408465 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:27.408438 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-51e091-5f4b488499-clrjg" Oct 30 09:28:27.469038 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:27.469006 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/633dc628-d461-4b26-874d-1bead2cea5b9-kube-api-access-tvsdq\" (UniqueName: \"kubernetes.io/projected/633dc628-d461-4b26-874d-1bead2cea5b9-kube-api-access-tvsdq\") pod \"633dc628-d461-4b26-874d-1bead2cea5b9\" (UID: \"633dc628-d461-4b26-874d-1bead2cea5b9\") " Oct 30 09:28:27.469215 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:27.469181 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1adfa80-f01a-40cb-bd1c-d1e98873d077-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1adfa80-f01a-40cb-bd1c-d1e98873d077-plugins-config-volume\") pod \"c1adfa80-f01a-40cb-bd1c-d1e98873d077\" (UID: \"c1adfa80-f01a-40cb-bd1c-d1e98873d077\") " Oct 30 09:28:27.469278 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:27.469263 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1adfa80-f01a-40cb-bd1c-d1e98873d077-kube-api-access-wmm9s\" (UniqueName: \"kubernetes.io/projected/c1adfa80-f01a-40cb-bd1c-d1e98873d077-kube-api-access-wmm9s\") pod \"c1adfa80-f01a-40cb-bd1c-d1e98873d077\" (UID: \"c1adfa80-f01a-40cb-bd1c-d1e98873d077\") " Oct 30 09:28:27.469678 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:27.469441 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/633dc628-d461-4b26-874d-1bead2cea5b9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/633dc628-d461-4b26-874d-1bead2cea5b9-plugins-config-volume\") pod \"633dc628-d461-4b26-874d-1bead2cea5b9\" (UID: \"633dc628-d461-4b26-874d-1bead2cea5b9\") " Oct 30 09:28:27.469678 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:27.469596 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1adfa80-f01a-40cb-bd1c-d1e98873d077-plugins-config-volume" pod "c1adfa80-f01a-40cb-bd1c-d1e98873d077" (UID: "c1adfa80-f01a-40cb-bd1c-d1e98873d077"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:28:27.470017 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:27.469993 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/633dc628-d461-4b26-874d-1bead2cea5b9-plugins-config-volume" pod "633dc628-d461-4b26-874d-1bead2cea5b9" (UID: "633dc628-d461-4b26-874d-1bead2cea5b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:28:27.470138 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:27.470117 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1adfa80-f01a-40cb-bd1c-d1e98873d077-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:28:27.470208 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:27.470144 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/633dc628-d461-4b26-874d-1bead2cea5b9-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:28:27.472117 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:27.472083 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1adfa80-f01a-40cb-bd1c-d1e98873d077-kube-api-access-wmm9s" pod "c1adfa80-f01a-40cb-bd1c-d1e98873d077" (UID: "c1adfa80-f01a-40cb-bd1c-d1e98873d077"). InnerVolumeSpecName "kube-api-access-wmm9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:28:27.472758 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:27.472732 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/633dc628-d461-4b26-874d-1bead2cea5b9-kube-api-access-tvsdq" pod "633dc628-d461-4b26-874d-1bead2cea5b9" (UID: "633dc628-d461-4b26-874d-1bead2cea5b9"). InnerVolumeSpecName "kube-api-access-tvsdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:28:27.571354 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:27.571282 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmm9s\" (UniqueName: \"kubernetes.io/projected/c1adfa80-f01a-40cb-bd1c-d1e98873d077-kube-api-access-wmm9s\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:28:27.571354 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:27.571308 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvsdq\" (UniqueName: \"kubernetes.io/projected/633dc628-d461-4b26-874d-1bead2cea5b9-kube-api-access-tvsdq\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:28:27.931562 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:27.931461 4562 generic.go:356] "Generic (PLEG): container finished" podID="c1adfa80-f01a-40cb-bd1c-d1e98873d077" containerID="20ced906e968a7f1bae8a63909b984d1b31fe8027bc207ed4b2d1340787306ca" exitCode=0 Oct 30 09:28:27.931562 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:27.931489 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-51e091-7956b856cb-qfl8z" Oct 30 09:28:27.931562 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:27.931535 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-51e091-7956b856cb-qfl8z" event={"ID":"c1adfa80-f01a-40cb-bd1c-d1e98873d077","Type":"ContainerDied","Data":"20ced906e968a7f1bae8a63909b984d1b31fe8027bc207ed4b2d1340787306ca"} Oct 30 09:28:27.931858 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:27.931586 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-51e091-7956b856cb-qfl8z" event={"ID":"c1adfa80-f01a-40cb-bd1c-d1e98873d077","Type":"ContainerDied","Data":"33c342e3f978d08acf3faf3fd839b2993ac43dd4abf26f8ec0971a61a3f263b2"} Oct 30 09:28:27.931858 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:27.931616 4562 scope.go:122] "RemoveContainer" containerID="20ced906e968a7f1bae8a63909b984d1b31fe8027bc207ed4b2d1340787306ca" Oct 30 09:28:27.933219 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:27.933148 4562 generic.go:356] "Generic (PLEG): container finished" podID="633dc628-d461-4b26-874d-1bead2cea5b9" containerID="d79daf111a47049e629ed1fc4c34a2dccd334e7cd396721c1955877370345b04" exitCode=0 Oct 30 09:28:27.933219 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:27.933190 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-51e091-5f4b488499-clrjg" Oct 30 09:28:27.933347 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:27.933247 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-51e091-5f4b488499-clrjg" event={"ID":"633dc628-d461-4b26-874d-1bead2cea5b9","Type":"ContainerDied","Data":"d79daf111a47049e629ed1fc4c34a2dccd334e7cd396721c1955877370345b04"} Oct 30 09:28:27.933347 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:27.933299 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-51e091-5f4b488499-clrjg" event={"ID":"633dc628-d461-4b26-874d-1bead2cea5b9","Type":"ContainerDied","Data":"dbca6c2c664a47c129312439404a9016a6d69f7972f71b011c9a7a6c465eb229"} Oct 30 09:28:27.945213 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:27.945194 4562 scope.go:122] "RemoveContainer" containerID="20ced906e968a7f1bae8a63909b984d1b31fe8027bc207ed4b2d1340787306ca" Oct 30 09:28:27.945530 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:28:27.945511 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20ced906e968a7f1bae8a63909b984d1b31fe8027bc207ed4b2d1340787306ca\": container with ID starting with 20ced906e968a7f1bae8a63909b984d1b31fe8027bc207ed4b2d1340787306ca not found: ID does not exist" containerID="20ced906e968a7f1bae8a63909b984d1b31fe8027bc207ed4b2d1340787306ca" Oct 30 09:28:27.945600 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:27.945539 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20ced906e968a7f1bae8a63909b984d1b31fe8027bc207ed4b2d1340787306ca"} err="failed to get container status \"20ced906e968a7f1bae8a63909b984d1b31fe8027bc207ed4b2d1340787306ca\": rpc error: code = NotFound desc = could not find container \"20ced906e968a7f1bae8a63909b984d1b31fe8027bc207ed4b2d1340787306ca\": container with ID starting with 20ced906e968a7f1bae8a63909b984d1b31fe8027bc207ed4b2d1340787306ca not found: ID does not exist" Oct 30 09:28:27.945600 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:27.945563 4562 scope.go:122] "RemoveContainer" containerID="d79daf111a47049e629ed1fc4c34a2dccd334e7cd396721c1955877370345b04" Oct 30 09:28:27.954166 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:27.954151 4562 scope.go:122] "RemoveContainer" containerID="d79daf111a47049e629ed1fc4c34a2dccd334e7cd396721c1955877370345b04" Oct 30 09:28:27.954452 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:28:27.954433 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d79daf111a47049e629ed1fc4c34a2dccd334e7cd396721c1955877370345b04\": container with ID starting with d79daf111a47049e629ed1fc4c34a2dccd334e7cd396721c1955877370345b04 not found: ID does not exist" containerID="d79daf111a47049e629ed1fc4c34a2dccd334e7cd396721c1955877370345b04" Oct 30 09:28:27.954515 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:27.954457 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d79daf111a47049e629ed1fc4c34a2dccd334e7cd396721c1955877370345b04"} err="failed to get container status \"d79daf111a47049e629ed1fc4c34a2dccd334e7cd396721c1955877370345b04\": rpc error: code = NotFound desc = could not find container \"d79daf111a47049e629ed1fc4c34a2dccd334e7cd396721c1955877370345b04\": container with ID starting with d79daf111a47049e629ed1fc4c34a2dccd334e7cd396721c1955877370345b04 not found: ID does not exist" Oct 30 09:28:27.990800 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:27.990759 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-51e091-5f4b488499-clrjg"] Oct 30 09:28:27.998910 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:27.998883 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-51e091-5f4b488499-clrjg"] Oct 30 09:28:28.034352 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:28.034327 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-51e091-7956b856cb-qfl8z"] Oct 30 09:28:28.039068 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:28.039047 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-51e091-7956b856cb-qfl8z"] Oct 30 09:28:28.623618 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:28:28.623575 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:28:29.874861 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:29.874828 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="633dc628-d461-4b26-874d-1bead2cea5b9" path="/var/lib/kubelet/pods/633dc628-d461-4b26-874d-1bead2cea5b9/volumes" Oct 30 09:28:29.875325 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:29.875257 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1adfa80-f01a-40cb-bd1c-d1e98873d077" path="/var/lib/kubelet/pods/c1adfa80-f01a-40cb-bd1c-d1e98873d077/volumes" Oct 30 09:28:41.505570 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:41.505537 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-51e091-769f5d944b-dgc2p"] Oct 30 09:28:41.506167 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:41.506153 4562 cpu_manager.go:414] "containerMap: removing container" podUID="633dc628-d461-4b26-874d-1bead2cea5b9" containerName="payload-pre-processing" Oct 30 09:28:41.506220 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:41.506168 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="633dc628-d461-4b26-874d-1bead2cea5b9" containerName="payload-pre-processing" Oct 30 09:28:41.506220 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:41.506188 4562 cpu_manager.go:414] "containerMap: removing container" podUID="e40a8439-563f-46dc-b272-17c688df5b6f" containerName="payload-processing" Oct 30 09:28:41.506220 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:41.506193 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e40a8439-563f-46dc-b272-17c688df5b6f" containerName="payload-processing" Oct 30 09:28:41.506220 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:41.506202 4562 cpu_manager.go:414] "containerMap: removing container" podUID="3b314f31-c731-467a-bee2-79f08cbc0573" containerName="payload-pre-processing" Oct 30 09:28:41.506220 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:41.506207 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b314f31-c731-467a-bee2-79f08cbc0573" containerName="payload-pre-processing" Oct 30 09:28:41.506404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:41.506238 4562 cpu_manager.go:414] "containerMap: removing container" podUID="c1adfa80-f01a-40cb-bd1c-d1e98873d077" containerName="payload-processing" Oct 30 09:28:41.506404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:41.506244 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1adfa80-f01a-40cb-bd1c-d1e98873d077" containerName="payload-processing" Oct 30 09:28:41.506404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:41.506322 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="633dc628-d461-4b26-874d-1bead2cea5b9" containerName="payload-pre-processing" Oct 30 09:28:41.506404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:41.506333 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="e40a8439-563f-46dc-b272-17c688df5b6f" containerName="payload-processing" Oct 30 09:28:41.506404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:41.506339 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b314f31-c731-467a-bee2-79f08cbc0573" containerName="payload-pre-processing" Oct 30 09:28:41.506404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:41.506349 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1adfa80-f01a-40cb-bd1c-d1e98873d077" containerName="payload-processing" Oct 30 09:28:41.511523 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:41.511502 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-51e091-769f5d944b-dgc2p" Oct 30 09:28:41.539835 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:41.539799 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-51e091-769f5d944b-dgc2p"] Oct 30 09:28:41.543714 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:41.543685 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-51e091-dockercfg-cr8jn\"" Oct 30 09:28:41.543884 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:41.543826 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-51e091\"" Oct 30 09:28:41.560192 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:41.560159 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plgvm\" (UniqueName: \"kubernetes.io/projected/94ef492d-56fb-4e63-9044-063e1ffe8b3a-kube-api-access-plgvm\") pod \"payload-processing-e2e-ipp-b-51e091-769f5d944b-dgc2p\" (UID: \"94ef492d-56fb-4e63-9044-063e1ffe8b3a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-51e091-769f5d944b-dgc2p" Oct 30 09:28:41.560339 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:41.560215 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94ef492d-56fb-4e63-9044-063e1ffe8b3a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-51e091-769f5d944b-dgc2p\" (UID: \"94ef492d-56fb-4e63-9044-063e1ffe8b3a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-51e091-769f5d944b-dgc2p" Oct 30 09:28:41.656790 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:41.656753 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-51e091-54f95fbdcf-lpl7d"] Oct 30 09:28:41.661362 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:41.661343 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-51e091-54f95fbdcf-lpl7d" Oct 30 09:28:41.661685 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:41.661661 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plgvm\" (UniqueName: \"kubernetes.io/projected/94ef492d-56fb-4e63-9044-063e1ffe8b3a-kube-api-access-plgvm\") pod \"payload-processing-e2e-ipp-b-51e091-769f5d944b-dgc2p\" (UID: \"94ef492d-56fb-4e63-9044-063e1ffe8b3a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-51e091-769f5d944b-dgc2p" Oct 30 09:28:41.661848 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:41.661826 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94ef492d-56fb-4e63-9044-063e1ffe8b3a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-51e091-769f5d944b-dgc2p\" (UID: \"94ef492d-56fb-4e63-9044-063e1ffe8b3a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-51e091-769f5d944b-dgc2p" Oct 30 09:28:41.662566 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:41.662545 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94ef492d-56fb-4e63-9044-063e1ffe8b3a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-51e091-769f5d944b-dgc2p\" (UID: \"94ef492d-56fb-4e63-9044-063e1ffe8b3a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-51e091-769f5d944b-dgc2p" Oct 30 09:28:41.702859 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:41.702823 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-51e091-54f95fbdcf-lpl7d"] Oct 30 09:28:41.709432 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:41.705121 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plgvm\" (UniqueName: \"kubernetes.io/projected/94ef492d-56fb-4e63-9044-063e1ffe8b3a-kube-api-access-plgvm\") pod \"payload-processing-e2e-ipp-b-51e091-769f5d944b-dgc2p\" (UID: \"94ef492d-56fb-4e63-9044-063e1ffe8b3a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-51e091-769f5d944b-dgc2p" Oct 30 09:28:41.763756 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:41.763659 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e873180-8cd3-4f72-ad69-2290f73570c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-51e091-54f95fbdcf-lpl7d\" (UID: \"3e873180-8cd3-4f72-ad69-2290f73570c6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-51e091-54f95fbdcf-lpl7d" Oct 30 09:28:41.764026 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:41.764005 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9ddd\" (UniqueName: \"kubernetes.io/projected/3e873180-8cd3-4f72-ad69-2290f73570c6-kube-api-access-h9ddd\") pod \"payload-pre-processing-e2e-ipp-b-51e091-54f95fbdcf-lpl7d\" (UID: \"3e873180-8cd3-4f72-ad69-2290f73570c6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-51e091-54f95fbdcf-lpl7d" Oct 30 09:28:41.840841 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:41.840811 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-51e091-769f5d944b-dgc2p" Oct 30 09:28:41.865079 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:41.865055 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9ddd\" (UniqueName: \"kubernetes.io/projected/3e873180-8cd3-4f72-ad69-2290f73570c6-kube-api-access-h9ddd\") pod \"payload-pre-processing-e2e-ipp-b-51e091-54f95fbdcf-lpl7d\" (UID: \"3e873180-8cd3-4f72-ad69-2290f73570c6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-51e091-54f95fbdcf-lpl7d" Oct 30 09:28:41.865194 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:41.865149 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e873180-8cd3-4f72-ad69-2290f73570c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-51e091-54f95fbdcf-lpl7d\" (UID: \"3e873180-8cd3-4f72-ad69-2290f73570c6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-51e091-54f95fbdcf-lpl7d" Oct 30 09:28:41.865849 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:41.865827 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e873180-8cd3-4f72-ad69-2290f73570c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-51e091-54f95fbdcf-lpl7d\" (UID: \"3e873180-8cd3-4f72-ad69-2290f73570c6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-51e091-54f95fbdcf-lpl7d" Oct 30 09:28:41.886497 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:41.886470 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9ddd\" (UniqueName: \"kubernetes.io/projected/3e873180-8cd3-4f72-ad69-2290f73570c6-kube-api-access-h9ddd\") pod \"payload-pre-processing-e2e-ipp-b-51e091-54f95fbdcf-lpl7d\" (UID: \"3e873180-8cd3-4f72-ad69-2290f73570c6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-51e091-54f95fbdcf-lpl7d" Oct 30 09:28:42.003696 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:42.003658 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-51e091-54f95fbdcf-lpl7d" Oct 30 09:28:42.145208 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:42.145108 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-51e091-769f5d944b-dgc2p"] Oct 30 09:28:42.145352 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:28:42.145243 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94ef492d_56fb_4e63_9044_063e1ffe8b3a.slice/crio-8f37892e1c11edeba0bfa9a50e6f6b62799c0b94241ae0a7df17bbf7d3eac751 WatchSource:0}: Error finding container 8f37892e1c11edeba0bfa9a50e6f6b62799c0b94241ae0a7df17bbf7d3eac751: Status 404 returned error can't find the container with id 8f37892e1c11edeba0bfa9a50e6f6b62799c0b94241ae0a7df17bbf7d3eac751 Oct 30 09:28:42.328433 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:42.324245 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-51e091-54f95fbdcf-lpl7d"] Oct 30 09:28:42.353045 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:28:42.353008 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e873180_8cd3_4f72_ad69_2290f73570c6.slice/crio-8eca257af17627590f758453758fd84848f63319cfe90be4fa9f8c1e9a25270c WatchSource:0}: Error finding container 8eca257af17627590f758453758fd84848f63319cfe90be4fa9f8c1e9a25270c: Status 404 returned error can't find the container with id 8eca257af17627590f758453758fd84848f63319cfe90be4fa9f8c1e9a25270c Oct 30 09:28:43.045443 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:43.045403 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-51e091-54f95fbdcf-lpl7d" event={"ID":"3e873180-8cd3-4f72-ad69-2290f73570c6","Type":"ContainerStarted","Data":"a55b8a37ea13ca0a361a5b9b2bf3e1f2844fd53e0aeb058c3f0a1bf5868218b1"} Oct 30 09:28:43.045876 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:43.045477 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-51e091-54f95fbdcf-lpl7d" Oct 30 09:28:43.045876 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:43.045501 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-51e091-54f95fbdcf-lpl7d" event={"ID":"3e873180-8cd3-4f72-ad69-2290f73570c6","Type":"ContainerStarted","Data":"8eca257af17627590f758453758fd84848f63319cfe90be4fa9f8c1e9a25270c"} Oct 30 09:28:43.047337 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:43.047310 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-51e091-769f5d944b-dgc2p" event={"ID":"94ef492d-56fb-4e63-9044-063e1ffe8b3a","Type":"ContainerStarted","Data":"089f06d3527363b1194ddecfba25ca5d7e05451c89891aba781e367f57faac8e"} Oct 30 09:28:43.047518 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:43.047358 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-51e091-769f5d944b-dgc2p" event={"ID":"94ef492d-56fb-4e63-9044-063e1ffe8b3a","Type":"ContainerStarted","Data":"8f37892e1c11edeba0bfa9a50e6f6b62799c0b94241ae0a7df17bbf7d3eac751"} Oct 30 09:28:43.047518 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:43.047415 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-51e091-769f5d944b-dgc2p" Oct 30 09:28:43.086845 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:43.086792 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-51e091-54f95fbdcf-lpl7d" podStartSLOduration=2.08676905 podStartE2EDuration="2.08676905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:28:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:28:43.085641813 +0000 UTC m=+2083.347202802" watchObservedRunningTime="2026-10-30 09:28:43.08676905 +0000 UTC m=+2083.348330037" Oct 30 09:28:43.128757 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:43.128704 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-51e091-769f5d944b-dgc2p" podStartSLOduration=2.12869149 podStartE2EDuration="2.12869149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:28:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:28:43.125265565 +0000 UTC m=+2083.386826555" watchObservedRunningTime="2026-10-30 09:28:43.12869149 +0000 UTC m=+2083.390252509" Oct 30 09:28:51.728161 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:51.728126 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-d3de9d-798f5684bd-t5k56"] Oct 30 09:28:51.752808 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:51.752775 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-d3de9d-6c6b4db5d7-jt9hq"] Oct 30 09:28:53.177746 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:53.177688 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d87c85-75b5984778-bpxt6"] Oct 30 09:28:53.192698 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:53.192665 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d87c85-5d8c96c5-fm4dg"] Oct 30 09:28:54.130478 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:54.130446 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-51e091-54f95fbdcf-lpl7d" Oct 30 09:28:54.131102 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:54.131065 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-51e091-769f5d944b-dgc2p" Oct 30 09:28:56.728671 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:56.728630 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-d3de9d-798f5684bd-t5k56" podUID="0d356da3-0606-4dd7-940c-f1ef62ba69f0" containerName="payload-processing" containerID="cri-o://73e35a1e40521bf62ed68e565d4d1ee124b206cb7feaf2ef36dfa59a0b273a8e" gracePeriod=25 Oct 30 09:28:56.753910 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:56.753878 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d3de9d-6c6b4db5d7-jt9hq" podUID="c80a8dfe-e8a6-4adc-bdc6-2019c66bce9a" containerName="payload-pre-processing" containerID="cri-o://2bafadeb7f5b8b0c8b2b1bd6dbf2552430a207385e80c64d03a0956159b9eefa" gracePeriod=25 Oct 30 09:28:57.055079 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:57.055052 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-d3de9d-798f5684bd-t5k56" Oct 30 09:28:57.087859 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:57.087834 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d356da3-0606-4dd7-940c-f1ef62ba69f0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d356da3-0606-4dd7-940c-f1ef62ba69f0-plugins-config-volume\") pod \"0d356da3-0606-4dd7-940c-f1ef62ba69f0\" (UID: \"0d356da3-0606-4dd7-940c-f1ef62ba69f0\") " Oct 30 09:28:57.088018 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:57.087991 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d356da3-0606-4dd7-940c-f1ef62ba69f0-kube-api-access-h2z8r\" (UniqueName: \"kubernetes.io/projected/0d356da3-0606-4dd7-940c-f1ef62ba69f0-kube-api-access-h2z8r\") pod \"0d356da3-0606-4dd7-940c-f1ef62ba69f0\" (UID: \"0d356da3-0606-4dd7-940c-f1ef62ba69f0\") " Oct 30 09:28:57.088177 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:57.088155 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d3de9d-6c6b4db5d7-jt9hq" Oct 30 09:28:57.088243 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:57.088207 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d356da3-0606-4dd7-940c-f1ef62ba69f0-plugins-config-volume" pod "0d356da3-0606-4dd7-940c-f1ef62ba69f0" (UID: "0d356da3-0606-4dd7-940c-f1ef62ba69f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:28:57.088584 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:57.088564 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d356da3-0606-4dd7-940c-f1ef62ba69f0-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:28:57.090047 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:57.090025 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d356da3-0606-4dd7-940c-f1ef62ba69f0-kube-api-access-h2z8r" pod "0d356da3-0606-4dd7-940c-f1ef62ba69f0" (UID: "0d356da3-0606-4dd7-940c-f1ef62ba69f0"). InnerVolumeSpecName "kube-api-access-h2z8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:28:57.190045 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:57.190013 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c80a8dfe-e8a6-4adc-bdc6-2019c66bce9a-kube-api-access-xk5cg\" (UniqueName: \"kubernetes.io/projected/c80a8dfe-e8a6-4adc-bdc6-2019c66bce9a-kube-api-access-xk5cg\") pod \"c80a8dfe-e8a6-4adc-bdc6-2019c66bce9a\" (UID: \"c80a8dfe-e8a6-4adc-bdc6-2019c66bce9a\") " Oct 30 09:28:57.190254 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:57.190083 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c80a8dfe-e8a6-4adc-bdc6-2019c66bce9a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c80a8dfe-e8a6-4adc-bdc6-2019c66bce9a-plugins-config-volume\") pod \"c80a8dfe-e8a6-4adc-bdc6-2019c66bce9a\" (UID: \"c80a8dfe-e8a6-4adc-bdc6-2019c66bce9a\") " Oct 30 09:28:57.190586 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:57.190555 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c80a8dfe-e8a6-4adc-bdc6-2019c66bce9a-plugins-config-volume" pod "c80a8dfe-e8a6-4adc-bdc6-2019c66bce9a" (UID: "c80a8dfe-e8a6-4adc-bdc6-2019c66bce9a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:28:57.190809 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:57.190785 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c80a8dfe-e8a6-4adc-bdc6-2019c66bce9a-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:28:57.190883 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:57.190816 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2z8r\" (UniqueName: \"kubernetes.io/projected/0d356da3-0606-4dd7-940c-f1ef62ba69f0-kube-api-access-h2z8r\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:28:57.192368 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:57.192342 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c80a8dfe-e8a6-4adc-bdc6-2019c66bce9a-kube-api-access-xk5cg" pod "c80a8dfe-e8a6-4adc-bdc6-2019c66bce9a" (UID: "c80a8dfe-e8a6-4adc-bdc6-2019c66bce9a"). InnerVolumeSpecName "kube-api-access-xk5cg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:28:57.244130 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:57.244091 4562 generic.go:356] "Generic (PLEG): container finished" podID="0d356da3-0606-4dd7-940c-f1ef62ba69f0" containerID="73e35a1e40521bf62ed68e565d4d1ee124b206cb7feaf2ef36dfa59a0b273a8e" exitCode=0 Oct 30 09:28:57.244291 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:57.244140 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-d3de9d-798f5684bd-t5k56" Oct 30 09:28:57.244291 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:57.244164 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-d3de9d-798f5684bd-t5k56" event={"ID":"0d356da3-0606-4dd7-940c-f1ef62ba69f0","Type":"ContainerDied","Data":"73e35a1e40521bf62ed68e565d4d1ee124b206cb7feaf2ef36dfa59a0b273a8e"} Oct 30 09:28:57.244291 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:57.244219 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-d3de9d-798f5684bd-t5k56" event={"ID":"0d356da3-0606-4dd7-940c-f1ef62ba69f0","Type":"ContainerDied","Data":"7b980fd44c6b64ced85920ec5e04ecf9a1eb2f191eb7c494010865f78aa8ec49"} Oct 30 09:28:57.244291 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:57.244241 4562 scope.go:122] "RemoveContainer" containerID="73e35a1e40521bf62ed68e565d4d1ee124b206cb7feaf2ef36dfa59a0b273a8e" Oct 30 09:28:57.245890 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:57.245865 4562 generic.go:356] "Generic (PLEG): container finished" podID="c80a8dfe-e8a6-4adc-bdc6-2019c66bce9a" containerID="2bafadeb7f5b8b0c8b2b1bd6dbf2552430a207385e80c64d03a0956159b9eefa" exitCode=0 Oct 30 09:28:57.246002 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:57.245913 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d3de9d-6c6b4db5d7-jt9hq" Oct 30 09:28:57.246002 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:57.245948 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d3de9d-6c6b4db5d7-jt9hq" event={"ID":"c80a8dfe-e8a6-4adc-bdc6-2019c66bce9a","Type":"ContainerDied","Data":"2bafadeb7f5b8b0c8b2b1bd6dbf2552430a207385e80c64d03a0956159b9eefa"} Oct 30 09:28:57.246120 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:57.245999 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d3de9d-6c6b4db5d7-jt9hq" event={"ID":"c80a8dfe-e8a6-4adc-bdc6-2019c66bce9a","Type":"ContainerDied","Data":"544536cf7bbfc4a1df5c076c7fe6d3331944805155a02b4e0e003e8ee04d4e8b"} Oct 30 09:28:57.254561 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:57.254416 4562 scope.go:122] "RemoveContainer" containerID="73e35a1e40521bf62ed68e565d4d1ee124b206cb7feaf2ef36dfa59a0b273a8e" Oct 30 09:28:57.254730 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:28:57.254710 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73e35a1e40521bf62ed68e565d4d1ee124b206cb7feaf2ef36dfa59a0b273a8e\": container with ID starting with 73e35a1e40521bf62ed68e565d4d1ee124b206cb7feaf2ef36dfa59a0b273a8e not found: ID does not exist" containerID="73e35a1e40521bf62ed68e565d4d1ee124b206cb7feaf2ef36dfa59a0b273a8e" Oct 30 09:28:57.254795 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:57.254738 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73e35a1e40521bf62ed68e565d4d1ee124b206cb7feaf2ef36dfa59a0b273a8e"} err="failed to get container status \"73e35a1e40521bf62ed68e565d4d1ee124b206cb7feaf2ef36dfa59a0b273a8e\": rpc error: code = NotFound desc = could not find container \"73e35a1e40521bf62ed68e565d4d1ee124b206cb7feaf2ef36dfa59a0b273a8e\": container with ID starting with 73e35a1e40521bf62ed68e565d4d1ee124b206cb7feaf2ef36dfa59a0b273a8e not found: ID does not exist" Oct 30 09:28:57.254795 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:57.254755 4562 scope.go:122] "RemoveContainer" containerID="2bafadeb7f5b8b0c8b2b1bd6dbf2552430a207385e80c64d03a0956159b9eefa" Oct 30 09:28:57.287927 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:57.287876 4562 scope.go:122] "RemoveContainer" containerID="2bafadeb7f5b8b0c8b2b1bd6dbf2552430a207385e80c64d03a0956159b9eefa" Oct 30 09:28:57.288726 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:28:57.288685 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bafadeb7f5b8b0c8b2b1bd6dbf2552430a207385e80c64d03a0956159b9eefa\": container with ID starting with 2bafadeb7f5b8b0c8b2b1bd6dbf2552430a207385e80c64d03a0956159b9eefa not found: ID does not exist" containerID="2bafadeb7f5b8b0c8b2b1bd6dbf2552430a207385e80c64d03a0956159b9eefa" Oct 30 09:28:57.288890 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:57.288734 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bafadeb7f5b8b0c8b2b1bd6dbf2552430a207385e80c64d03a0956159b9eefa"} err="failed to get container status \"2bafadeb7f5b8b0c8b2b1bd6dbf2552430a207385e80c64d03a0956159b9eefa\": rpc error: code = NotFound desc = could not find container \"2bafadeb7f5b8b0c8b2b1bd6dbf2552430a207385e80c64d03a0956159b9eefa\": container with ID starting with 2bafadeb7f5b8b0c8b2b1bd6dbf2552430a207385e80c64d03a0956159b9eefa not found: ID does not exist" Oct 30 09:28:57.300779 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:57.295642 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk5cg\" (UniqueName: \"kubernetes.io/projected/c80a8dfe-e8a6-4adc-bdc6-2019c66bce9a-kube-api-access-xk5cg\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:28:57.334858 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:57.334818 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-d3de9d-798f5684bd-t5k56"] Oct 30 09:28:57.338424 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:57.338396 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-d3de9d-798f5684bd-t5k56"] Oct 30 09:28:57.360407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:57.360352 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-d3de9d-6c6b4db5d7-jt9hq"] Oct 30 09:28:57.364197 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:57.364173 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-d3de9d-6c6b4db5d7-jt9hq"] Oct 30 09:28:57.875485 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:57.875451 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d356da3-0606-4dd7-940c-f1ef62ba69f0" path="/var/lib/kubelet/pods/0d356da3-0606-4dd7-940c-f1ef62ba69f0/volumes" Oct 30 09:28:57.875871 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:57.875857 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c80a8dfe-e8a6-4adc-bdc6-2019c66bce9a" path="/var/lib/kubelet/pods/c80a8dfe-e8a6-4adc-bdc6-2019c66bce9a/volumes" Oct 30 09:28:57.939299 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:57.939262 4562 pod_container_manager_linux.go:218] "Failed to delete cgroup paths" cgroupName=["kubepods","burstable","podc1adfa80-f01a-40cb-bd1c-d1e98873d077"] err="unable to destroy cgroup paths for cgroup [kubepods burstable podc1adfa80-f01a-40cb-bd1c-d1e98873d077] : Timed out while waiting for systemd to remove kubepods-burstable-podc1adfa80_f01a_40cb_bd1c_d1e98873d077.slice" Oct 30 09:28:57.939299 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:57.939299 4562 kubelet_pods.go:2797] "Failed to destroy orphaned pod cgroup" podUID="c1adfa80-f01a-40cb-bd1c-d1e98873d077" err="failed to delete cgroup paths for [kubepods burstable podc1adfa80-f01a-40cb-bd1c-d1e98873d077] : unable to destroy cgroup paths for cgroup [kubepods burstable podc1adfa80-f01a-40cb-bd1c-d1e98873d077] : Timed out while waiting for systemd to remove kubepods-burstable-podc1adfa80_f01a_40cb_bd1c_d1e98873d077.slice" Oct 30 09:28:58.179690 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:58.179575 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-d87c85-75b5984778-bpxt6" podUID="c4fea576-dd2f-4ab6-bcaa-ed3169d2f4dc" containerName="payload-processing" containerID="cri-o://97cd135e19a53bd5a3683f9cd092e1cf72cb567d92e46b002de3bac62e2399e7" gracePeriod=25 Oct 30 09:28:58.194922 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:58.194886 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d87c85-5d8c96c5-fm4dg" podUID="7110103e-fba8-4baf-a62e-9942f43f0f71" containerName="payload-pre-processing" containerID="cri-o://924b566505798ec5081d8638c3529ade417c2d4e2d126a710993732e23a32796" gracePeriod=25 Oct 30 09:28:58.487505 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:58.487477 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d87c85-75b5984778-bpxt6" Oct 30 09:28:58.503633 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:58.503606 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d87c85-5d8c96c5-fm4dg" Oct 30 09:28:58.511056 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:58.511031 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4fea576-dd2f-4ab6-bcaa-ed3169d2f4dc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4fea576-dd2f-4ab6-bcaa-ed3169d2f4dc-plugins-config-volume\") pod \"c4fea576-dd2f-4ab6-bcaa-ed3169d2f4dc\" (UID: \"c4fea576-dd2f-4ab6-bcaa-ed3169d2f4dc\") " Oct 30 09:28:58.511171 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:58.511092 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4fea576-dd2f-4ab6-bcaa-ed3169d2f4dc-kube-api-access-v9wgm\" (UniqueName: \"kubernetes.io/projected/c4fea576-dd2f-4ab6-bcaa-ed3169d2f4dc-kube-api-access-v9wgm\") pod \"c4fea576-dd2f-4ab6-bcaa-ed3169d2f4dc\" (UID: \"c4fea576-dd2f-4ab6-bcaa-ed3169d2f4dc\") " Oct 30 09:28:58.511548 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:58.511515 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4fea576-dd2f-4ab6-bcaa-ed3169d2f4dc-plugins-config-volume" pod "c4fea576-dd2f-4ab6-bcaa-ed3169d2f4dc" (UID: "c4fea576-dd2f-4ab6-bcaa-ed3169d2f4dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:28:58.513125 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:58.513104 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4fea576-dd2f-4ab6-bcaa-ed3169d2f4dc-kube-api-access-v9wgm" pod "c4fea576-dd2f-4ab6-bcaa-ed3169d2f4dc" (UID: "c4fea576-dd2f-4ab6-bcaa-ed3169d2f4dc"). InnerVolumeSpecName "kube-api-access-v9wgm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:28:58.602613 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:28:58.602559 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:28:58.612602 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:58.612576 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7110103e-fba8-4baf-a62e-9942f43f0f71-kube-api-access-vss87\" (UniqueName: \"kubernetes.io/projected/7110103e-fba8-4baf-a62e-9942f43f0f71-kube-api-access-vss87\") pod \"7110103e-fba8-4baf-a62e-9942f43f0f71\" (UID: \"7110103e-fba8-4baf-a62e-9942f43f0f71\") " Oct 30 09:28:58.612724 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:58.612672 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7110103e-fba8-4baf-a62e-9942f43f0f71-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7110103e-fba8-4baf-a62e-9942f43f0f71-plugins-config-volume\") pod \"7110103e-fba8-4baf-a62e-9942f43f0f71\" (UID: \"7110103e-fba8-4baf-a62e-9942f43f0f71\") " Oct 30 09:28:58.613090 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:58.612983 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7110103e-fba8-4baf-a62e-9942f43f0f71-plugins-config-volume" pod "7110103e-fba8-4baf-a62e-9942f43f0f71" (UID: "7110103e-fba8-4baf-a62e-9942f43f0f71"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:28:58.613179 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:58.613097 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9wgm\" (UniqueName: \"kubernetes.io/projected/c4fea576-dd2f-4ab6-bcaa-ed3169d2f4dc-kube-api-access-v9wgm\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:28:58.613179 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:58.613111 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7110103e-fba8-4baf-a62e-9942f43f0f71-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:28:58.613179 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:58.613121 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4fea576-dd2f-4ab6-bcaa-ed3169d2f4dc-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:28:58.614765 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:58.614740 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7110103e-fba8-4baf-a62e-9942f43f0f71-kube-api-access-vss87" pod "7110103e-fba8-4baf-a62e-9942f43f0f71" (UID: "7110103e-fba8-4baf-a62e-9942f43f0f71"). InnerVolumeSpecName "kube-api-access-vss87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:28:58.626800 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:58.626771 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-15e1cc-674c7798f9-9vpwd"] Oct 30 09:28:58.627426 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:58.627412 4562 cpu_manager.go:414] "containerMap: removing container" podUID="c80a8dfe-e8a6-4adc-bdc6-2019c66bce9a" containerName="payload-pre-processing" Oct 30 09:28:58.627479 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:58.627426 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c80a8dfe-e8a6-4adc-bdc6-2019c66bce9a" containerName="payload-pre-processing" Oct 30 09:28:58.627479 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:58.627446 4562 cpu_manager.go:414] "containerMap: removing container" podUID="0d356da3-0606-4dd7-940c-f1ef62ba69f0" containerName="payload-processing" Oct 30 09:28:58.627479 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:58.627451 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d356da3-0606-4dd7-940c-f1ef62ba69f0" containerName="payload-processing" Oct 30 09:28:58.627479 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:58.627472 4562 cpu_manager.go:414] "containerMap: removing container" podUID="7110103e-fba8-4baf-a62e-9942f43f0f71" containerName="payload-pre-processing" Oct 30 09:28:58.627479 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:58.627477 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7110103e-fba8-4baf-a62e-9942f43f0f71" containerName="payload-pre-processing" Oct 30 09:28:58.627633 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:58.627494 4562 cpu_manager.go:414] "containerMap: removing container" podUID="c4fea576-dd2f-4ab6-bcaa-ed3169d2f4dc" containerName="payload-processing" Oct 30 09:28:58.627633 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:58.627499 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4fea576-dd2f-4ab6-bcaa-ed3169d2f4dc" containerName="payload-processing" Oct 30 09:28:58.627633 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:58.627583 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4fea576-dd2f-4ab6-bcaa-ed3169d2f4dc" containerName="payload-processing" Oct 30 09:28:58.627633 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:58.627594 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="c80a8dfe-e8a6-4adc-bdc6-2019c66bce9a" containerName="payload-pre-processing" Oct 30 09:28:58.627633 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:58.627603 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d356da3-0606-4dd7-940c-f1ef62ba69f0" containerName="payload-processing" Oct 30 09:28:58.627633 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:58.627612 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="7110103e-fba8-4baf-a62e-9942f43f0f71" containerName="payload-pre-processing" Oct 30 09:28:58.631316 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:58.631299 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-15e1cc-674c7798f9-9vpwd" Oct 30 09:28:58.648581 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:58.648560 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-15e1cc\"" Oct 30 09:28:58.649618 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:58.649595 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-15e1cc-674c7798f9-9vpwd"] Oct 30 09:28:58.696459 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:58.673532 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-15e1cc-dockercfg-qn7s8\"" Oct 30 09:28:58.721451 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:58.721411 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9597695b-a089-4c8f-8a65-2ec83df78302-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-15e1cc-674c7798f9-9vpwd\" (UID: \"9597695b-a089-4c8f-8a65-2ec83df78302\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-15e1cc-674c7798f9-9vpwd" Oct 30 09:28:58.766714 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:58.755787 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbjrg\" (UniqueName: \"kubernetes.io/projected/9597695b-a089-4c8f-8a65-2ec83df78302-kube-api-access-tbjrg\") pod \"payload-pre-processing-e2e-ipp-b-15e1cc-674c7798f9-9vpwd\" (UID: \"9597695b-a089-4c8f-8a65-2ec83df78302\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-15e1cc-674c7798f9-9vpwd" Oct 30 09:28:58.766714 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:58.756259 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vss87\" (UniqueName: \"kubernetes.io/projected/7110103e-fba8-4baf-a62e-9942f43f0f71-kube-api-access-vss87\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:28:58.840846 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:58.840806 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-15e1cc-645f5cbbc4-j4j7f"] Oct 30 09:28:58.861156 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:58.860909 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9597695b-a089-4c8f-8a65-2ec83df78302-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-15e1cc-674c7798f9-9vpwd\" (UID: \"9597695b-a089-4c8f-8a65-2ec83df78302\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-15e1cc-674c7798f9-9vpwd" Oct 30 09:28:58.861156 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:58.860977 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbjrg\" (UniqueName: \"kubernetes.io/projected/9597695b-a089-4c8f-8a65-2ec83df78302-kube-api-access-tbjrg\") pod \"payload-pre-processing-e2e-ipp-b-15e1cc-674c7798f9-9vpwd\" (UID: \"9597695b-a089-4c8f-8a65-2ec83df78302\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-15e1cc-674c7798f9-9vpwd" Oct 30 09:28:58.891405 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:58.877338 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9597695b-a089-4c8f-8a65-2ec83df78302-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-15e1cc-674c7798f9-9vpwd\" (UID: \"9597695b-a089-4c8f-8a65-2ec83df78302\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-15e1cc-674c7798f9-9vpwd" Oct 30 09:28:58.893179 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:58.893149 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-15e1cc-645f5cbbc4-j4j7f" Oct 30 09:28:58.933771 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:58.899501 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbjrg\" (UniqueName: \"kubernetes.io/projected/9597695b-a089-4c8f-8a65-2ec83df78302-kube-api-access-tbjrg\") pod \"payload-pre-processing-e2e-ipp-b-15e1cc-674c7798f9-9vpwd\" (UID: \"9597695b-a089-4c8f-8a65-2ec83df78302\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-15e1cc-674c7798f9-9vpwd" Oct 30 09:28:58.947245 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:58.947179 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-15e1cc-674c7798f9-9vpwd" Oct 30 09:28:58.950763 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:58.950733 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-15e1cc-645f5cbbc4-j4j7f"] Oct 30 09:28:58.962289 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:58.962253 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cab9c95-15d2-48f5-99a1-1d6df3fd2333-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-15e1cc-645f5cbbc4-j4j7f\" (UID: \"8cab9c95-15d2-48f5-99a1-1d6df3fd2333\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-15e1cc-645f5cbbc4-j4j7f" Oct 30 09:28:58.962607 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:58.962581 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7z78\" (UniqueName: \"kubernetes.io/projected/8cab9c95-15d2-48f5-99a1-1d6df3fd2333-kube-api-access-z7z78\") pod \"payload-processing-e2e-ipp-b-15e1cc-645f5cbbc4-j4j7f\" (UID: \"8cab9c95-15d2-48f5-99a1-1d6df3fd2333\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-15e1cc-645f5cbbc4-j4j7f" Oct 30 09:28:59.065411 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:59.064516 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cab9c95-15d2-48f5-99a1-1d6df3fd2333-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-15e1cc-645f5cbbc4-j4j7f\" (UID: \"8cab9c95-15d2-48f5-99a1-1d6df3fd2333\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-15e1cc-645f5cbbc4-j4j7f" Oct 30 09:28:59.065411 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:59.064767 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7z78\" (UniqueName: \"kubernetes.io/projected/8cab9c95-15d2-48f5-99a1-1d6df3fd2333-kube-api-access-z7z78\") pod \"payload-processing-e2e-ipp-b-15e1cc-645f5cbbc4-j4j7f\" (UID: \"8cab9c95-15d2-48f5-99a1-1d6df3fd2333\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-15e1cc-645f5cbbc4-j4j7f" Oct 30 09:28:59.066486 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:59.066348 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cab9c95-15d2-48f5-99a1-1d6df3fd2333-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-15e1cc-645f5cbbc4-j4j7f\" (UID: \"8cab9c95-15d2-48f5-99a1-1d6df3fd2333\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-15e1cc-645f5cbbc4-j4j7f" Oct 30 09:28:59.087062 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:59.087030 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7z78\" (UniqueName: \"kubernetes.io/projected/8cab9c95-15d2-48f5-99a1-1d6df3fd2333-kube-api-access-z7z78\") pod \"payload-processing-e2e-ipp-b-15e1cc-645f5cbbc4-j4j7f\" (UID: \"8cab9c95-15d2-48f5-99a1-1d6df3fd2333\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-15e1cc-645f5cbbc4-j4j7f" Oct 30 09:28:59.273559 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:59.272914 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-15e1cc-645f5cbbc4-j4j7f" Oct 30 09:28:59.275703 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:59.275247 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-15e1cc-674c7798f9-9vpwd"] Oct 30 09:28:59.289812 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:59.289782 4562 generic.go:356] "Generic (PLEG): container finished" podID="7110103e-fba8-4baf-a62e-9942f43f0f71" containerID="924b566505798ec5081d8638c3529ade417c2d4e2d126a710993732e23a32796" exitCode=0 Oct 30 09:28:59.289966 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:59.289830 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d87c85-5d8c96c5-fm4dg" Oct 30 09:28:59.289966 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:59.289864 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d87c85-5d8c96c5-fm4dg" event={"ID":"7110103e-fba8-4baf-a62e-9942f43f0f71","Type":"ContainerDied","Data":"924b566505798ec5081d8638c3529ade417c2d4e2d126a710993732e23a32796"} Oct 30 09:28:59.289966 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:59.289930 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d87c85-5d8c96c5-fm4dg" event={"ID":"7110103e-fba8-4baf-a62e-9942f43f0f71","Type":"ContainerDied","Data":"87862e32c09abce2efa92696f0e97390f144b33150b5845989b6805b5b93af0c"} Oct 30 09:28:59.289966 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:59.289960 4562 scope.go:122] "RemoveContainer" containerID="924b566505798ec5081d8638c3529ade417c2d4e2d126a710993732e23a32796" Oct 30 09:28:59.291639 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:59.291617 4562 generic.go:356] "Generic (PLEG): container finished" podID="c4fea576-dd2f-4ab6-bcaa-ed3169d2f4dc" containerID="97cd135e19a53bd5a3683f9cd092e1cf72cb567d92e46b002de3bac62e2399e7" exitCode=0 Oct 30 09:28:59.291777 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:59.291676 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d87c85-75b5984778-bpxt6" Oct 30 09:28:59.291777 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:59.291718 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d87c85-75b5984778-bpxt6" event={"ID":"c4fea576-dd2f-4ab6-bcaa-ed3169d2f4dc","Type":"ContainerDied","Data":"97cd135e19a53bd5a3683f9cd092e1cf72cb567d92e46b002de3bac62e2399e7"} Oct 30 09:28:59.291777 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:59.291762 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d87c85-75b5984778-bpxt6" event={"ID":"c4fea576-dd2f-4ab6-bcaa-ed3169d2f4dc","Type":"ContainerDied","Data":"cddcbf66f9fa9a0bed792d28a875bd7d297fc87632fe12ae997f3b71ca88d652"} Oct 30 09:28:59.294172 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:59.294137 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-15e1cc-674c7798f9-9vpwd" event={"ID":"9597695b-a089-4c8f-8a65-2ec83df78302","Type":"ContainerStarted","Data":"f26f341488fd76e0d73246ee7069535612a8f84198351a5b8395c7645127a695"} Oct 30 09:28:59.306752 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:59.306730 4562 scope.go:122] "RemoveContainer" containerID="924b566505798ec5081d8638c3529ade417c2d4e2d126a710993732e23a32796" Oct 30 09:28:59.307076 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:28:59.307058 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"924b566505798ec5081d8638c3529ade417c2d4e2d126a710993732e23a32796\": container with ID starting with 924b566505798ec5081d8638c3529ade417c2d4e2d126a710993732e23a32796 not found: ID does not exist" containerID="924b566505798ec5081d8638c3529ade417c2d4e2d126a710993732e23a32796" Oct 30 09:28:59.307139 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:59.307085 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"924b566505798ec5081d8638c3529ade417c2d4e2d126a710993732e23a32796"} err="failed to get container status \"924b566505798ec5081d8638c3529ade417c2d4e2d126a710993732e23a32796\": rpc error: code = NotFound desc = could not find container \"924b566505798ec5081d8638c3529ade417c2d4e2d126a710993732e23a32796\": container with ID starting with 924b566505798ec5081d8638c3529ade417c2d4e2d126a710993732e23a32796 not found: ID does not exist" Oct 30 09:28:59.307139 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:59.307103 4562 scope.go:122] "RemoveContainer" containerID="97cd135e19a53bd5a3683f9cd092e1cf72cb567d92e46b002de3bac62e2399e7" Oct 30 09:28:59.320044 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:59.320010 4562 scope.go:122] "RemoveContainer" containerID="97cd135e19a53bd5a3683f9cd092e1cf72cb567d92e46b002de3bac62e2399e7" Oct 30 09:28:59.320421 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:28:59.320373 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97cd135e19a53bd5a3683f9cd092e1cf72cb567d92e46b002de3bac62e2399e7\": container with ID starting with 97cd135e19a53bd5a3683f9cd092e1cf72cb567d92e46b002de3bac62e2399e7 not found: ID does not exist" containerID="97cd135e19a53bd5a3683f9cd092e1cf72cb567d92e46b002de3bac62e2399e7" Oct 30 09:28:59.320506 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:59.320428 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97cd135e19a53bd5a3683f9cd092e1cf72cb567d92e46b002de3bac62e2399e7"} err="failed to get container status \"97cd135e19a53bd5a3683f9cd092e1cf72cb567d92e46b002de3bac62e2399e7\": rpc error: code = NotFound desc = could not find container \"97cd135e19a53bd5a3683f9cd092e1cf72cb567d92e46b002de3bac62e2399e7\": container with ID starting with 97cd135e19a53bd5a3683f9cd092e1cf72cb567d92e46b002de3bac62e2399e7 not found: ID does not exist" Oct 30 09:28:59.347276 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:59.347249 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d87c85-5d8c96c5-fm4dg"] Oct 30 09:28:59.366591 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:59.366555 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d87c85-5d8c96c5-fm4dg"] Oct 30 09:28:59.397195 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:59.396757 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d87c85-75b5984778-bpxt6"] Oct 30 09:28:59.401982 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:59.401957 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d87c85-75b5984778-bpxt6"] Oct 30 09:28:59.583512 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:28:59.583470 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cab9c95_15d2_48f5_99a1_1d6df3fd2333.slice/crio-5903d12aa2e2cb4cbc1f4f39455a93c4cee577f4dafc546d4739f8e073b4424a WatchSource:0}: Error finding container 5903d12aa2e2cb4cbc1f4f39455a93c4cee577f4dafc546d4739f8e073b4424a: Status 404 returned error can't find the container with id 5903d12aa2e2cb4cbc1f4f39455a93c4cee577f4dafc546d4739f8e073b4424a Oct 30 09:28:59.584810 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:59.584778 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-15e1cc-645f5cbbc4-j4j7f"] Oct 30 09:28:59.890633 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:59.890563 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7110103e-fba8-4baf-a62e-9942f43f0f71" path="/var/lib/kubelet/pods/7110103e-fba8-4baf-a62e-9942f43f0f71/volumes" Oct 30 09:28:59.890966 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:28:59.890951 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4fea576-dd2f-4ab6-bcaa-ed3169d2f4dc" path="/var/lib/kubelet/pods/c4fea576-dd2f-4ab6-bcaa-ed3169d2f4dc/volumes" Oct 30 09:29:00.143178 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:00.143107 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6q7n_1f51db5a-5720-4dea-ad02-f336b35d43b1/ovn-acl-logging/0.log" Oct 30 09:29:00.156619 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:00.156598 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6q7n_1f51db5a-5720-4dea-ad02-f336b35d43b1/ovn-acl-logging/0.log" Oct 30 09:29:00.303124 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:00.303081 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-15e1cc-674c7798f9-9vpwd" event={"ID":"9597695b-a089-4c8f-8a65-2ec83df78302","Type":"ContainerStarted","Data":"2f5b1d301ee52170d64f75523e1a14dc634edbbe7fe57cebc51ab5f26b437b1d"} Oct 30 09:29:00.303316 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:00.303246 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-15e1cc-674c7798f9-9vpwd" Oct 30 09:29:00.305740 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:00.305717 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-15e1cc-645f5cbbc4-j4j7f" event={"ID":"8cab9c95-15d2-48f5-99a1-1d6df3fd2333","Type":"ContainerStarted","Data":"682090c047dddac9e841455437acca838bc793f79bfbff46dc30913d27d9c91f"} Oct 30 09:29:00.305854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:00.305759 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-15e1cc-645f5cbbc4-j4j7f" event={"ID":"8cab9c95-15d2-48f5-99a1-1d6df3fd2333","Type":"ContainerStarted","Data":"5903d12aa2e2cb4cbc1f4f39455a93c4cee577f4dafc546d4739f8e073b4424a"} Oct 30 09:29:00.305854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:00.305787 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-15e1cc-645f5cbbc4-j4j7f" Oct 30 09:29:00.344821 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:00.344777 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-15e1cc-674c7798f9-9vpwd" podStartSLOduration=2.3447646 podStartE2EDuration="2.3447646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:28:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:29:00.342794424 +0000 UTC m=+2100.604355412" watchObservedRunningTime="2026-10-30 09:29:00.3447646 +0000 UTC m=+2100.606325589" Oct 30 09:29:00.383617 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:00.383562 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-15e1cc-645f5cbbc4-j4j7f" podStartSLOduration=2.383547947 podStartE2EDuration="2.383547947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:28:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:29:00.382087193 +0000 UTC m=+2100.643648182" watchObservedRunningTime="2026-10-30 09:29:00.383547947 +0000 UTC m=+2100.645108936" Oct 30 09:29:01.906546 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:01.906516 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-85rhg_2f5e6dbb-9b81-4734-ba2a-5c6a677e49e8/payload-processing/0.log" Oct 30 09:29:02.133794 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:02.133768 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-85rhg_2f5e6dbb-9b81-4734-ba2a-5c6a677e49e8/payload-processing/0.log" Oct 30 09:29:02.320444 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:02.320418 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-51e091-769f5d944b-dgc2p_94ef492d-56fb-4e63-9044-063e1ffe8b3a/payload-processing/0.log" Oct 30 09:29:11.315590 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:11.315556 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-15e1cc-645f5cbbc4-j4j7f" Oct 30 09:29:11.315983 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:11.315627 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-15e1cc-674c7798f9-9vpwd" Oct 30 09:29:14.115965 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:14.115885 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp"] Oct 30 09:29:14.116444 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:14.116078 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp" podUID="c482f1ed-23a5-4cbb-a077-c907faac860a" containerName="istio-proxy" containerID="cri-o://5d7f89f3f4d2edba5bee12b91918478c155b8634625967fbc57b3372b1de93d8" gracePeriod=30 Oct 30 09:29:15.452202 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:15.452164 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cbd96e-58fb495dc7-m7gn6"] Oct 30 09:29:15.453293 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:15.453225 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cbd96e-58fb495dc7-m7gn6" podUID="56e6a564-249b-463f-a531-e1177d1c8150" containerName="maas-api" containerID="cri-o://26b08e4b1a3b566e9a56caf92a3768b162939267d15af15d5540b35fce2d1b3b" gracePeriod=30 Oct 30 09:29:15.955431 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:15.945586 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cbd96e-58fb495dc7-m7gn6" Oct 30 09:29:15.984679 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:15.972986 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56e6a564-249b-463f-a531-e1177d1c8150-kube-api-access-nzh2z\" (UniqueName: \"kubernetes.io/projected/56e6a564-249b-463f-a531-e1177d1c8150-kube-api-access-nzh2z\") pod \"56e6a564-249b-463f-a531-e1177d1c8150\" (UID: \"56e6a564-249b-463f-a531-e1177d1c8150\") " Oct 30 09:29:15.985040 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:15.985012 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56e6a564-249b-463f-a531-e1177d1c8150-metrics-certs\" (UniqueName: \"kubernetes.io/secret/56e6a564-249b-463f-a531-e1177d1c8150-metrics-certs\") pod \"56e6a564-249b-463f-a531-e1177d1c8150\" (UID: \"56e6a564-249b-463f-a531-e1177d1c8150\") " Oct 30 09:29:15.997236 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:15.997190 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56e6a564-249b-463f-a531-e1177d1c8150-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56e6a564-249b-463f-a531-e1177d1c8150-maas-api-tls\") pod \"56e6a564-249b-463f-a531-e1177d1c8150\" (UID: \"56e6a564-249b-463f-a531-e1177d1c8150\") " Oct 30 09:29:15.998078 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:15.998053 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56e6a564-249b-463f-a531-e1177d1c8150-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56e6a564-249b-463f-a531-e1177d1c8150-openshift-service-ca\") pod \"56e6a564-249b-463f-a531-e1177d1c8150\" (UID: \"56e6a564-249b-463f-a531-e1177d1c8150\") " Oct 30 09:29:16.000208 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:16.000170 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56e6a564-249b-463f-a531-e1177d1c8150-openshift-service-ca" pod "56e6a564-249b-463f-a531-e1177d1c8150" (UID: "56e6a564-249b-463f-a531-e1177d1c8150"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:29:16.065829 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:16.065745 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56e6a564-249b-463f-a531-e1177d1c8150-metrics-certs" pod "56e6a564-249b-463f-a531-e1177d1c8150" (UID: "56e6a564-249b-463f-a531-e1177d1c8150"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:29:16.065993 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:16.065903 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56e6a564-249b-463f-a531-e1177d1c8150-kube-api-access-nzh2z" pod "56e6a564-249b-463f-a531-e1177d1c8150" (UID: "56e6a564-249b-463f-a531-e1177d1c8150"). InnerVolumeSpecName "kube-api-access-nzh2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:29:16.072435 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:16.068485 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56e6a564-249b-463f-a531-e1177d1c8150-maas-api-tls" pod "56e6a564-249b-463f-a531-e1177d1c8150" (UID: "56e6a564-249b-463f-a531-e1177d1c8150"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:29:16.100489 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:16.100399 4562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56e6a564-249b-463f-a531-e1177d1c8150-metrics-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:29:16.100489 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:16.100434 4562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56e6a564-249b-463f-a531-e1177d1c8150-maas-api-tls\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:29:16.100489 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:16.100448 4562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56e6a564-249b-463f-a531-e1177d1c8150-openshift-service-ca\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:29:16.100489 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:16.100462 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzh2z\" (UniqueName: \"kubernetes.io/projected/56e6a564-249b-463f-a531-e1177d1c8150-kube-api-access-nzh2z\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:29:16.102786 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:16.102753 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9d39c0-648d978b78-446d4"] Oct 30 09:29:16.103004 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:16.102976 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-9d39c0-648d978b78-446d4" podUID="0141a7df-acc1-4bb4-9cd8-c66e44ee4eb4" containerName="payload-processing" containerID="cri-o://2bd5c139ae0e3da61ce049fc14e5f017cd3f6c6a29419358d8f13f173a64e21f" gracePeriod=30 Oct 30 09:29:16.107016 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:16.106990 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9d39c0-7fc7b8887f-rfr9x"] Oct 30 09:29:16.107338 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:16.107312 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9d39c0-7fc7b8887f-rfr9x" podUID="80a75bd3-f6d2-4a1b-ab28-d7111f61f5e0" containerName="payload-pre-processing" containerID="cri-o://7cd7c914d6399085825ad11c40b0c2373320b53de6f3ea1c59a3f63b78ebe37a" gracePeriod=30 Oct 30 09:29:16.581777 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:16.581539 4562 generic.go:356] "Generic (PLEG): container finished" podID="80a75bd3-f6d2-4a1b-ab28-d7111f61f5e0" containerID="7cd7c914d6399085825ad11c40b0c2373320b53de6f3ea1c59a3f63b78ebe37a" exitCode=0 Oct 30 09:29:16.581777 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:16.581644 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9d39c0-7fc7b8887f-rfr9x" event={"ID":"80a75bd3-f6d2-4a1b-ab28-d7111f61f5e0","Type":"ContainerDied","Data":"7cd7c914d6399085825ad11c40b0c2373320b53de6f3ea1c59a3f63b78ebe37a"} Oct 30 09:29:16.585204 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:16.584182 4562 generic.go:356] "Generic (PLEG): container finished" podID="56e6a564-249b-463f-a531-e1177d1c8150" containerID="26b08e4b1a3b566e9a56caf92a3768b162939267d15af15d5540b35fce2d1b3b" exitCode=0 Oct 30 09:29:16.585204 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:16.584498 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cbd96e-58fb495dc7-m7gn6" event={"ID":"56e6a564-249b-463f-a531-e1177d1c8150","Type":"ContainerDied","Data":"26b08e4b1a3b566e9a56caf92a3768b162939267d15af15d5540b35fce2d1b3b"} Oct 30 09:29:16.585204 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:16.584558 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cbd96e-58fb495dc7-m7gn6" event={"ID":"56e6a564-249b-463f-a531-e1177d1c8150","Type":"ContainerDied","Data":"3062bc38ff6a52a0b99a4f3deae683c3f36696d8361be7c23128289dd8e2e730"} Oct 30 09:29:16.585204 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:16.584591 4562 scope.go:122] "RemoveContainer" containerID="26b08e4b1a3b566e9a56caf92a3768b162939267d15af15d5540b35fce2d1b3b" Oct 30 09:29:16.585204 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:16.584747 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cbd96e-58fb495dc7-m7gn6" Oct 30 09:29:16.595705 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:16.595645 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9d39c0-648d978b78-446d4" event={"ID":"0141a7df-acc1-4bb4-9cd8-c66e44ee4eb4","Type":"ContainerDied","Data":"2bd5c139ae0e3da61ce049fc14e5f017cd3f6c6a29419358d8f13f173a64e21f"} Oct 30 09:29:16.595949 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:16.595537 4562 generic.go:356] "Generic (PLEG): container finished" podID="0141a7df-acc1-4bb4-9cd8-c66e44ee4eb4" containerID="2bd5c139ae0e3da61ce049fc14e5f017cd3f6c6a29419358d8f13f173a64e21f" exitCode=0 Oct 30 09:29:16.634958 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:16.634935 4562 scope.go:122] "RemoveContainer" containerID="26b08e4b1a3b566e9a56caf92a3768b162939267d15af15d5540b35fce2d1b3b" Oct 30 09:29:16.635293 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:16.635273 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9d39c0-648d978b78-446d4" Oct 30 09:29:16.635426 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:29:16.635267 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26b08e4b1a3b566e9a56caf92a3768b162939267d15af15d5540b35fce2d1b3b\": container with ID starting with 26b08e4b1a3b566e9a56caf92a3768b162939267d15af15d5540b35fce2d1b3b not found: ID does not exist" containerID="26b08e4b1a3b566e9a56caf92a3768b162939267d15af15d5540b35fce2d1b3b" Oct 30 09:29:16.635426 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:16.635351 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26b08e4b1a3b566e9a56caf92a3768b162939267d15af15d5540b35fce2d1b3b"} err="failed to get container status \"26b08e4b1a3b566e9a56caf92a3768b162939267d15af15d5540b35fce2d1b3b\": rpc error: code = NotFound desc = could not find container \"26b08e4b1a3b566e9a56caf92a3768b162939267d15af15d5540b35fce2d1b3b\": container with ID starting with 26b08e4b1a3b566e9a56caf92a3768b162939267d15af15d5540b35fce2d1b3b not found: ID does not exist" Oct 30 09:29:16.640601 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:16.640575 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9d39c0-7fc7b8887f-rfr9x" Oct 30 09:29:16.676823 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:16.676784 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cbd96e-58fb495dc7-m7gn6"] Oct 30 09:29:16.680545 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:16.680490 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cbd96e-58fb495dc7-m7gn6"] Oct 30 09:29:16.780916 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:16.780881 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80a75bd3-f6d2-4a1b-ab28-d7111f61f5e0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80a75bd3-f6d2-4a1b-ab28-d7111f61f5e0-plugins-config-volume\") pod \"80a75bd3-f6d2-4a1b-ab28-d7111f61f5e0\" (UID: \"80a75bd3-f6d2-4a1b-ab28-d7111f61f5e0\") " Oct 30 09:29:16.781116 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:16.780950 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0141a7df-acc1-4bb4-9cd8-c66e44ee4eb4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0141a7df-acc1-4bb4-9cd8-c66e44ee4eb4-plugins-config-volume\") pod \"0141a7df-acc1-4bb4-9cd8-c66e44ee4eb4\" (UID: \"0141a7df-acc1-4bb4-9cd8-c66e44ee4eb4\") " Oct 30 09:29:16.781189 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:16.781111 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80a75bd3-f6d2-4a1b-ab28-d7111f61f5e0-kube-api-access-bxhj8\" (UniqueName: \"kubernetes.io/projected/80a75bd3-f6d2-4a1b-ab28-d7111f61f5e0-kube-api-access-bxhj8\") pod \"80a75bd3-f6d2-4a1b-ab28-d7111f61f5e0\" (UID: \"80a75bd3-f6d2-4a1b-ab28-d7111f61f5e0\") " Oct 30 09:29:16.781189 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:16.781153 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0141a7df-acc1-4bb4-9cd8-c66e44ee4eb4-kube-api-access-xqjfw\" (UniqueName: \"kubernetes.io/projected/0141a7df-acc1-4bb4-9cd8-c66e44ee4eb4-kube-api-access-xqjfw\") pod \"0141a7df-acc1-4bb4-9cd8-c66e44ee4eb4\" (UID: \"0141a7df-acc1-4bb4-9cd8-c66e44ee4eb4\") " Oct 30 09:29:16.781859 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:16.781826 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0141a7df-acc1-4bb4-9cd8-c66e44ee4eb4-plugins-config-volume" pod "0141a7df-acc1-4bb4-9cd8-c66e44ee4eb4" (UID: "0141a7df-acc1-4bb4-9cd8-c66e44ee4eb4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:29:16.782272 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:16.782246 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80a75bd3-f6d2-4a1b-ab28-d7111f61f5e0-plugins-config-volume" pod "80a75bd3-f6d2-4a1b-ab28-d7111f61f5e0" (UID: "80a75bd3-f6d2-4a1b-ab28-d7111f61f5e0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:29:16.783377 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:16.783350 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80a75bd3-f6d2-4a1b-ab28-d7111f61f5e0-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:29:16.783498 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:16.783403 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0141a7df-acc1-4bb4-9cd8-c66e44ee4eb4-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:29:16.784319 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:16.784297 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80a75bd3-f6d2-4a1b-ab28-d7111f61f5e0-kube-api-access-bxhj8" pod "80a75bd3-f6d2-4a1b-ab28-d7111f61f5e0" (UID: "80a75bd3-f6d2-4a1b-ab28-d7111f61f5e0"). InnerVolumeSpecName "kube-api-access-bxhj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:29:16.785413 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:16.785345 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0141a7df-acc1-4bb4-9cd8-c66e44ee4eb4-kube-api-access-xqjfw" pod "0141a7df-acc1-4bb4-9cd8-c66e44ee4eb4" (UID: "0141a7df-acc1-4bb4-9cd8-c66e44ee4eb4"). InnerVolumeSpecName "kube-api-access-xqjfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:29:16.884864 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:16.884784 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxhj8\" (UniqueName: \"kubernetes.io/projected/80a75bd3-f6d2-4a1b-ab28-d7111f61f5e0-kube-api-access-bxhj8\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:29:16.884864 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:16.884815 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqjfw\" (UniqueName: \"kubernetes.io/projected/0141a7df-acc1-4bb4-9cd8-c66e44ee4eb4-kube-api-access-xqjfw\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:29:17.604590 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:17.604562 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9d39c0-648d978b78-446d4" Oct 30 09:29:17.605190 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:17.604578 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9d39c0-648d978b78-446d4" event={"ID":"0141a7df-acc1-4bb4-9cd8-c66e44ee4eb4","Type":"ContainerDied","Data":"01a970641c4b413e821dffe4c76f8676dc7194a89bd9f732912a976279a0a226"} Oct 30 09:29:17.605320 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:17.605234 4562 scope.go:122] "RemoveContainer" containerID="2bd5c139ae0e3da61ce049fc14e5f017cd3f6c6a29419358d8f13f173a64e21f" Oct 30 09:29:17.606625 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:17.606251 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9d39c0-7fc7b8887f-rfr9x" Oct 30 09:29:17.606625 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:17.606279 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9d39c0-7fc7b8887f-rfr9x" event={"ID":"80a75bd3-f6d2-4a1b-ab28-d7111f61f5e0","Type":"ContainerDied","Data":"828305c94c3d566ae9c4f6b8260667e25b6af88537edf16ecd022933283e7a70"} Oct 30 09:29:17.616970 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:17.616949 4562 scope.go:122] "RemoveContainer" containerID="7cd7c914d6399085825ad11c40b0c2373320b53de6f3ea1c59a3f63b78ebe37a" Oct 30 09:29:17.661473 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:17.661443 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9d39c0-648d978b78-446d4"] Oct 30 09:29:17.674765 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:17.674738 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9d39c0-648d978b78-446d4"] Oct 30 09:29:17.699086 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:17.699054 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9d39c0-7fc7b8887f-rfr9x"] Oct 30 09:29:17.702145 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:17.702121 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9d39c0-7fc7b8887f-rfr9x"] Oct 30 09:29:17.874932 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:17.874849 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0141a7df-acc1-4bb4-9cd8-c66e44ee4eb4" path="/var/lib/kubelet/pods/0141a7df-acc1-4bb4-9cd8-c66e44ee4eb4/volumes" Oct 30 09:29:17.875412 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:17.875371 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56e6a564-249b-463f-a531-e1177d1c8150" path="/var/lib/kubelet/pods/56e6a564-249b-463f-a531-e1177d1c8150/volumes" Oct 30 09:29:17.875839 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:17.875820 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80a75bd3-f6d2-4a1b-ab28-d7111f61f5e0" path="/var/lib/kubelet/pods/80a75bd3-f6d2-4a1b-ab28-d7111f61f5e0/volumes" Oct 30 09:29:19.390128 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:19.390069 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp" Oct 30 09:29:19.459811 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:19.458703 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c482f1ed-23a5-4cbb-a077-c907faac860a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c482f1ed-23a5-4cbb-a077-c907faac860a-workload-certs\") pod \"c482f1ed-23a5-4cbb-a077-c907faac860a\" (UID: \"c482f1ed-23a5-4cbb-a077-c907faac860a\") " Oct 30 09:29:19.459811 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:19.458768 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c482f1ed-23a5-4cbb-a077-c907faac860a-istio-token\" (UniqueName: \"kubernetes.io/projected/c482f1ed-23a5-4cbb-a077-c907faac860a-istio-token\") pod \"c482f1ed-23a5-4cbb-a077-c907faac860a\" (UID: \"c482f1ed-23a5-4cbb-a077-c907faac860a\") " Oct 30 09:29:19.459811 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:19.458808 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c482f1ed-23a5-4cbb-a077-c907faac860a-kube-api-access-6mg7m\" (UniqueName: \"kubernetes.io/projected/c482f1ed-23a5-4cbb-a077-c907faac860a-kube-api-access-6mg7m\") pod \"c482f1ed-23a5-4cbb-a077-c907faac860a\" (UID: \"c482f1ed-23a5-4cbb-a077-c907faac860a\") " Oct 30 09:29:19.459811 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:19.458840 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c482f1ed-23a5-4cbb-a077-c907faac860a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c482f1ed-23a5-4cbb-a077-c907faac860a-istio-data\") pod \"c482f1ed-23a5-4cbb-a077-c907faac860a\" (UID: \"c482f1ed-23a5-4cbb-a077-c907faac860a\") " Oct 30 09:29:19.459811 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:19.458922 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c482f1ed-23a5-4cbb-a077-c907faac860a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c482f1ed-23a5-4cbb-a077-c907faac860a-istio-podinfo\") pod \"c482f1ed-23a5-4cbb-a077-c907faac860a\" (UID: \"c482f1ed-23a5-4cbb-a077-c907faac860a\") " Oct 30 09:29:19.459811 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:19.459053 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c482f1ed-23a5-4cbb-a077-c907faac860a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c482f1ed-23a5-4cbb-a077-c907faac860a-istio-envoy\") pod \"c482f1ed-23a5-4cbb-a077-c907faac860a\" (UID: \"c482f1ed-23a5-4cbb-a077-c907faac860a\") " Oct 30 09:29:19.459811 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:19.459137 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c482f1ed-23a5-4cbb-a077-c907faac860a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c482f1ed-23a5-4cbb-a077-c907faac860a-credential-socket\") pod \"c482f1ed-23a5-4cbb-a077-c907faac860a\" (UID: \"c482f1ed-23a5-4cbb-a077-c907faac860a\") " Oct 30 09:29:19.459811 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:19.459211 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c482f1ed-23a5-4cbb-a077-c907faac860a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c482f1ed-23a5-4cbb-a077-c907faac860a-istiod-ca-cert\") pod \"c482f1ed-23a5-4cbb-a077-c907faac860a\" (UID: \"c482f1ed-23a5-4cbb-a077-c907faac860a\") " Oct 30 09:29:19.459811 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:19.459236 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c482f1ed-23a5-4cbb-a077-c907faac860a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c482f1ed-23a5-4cbb-a077-c907faac860a-workload-socket\") pod \"c482f1ed-23a5-4cbb-a077-c907faac860a\" (UID: \"c482f1ed-23a5-4cbb-a077-c907faac860a\") " Oct 30 09:29:19.461533 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:19.460521 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c482f1ed-23a5-4cbb-a077-c907faac860a-workload-socket" pod "c482f1ed-23a5-4cbb-a077-c907faac860a" (UID: "c482f1ed-23a5-4cbb-a077-c907faac860a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:29:19.461533 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:19.460710 4562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c482f1ed-23a5-4cbb-a077-c907faac860a-workload-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:29:19.461533 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:19.461496 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c482f1ed-23a5-4cbb-a077-c907faac860a-istiod-ca-cert" pod "c482f1ed-23a5-4cbb-a077-c907faac860a" (UID: "c482f1ed-23a5-4cbb-a077-c907faac860a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:29:19.461851 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:19.461820 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c482f1ed-23a5-4cbb-a077-c907faac860a-workload-certs" pod "c482f1ed-23a5-4cbb-a077-c907faac860a" (UID: "c482f1ed-23a5-4cbb-a077-c907faac860a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:29:19.461968 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:19.461889 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c482f1ed-23a5-4cbb-a077-c907faac860a-credential-socket" pod "c482f1ed-23a5-4cbb-a077-c907faac860a" (UID: "c482f1ed-23a5-4cbb-a077-c907faac860a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:29:19.463645 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:19.463582 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c482f1ed-23a5-4cbb-a077-c907faac860a-istio-data" pod "c482f1ed-23a5-4cbb-a077-c907faac860a" (UID: "c482f1ed-23a5-4cbb-a077-c907faac860a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:29:19.468364 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:19.468343 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c482f1ed-23a5-4cbb-a077-c907faac860a-istio-podinfo" pod "c482f1ed-23a5-4cbb-a077-c907faac860a" (UID: "c482f1ed-23a5-4cbb-a077-c907faac860a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:29:19.469830 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:19.469803 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c482f1ed-23a5-4cbb-a077-c907faac860a-kube-api-access-6mg7m" pod "c482f1ed-23a5-4cbb-a077-c907faac860a" (UID: "c482f1ed-23a5-4cbb-a077-c907faac860a"). InnerVolumeSpecName "kube-api-access-6mg7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:29:19.471618 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:19.471584 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c482f1ed-23a5-4cbb-a077-c907faac860a-istio-envoy" pod "c482f1ed-23a5-4cbb-a077-c907faac860a" (UID: "c482f1ed-23a5-4cbb-a077-c907faac860a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:29:19.476830 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:19.476807 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c482f1ed-23a5-4cbb-a077-c907faac860a-istio-token" pod "c482f1ed-23a5-4cbb-a077-c907faac860a" (UID: "c482f1ed-23a5-4cbb-a077-c907faac860a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:29:19.562558 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:19.562523 4562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c482f1ed-23a5-4cbb-a077-c907faac860a-credential-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:29:19.562558 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:19.562555 4562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c482f1ed-23a5-4cbb-a077-c907faac860a-istiod-ca-cert\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:29:19.562558 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:19.562564 4562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c482f1ed-23a5-4cbb-a077-c907faac860a-workload-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:29:19.562818 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:19.562574 4562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c482f1ed-23a5-4cbb-a077-c907faac860a-istio-token\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:29:19.562818 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:19.562582 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mg7m\" (UniqueName: \"kubernetes.io/projected/c482f1ed-23a5-4cbb-a077-c907faac860a-kube-api-access-6mg7m\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:29:19.562818 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:19.562590 4562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c482f1ed-23a5-4cbb-a077-c907faac860a-istio-data\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:29:19.562818 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:19.562598 4562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c482f1ed-23a5-4cbb-a077-c907faac860a-istio-podinfo\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:29:19.562818 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:19.562606 4562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c482f1ed-23a5-4cbb-a077-c907faac860a-istio-envoy\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:29:19.622801 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:19.622757 4562 generic.go:356] "Generic (PLEG): container finished" podID="c482f1ed-23a5-4cbb-a077-c907faac860a" containerID="5d7f89f3f4d2edba5bee12b91918478c155b8634625967fbc57b3372b1de93d8" exitCode=0 Oct 30 09:29:19.623023 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:19.622838 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp" Oct 30 09:29:19.623023 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:19.622843 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp" event={"ID":"c482f1ed-23a5-4cbb-a077-c907faac860a","Type":"ContainerDied","Data":"5d7f89f3f4d2edba5bee12b91918478c155b8634625967fbc57b3372b1de93d8"} Oct 30 09:29:19.623023 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:19.622943 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp" event={"ID":"c482f1ed-23a5-4cbb-a077-c907faac860a","Type":"ContainerDied","Data":"613fc9f2946802836accaa60df6826bf19fdc301b1b29d692104e4120bd61368"} Oct 30 09:29:19.623023 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:19.622976 4562 scope.go:122] "RemoveContainer" containerID="5d7f89f3f4d2edba5bee12b91918478c155b8634625967fbc57b3372b1de93d8" Oct 30 09:29:19.633691 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:19.633670 4562 scope.go:122] "RemoveContainer" containerID="5d7f89f3f4d2edba5bee12b91918478c155b8634625967fbc57b3372b1de93d8" Oct 30 09:29:19.633947 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:29:19.633928 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d7f89f3f4d2edba5bee12b91918478c155b8634625967fbc57b3372b1de93d8\": container with ID starting with 5d7f89f3f4d2edba5bee12b91918478c155b8634625967fbc57b3372b1de93d8 not found: ID does not exist" containerID="5d7f89f3f4d2edba5bee12b91918478c155b8634625967fbc57b3372b1de93d8" Oct 30 09:29:19.634000 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:19.633955 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d7f89f3f4d2edba5bee12b91918478c155b8634625967fbc57b3372b1de93d8"} err="failed to get container status \"5d7f89f3f4d2edba5bee12b91918478c155b8634625967fbc57b3372b1de93d8\": rpc error: code = NotFound desc = could not find container \"5d7f89f3f4d2edba5bee12b91918478c155b8634625967fbc57b3372b1de93d8\": container with ID starting with 5d7f89f3f4d2edba5bee12b91918478c155b8634625967fbc57b3372b1de93d8 not found: ID does not exist" Oct 30 09:29:19.679366 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:19.679267 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp"] Oct 30 09:29:19.683530 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:19.683496 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-d3de9d-openshift-default-84f8bd4445-2qrrp"] Oct 30 09:29:19.875332 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:19.875303 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c482f1ed-23a5-4cbb-a077-c907faac860a" path="/var/lib/kubelet/pods/c482f1ed-23a5-4cbb-a077-c907faac860a/volumes" Oct 30 09:29:22.756538 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:22.756504 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd"] Oct 30 09:29:22.757082 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:22.756703 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd" podUID="1eeb2baa-0243-4032-94c4-30be54e2cef5" containerName="istio-proxy" containerID="cri-o://e3951ff3cc2e9faa615cbacb394823ca303e6c5ef94243bffb871db278878cdd" gracePeriod=30 Oct 30 09:29:26.686558 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:26.686521 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw"] Oct 30 09:29:26.686988 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:26.686724 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw" podUID="0ba1f223-8fd5-499b-ac09-73fc50031a7d" containerName="istio-proxy" containerID="cri-o://d3577f1fdce308122eee270b739d549f3ab220909353ec1bad3092800432050e" gracePeriod=30 Oct 30 09:29:27.508148 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:27.504709 4562 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 09:29:27.508148 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:27.504817 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd" podUID="1eeb2baa-0243-4032-94c4-30be54e2cef5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 09:29:28.027086 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:28.027028 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd" Oct 30 09:29:28.144859 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:28.144826 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eeb2baa-0243-4032-94c4-30be54e2cef5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eeb2baa-0243-4032-94c4-30be54e2cef5-istio-envoy\") pod \"1eeb2baa-0243-4032-94c4-30be54e2cef5\" (UID: \"1eeb2baa-0243-4032-94c4-30be54e2cef5\") " Oct 30 09:29:28.145065 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:28.144878 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1eeb2baa-0243-4032-94c4-30be54e2cef5-kube-api-access-x8jkv\" (UniqueName: \"kubernetes.io/projected/1eeb2baa-0243-4032-94c4-30be54e2cef5-kube-api-access-x8jkv\") pod \"1eeb2baa-0243-4032-94c4-30be54e2cef5\" (UID: \"1eeb2baa-0243-4032-94c4-30be54e2cef5\") " Oct 30 09:29:28.145065 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:28.144918 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eeb2baa-0243-4032-94c4-30be54e2cef5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eeb2baa-0243-4032-94c4-30be54e2cef5-workload-certs\") pod \"1eeb2baa-0243-4032-94c4-30be54e2cef5\" (UID: \"1eeb2baa-0243-4032-94c4-30be54e2cef5\") " Oct 30 09:29:28.145065 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:28.144998 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1eeb2baa-0243-4032-94c4-30be54e2cef5-istio-token\" (UniqueName: \"kubernetes.io/projected/1eeb2baa-0243-4032-94c4-30be54e2cef5-istio-token\") pod \"1eeb2baa-0243-4032-94c4-30be54e2cef5\" (UID: \"1eeb2baa-0243-4032-94c4-30be54e2cef5\") " Oct 30 09:29:28.145065 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:28.145054 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1eeb2baa-0243-4032-94c4-30be54e2cef5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eeb2baa-0243-4032-94c4-30be54e2cef5-istiod-ca-cert\") pod \"1eeb2baa-0243-4032-94c4-30be54e2cef5\" (UID: \"1eeb2baa-0243-4032-94c4-30be54e2cef5\") " Oct 30 09:29:28.145293 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:28.145097 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1eeb2baa-0243-4032-94c4-30be54e2cef5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eeb2baa-0243-4032-94c4-30be54e2cef5-istio-podinfo\") pod \"1eeb2baa-0243-4032-94c4-30be54e2cef5\" (UID: \"1eeb2baa-0243-4032-94c4-30be54e2cef5\") " Oct 30 09:29:28.145293 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:28.145211 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eeb2baa-0243-4032-94c4-30be54e2cef5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eeb2baa-0243-4032-94c4-30be54e2cef5-workload-socket\") pod \"1eeb2baa-0243-4032-94c4-30be54e2cef5\" (UID: \"1eeb2baa-0243-4032-94c4-30be54e2cef5\") " Oct 30 09:29:28.145429 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:28.145276 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eeb2baa-0243-4032-94c4-30be54e2cef5-workload-certs" pod "1eeb2baa-0243-4032-94c4-30be54e2cef5" (UID: "1eeb2baa-0243-4032-94c4-30be54e2cef5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:29:28.145429 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:28.145332 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eeb2baa-0243-4032-94c4-30be54e2cef5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eeb2baa-0243-4032-94c4-30be54e2cef5-istio-data\") pod \"1eeb2baa-0243-4032-94c4-30be54e2cef5\" (UID: \"1eeb2baa-0243-4032-94c4-30be54e2cef5\") " Oct 30 09:29:28.145549 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:28.145434 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eeb2baa-0243-4032-94c4-30be54e2cef5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eeb2baa-0243-4032-94c4-30be54e2cef5-credential-socket\") pod \"1eeb2baa-0243-4032-94c4-30be54e2cef5\" (UID: \"1eeb2baa-0243-4032-94c4-30be54e2cef5\") " Oct 30 09:29:28.145549 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:28.145497 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eeb2baa-0243-4032-94c4-30be54e2cef5-workload-socket" pod "1eeb2baa-0243-4032-94c4-30be54e2cef5" (UID: "1eeb2baa-0243-4032-94c4-30be54e2cef5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:29:28.145549 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:28.145510 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eeb2baa-0243-4032-94c4-30be54e2cef5-istiod-ca-cert" pod "1eeb2baa-0243-4032-94c4-30be54e2cef5" (UID: "1eeb2baa-0243-4032-94c4-30be54e2cef5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:29:28.145714 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:28.145680 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eeb2baa-0243-4032-94c4-30be54e2cef5-istio-data" pod "1eeb2baa-0243-4032-94c4-30be54e2cef5" (UID: "1eeb2baa-0243-4032-94c4-30be54e2cef5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:29:28.145862 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:28.145838 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eeb2baa-0243-4032-94c4-30be54e2cef5-credential-socket" pod "1eeb2baa-0243-4032-94c4-30be54e2cef5" (UID: "1eeb2baa-0243-4032-94c4-30be54e2cef5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:29:28.146209 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:28.146181 4562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eeb2baa-0243-4032-94c4-30be54e2cef5-istiod-ca-cert\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:29:28.146209 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:28.146205 4562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eeb2baa-0243-4032-94c4-30be54e2cef5-workload-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:29:28.146431 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:28.146220 4562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eeb2baa-0243-4032-94c4-30be54e2cef5-istio-data\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:29:28.146431 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:28.146234 4562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eeb2baa-0243-4032-94c4-30be54e2cef5-credential-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:29:28.146431 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:28.146249 4562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eeb2baa-0243-4032-94c4-30be54e2cef5-workload-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:29:28.147689 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:28.147661 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1eeb2baa-0243-4032-94c4-30be54e2cef5-istio-podinfo" pod "1eeb2baa-0243-4032-94c4-30be54e2cef5" (UID: "1eeb2baa-0243-4032-94c4-30be54e2cef5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:29:28.147796 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:28.147770 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eeb2baa-0243-4032-94c4-30be54e2cef5-kube-api-access-x8jkv" pod "1eeb2baa-0243-4032-94c4-30be54e2cef5" (UID: "1eeb2baa-0243-4032-94c4-30be54e2cef5"). InnerVolumeSpecName "kube-api-access-x8jkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:29:28.148225 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:28.148202 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eeb2baa-0243-4032-94c4-30be54e2cef5-istio-envoy" pod "1eeb2baa-0243-4032-94c4-30be54e2cef5" (UID: "1eeb2baa-0243-4032-94c4-30be54e2cef5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:29:28.149309 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:28.149284 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eeb2baa-0243-4032-94c4-30be54e2cef5-istio-token" pod "1eeb2baa-0243-4032-94c4-30be54e2cef5" (UID: "1eeb2baa-0243-4032-94c4-30be54e2cef5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:29:28.247347 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:28.247314 4562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1eeb2baa-0243-4032-94c4-30be54e2cef5-istio-token\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:29:28.247347 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:28.247343 4562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eeb2baa-0243-4032-94c4-30be54e2cef5-istio-podinfo\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:29:28.247347 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:28.247352 4562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eeb2baa-0243-4032-94c4-30be54e2cef5-istio-envoy\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:29:28.247609 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:28.247363 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8jkv\" (UniqueName: \"kubernetes.io/projected/1eeb2baa-0243-4032-94c4-30be54e2cef5-kube-api-access-x8jkv\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:29:28.605706 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:29:28.605668 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:29:28.728442 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:28.728409 4562 generic.go:356] "Generic (PLEG): container finished" podID="1eeb2baa-0243-4032-94c4-30be54e2cef5" containerID="e3951ff3cc2e9faa615cbacb394823ca303e6c5ef94243bffb871db278878cdd" exitCode=0 Oct 30 09:29:28.728637 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:28.728498 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd" Oct 30 09:29:28.728637 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:28.728534 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd" event={"ID":"1eeb2baa-0243-4032-94c4-30be54e2cef5","Type":"ContainerDied","Data":"e3951ff3cc2e9faa615cbacb394823ca303e6c5ef94243bffb871db278878cdd"} Oct 30 09:29:28.728637 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:28.728600 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd" event={"ID":"1eeb2baa-0243-4032-94c4-30be54e2cef5","Type":"ContainerDied","Data":"9b1fa4ade541dffb2b29f4afa998214806ab5748f1869488dd2c9e3af4bfd0fd"} Oct 30 09:29:28.728637 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:28.728628 4562 scope.go:122] "RemoveContainer" containerID="e3951ff3cc2e9faa615cbacb394823ca303e6c5ef94243bffb871db278878cdd" Oct 30 09:29:28.739067 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:28.739046 4562 scope.go:122] "RemoveContainer" containerID="e3951ff3cc2e9faa615cbacb394823ca303e6c5ef94243bffb871db278878cdd" Oct 30 09:29:28.739363 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:29:28.739347 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3951ff3cc2e9faa615cbacb394823ca303e6c5ef94243bffb871db278878cdd\": container with ID starting with e3951ff3cc2e9faa615cbacb394823ca303e6c5ef94243bffb871db278878cdd not found: ID does not exist" containerID="e3951ff3cc2e9faa615cbacb394823ca303e6c5ef94243bffb871db278878cdd" Oct 30 09:29:28.739463 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:28.739374 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3951ff3cc2e9faa615cbacb394823ca303e6c5ef94243bffb871db278878cdd"} err="failed to get container status \"e3951ff3cc2e9faa615cbacb394823ca303e6c5ef94243bffb871db278878cdd\": rpc error: code = NotFound desc = could not find container \"e3951ff3cc2e9faa615cbacb394823ca303e6c5ef94243bffb871db278878cdd\": container with ID starting with e3951ff3cc2e9faa615cbacb394823ca303e6c5ef94243bffb871db278878cdd not found: ID does not exist" Oct 30 09:29:28.803027 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:28.798559 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd"] Oct 30 09:29:28.809474 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:28.809447 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-nj5dd"] Oct 30 09:29:29.995800 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:29.995752 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1eeb2baa-0243-4032-94c4-30be54e2cef5" path="/var/lib/kubelet/pods/1eeb2baa-0243-4032-94c4-30be54e2cef5/volumes" Oct 30 09:29:32.232245 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:32.232217 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw" Oct 30 09:29:32.235334 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:32.235295 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ba1f223-8fd5-499b-ac09-73fc50031a7d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ba1f223-8fd5-499b-ac09-73fc50031a7d-workload-certs\") pod \"0ba1f223-8fd5-499b-ac09-73fc50031a7d\" (UID: \"0ba1f223-8fd5-499b-ac09-73fc50031a7d\") " Oct 30 09:29:32.235506 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:32.235343 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ba1f223-8fd5-499b-ac09-73fc50031a7d-kube-api-access-qgvxm\" (UniqueName: \"kubernetes.io/projected/0ba1f223-8fd5-499b-ac09-73fc50031a7d-kube-api-access-qgvxm\") pod \"0ba1f223-8fd5-499b-ac09-73fc50031a7d\" (UID: \"0ba1f223-8fd5-499b-ac09-73fc50031a7d\") " Oct 30 09:29:32.235506 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:32.235409 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ba1f223-8fd5-499b-ac09-73fc50031a7d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ba1f223-8fd5-499b-ac09-73fc50031a7d-istio-envoy\") pod \"0ba1f223-8fd5-499b-ac09-73fc50031a7d\" (UID: \"0ba1f223-8fd5-499b-ac09-73fc50031a7d\") " Oct 30 09:29:32.235595 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:32.235540 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ba1f223-8fd5-499b-ac09-73fc50031a7d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ba1f223-8fd5-499b-ac09-73fc50031a7d-workload-socket\") pod \"0ba1f223-8fd5-499b-ac09-73fc50031a7d\" (UID: \"0ba1f223-8fd5-499b-ac09-73fc50031a7d\") " Oct 30 09:29:32.235595 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:32.235574 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ba1f223-8fd5-499b-ac09-73fc50031a7d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ba1f223-8fd5-499b-ac09-73fc50031a7d-istio-data\") pod \"0ba1f223-8fd5-499b-ac09-73fc50031a7d\" (UID: \"0ba1f223-8fd5-499b-ac09-73fc50031a7d\") " Oct 30 09:29:32.235664 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:32.235602 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ba1f223-8fd5-499b-ac09-73fc50031a7d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ba1f223-8fd5-499b-ac09-73fc50031a7d-credential-socket\") pod \"0ba1f223-8fd5-499b-ac09-73fc50031a7d\" (UID: \"0ba1f223-8fd5-499b-ac09-73fc50031a7d\") " Oct 30 09:29:32.235664 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:32.235658 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ba1f223-8fd5-499b-ac09-73fc50031a7d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ba1f223-8fd5-499b-ac09-73fc50031a7d-istiod-ca-cert\") pod \"0ba1f223-8fd5-499b-ac09-73fc50031a7d\" (UID: \"0ba1f223-8fd5-499b-ac09-73fc50031a7d\") " Oct 30 09:29:32.235731 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:32.235681 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ba1f223-8fd5-499b-ac09-73fc50031a7d-istio-token\" (UniqueName: \"kubernetes.io/projected/0ba1f223-8fd5-499b-ac09-73fc50031a7d-istio-token\") pod \"0ba1f223-8fd5-499b-ac09-73fc50031a7d\" (UID: \"0ba1f223-8fd5-499b-ac09-73fc50031a7d\") " Oct 30 09:29:32.235731 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:32.235703 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ba1f223-8fd5-499b-ac09-73fc50031a7d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ba1f223-8fd5-499b-ac09-73fc50031a7d-istio-podinfo\") pod \"0ba1f223-8fd5-499b-ac09-73fc50031a7d\" (UID: \"0ba1f223-8fd5-499b-ac09-73fc50031a7d\") " Oct 30 09:29:32.236556 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:32.236529 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ba1f223-8fd5-499b-ac09-73fc50031a7d-credential-socket" pod "0ba1f223-8fd5-499b-ac09-73fc50031a7d" (UID: "0ba1f223-8fd5-499b-ac09-73fc50031a7d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:29:32.236556 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:32.236516 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ba1f223-8fd5-499b-ac09-73fc50031a7d-workload-socket" pod "0ba1f223-8fd5-499b-ac09-73fc50031a7d" (UID: "0ba1f223-8fd5-499b-ac09-73fc50031a7d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:29:32.236707 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:32.236613 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ba1f223-8fd5-499b-ac09-73fc50031a7d-istio-data" pod "0ba1f223-8fd5-499b-ac09-73fc50031a7d" (UID: "0ba1f223-8fd5-499b-ac09-73fc50031a7d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:29:32.236821 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:32.236786 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ba1f223-8fd5-499b-ac09-73fc50031a7d-workload-certs" pod "0ba1f223-8fd5-499b-ac09-73fc50031a7d" (UID: "0ba1f223-8fd5-499b-ac09-73fc50031a7d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:29:32.236900 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:32.236880 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ba1f223-8fd5-499b-ac09-73fc50031a7d-istiod-ca-cert" pod "0ba1f223-8fd5-499b-ac09-73fc50031a7d" (UID: "0ba1f223-8fd5-499b-ac09-73fc50031a7d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:29:32.238441 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:32.238398 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ba1f223-8fd5-499b-ac09-73fc50031a7d-istio-podinfo" pod "0ba1f223-8fd5-499b-ac09-73fc50031a7d" (UID: "0ba1f223-8fd5-499b-ac09-73fc50031a7d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:29:32.238628 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:32.238597 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ba1f223-8fd5-499b-ac09-73fc50031a7d-kube-api-access-qgvxm" pod "0ba1f223-8fd5-499b-ac09-73fc50031a7d" (UID: "0ba1f223-8fd5-499b-ac09-73fc50031a7d"). InnerVolumeSpecName "kube-api-access-qgvxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:29:32.238878 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:32.238852 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ba1f223-8fd5-499b-ac09-73fc50031a7d-istio-token" pod "0ba1f223-8fd5-499b-ac09-73fc50031a7d" (UID: "0ba1f223-8fd5-499b-ac09-73fc50031a7d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:29:32.239362 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:32.239342 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ba1f223-8fd5-499b-ac09-73fc50031a7d-istio-envoy" pod "0ba1f223-8fd5-499b-ac09-73fc50031a7d" (UID: "0ba1f223-8fd5-499b-ac09-73fc50031a7d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:29:32.336900 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:32.336822 4562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ba1f223-8fd5-499b-ac09-73fc50031a7d-istio-podinfo\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:29:32.336900 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:32.336849 4562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ba1f223-8fd5-499b-ac09-73fc50031a7d-workload-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:29:32.336900 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:32.336859 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgvxm\" (UniqueName: \"kubernetes.io/projected/0ba1f223-8fd5-499b-ac09-73fc50031a7d-kube-api-access-qgvxm\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:29:32.336900 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:32.336868 4562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ba1f223-8fd5-499b-ac09-73fc50031a7d-istio-envoy\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:29:32.336900 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:32.336878 4562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ba1f223-8fd5-499b-ac09-73fc50031a7d-workload-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:29:32.336900 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:32.336887 4562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ba1f223-8fd5-499b-ac09-73fc50031a7d-istio-data\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:29:32.336900 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:32.336895 4562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ba1f223-8fd5-499b-ac09-73fc50031a7d-credential-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:29:32.336900 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:32.336903 4562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ba1f223-8fd5-499b-ac09-73fc50031a7d-istiod-ca-cert\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:29:32.337257 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:32.336911 4562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ba1f223-8fd5-499b-ac09-73fc50031a7d-istio-token\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:29:32.841289 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:32.841247 4562 generic.go:356] "Generic (PLEG): container finished" podID="0ba1f223-8fd5-499b-ac09-73fc50031a7d" containerID="d3577f1fdce308122eee270b739d549f3ab220909353ec1bad3092800432050e" exitCode=0 Oct 30 09:29:32.841515 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:32.841301 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw" Oct 30 09:29:32.841515 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:32.841329 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw" event={"ID":"0ba1f223-8fd5-499b-ac09-73fc50031a7d","Type":"ContainerDied","Data":"d3577f1fdce308122eee270b739d549f3ab220909353ec1bad3092800432050e"} Oct 30 09:29:32.841515 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:32.841415 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw" event={"ID":"0ba1f223-8fd5-499b-ac09-73fc50031a7d","Type":"ContainerDied","Data":"ab840124578573b5cf7b6be2b51050bcb655b64b3bca0286fb9846c519c113c3"} Oct 30 09:29:32.841515 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:32.841452 4562 scope.go:122] "RemoveContainer" containerID="d3577f1fdce308122eee270b739d549f3ab220909353ec1bad3092800432050e" Oct 30 09:29:32.855863 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:32.852656 4562 scope.go:122] "RemoveContainer" containerID="ed8d5f5580de3dbeb252ed0bb25fd1f08967b60724caa0a6e6ef3f6e4c964500" Oct 30 09:29:32.863260 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:32.863235 4562 scope.go:122] "RemoveContainer" containerID="d3577f1fdce308122eee270b739d549f3ab220909353ec1bad3092800432050e" Oct 30 09:29:32.863561 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:29:32.863541 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3577f1fdce308122eee270b739d549f3ab220909353ec1bad3092800432050e\": container with ID starting with d3577f1fdce308122eee270b739d549f3ab220909353ec1bad3092800432050e not found: ID does not exist" containerID="d3577f1fdce308122eee270b739d549f3ab220909353ec1bad3092800432050e" Oct 30 09:29:32.863624 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:32.863568 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3577f1fdce308122eee270b739d549f3ab220909353ec1bad3092800432050e"} err="failed to get container status \"d3577f1fdce308122eee270b739d549f3ab220909353ec1bad3092800432050e\": rpc error: code = NotFound desc = could not find container \"d3577f1fdce308122eee270b739d549f3ab220909353ec1bad3092800432050e\": container with ID starting with d3577f1fdce308122eee270b739d549f3ab220909353ec1bad3092800432050e not found: ID does not exist" Oct 30 09:29:32.863624 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:32.863586 4562 scope.go:122] "RemoveContainer" containerID="ed8d5f5580de3dbeb252ed0bb25fd1f08967b60724caa0a6e6ef3f6e4c964500" Oct 30 09:29:32.863828 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:29:32.863815 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed8d5f5580de3dbeb252ed0bb25fd1f08967b60724caa0a6e6ef3f6e4c964500\": container with ID starting with ed8d5f5580de3dbeb252ed0bb25fd1f08967b60724caa0a6e6ef3f6e4c964500 not found: ID does not exist" containerID="ed8d5f5580de3dbeb252ed0bb25fd1f08967b60724caa0a6e6ef3f6e4c964500" Oct 30 09:29:32.863870 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:32.863832 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed8d5f5580de3dbeb252ed0bb25fd1f08967b60724caa0a6e6ef3f6e4c964500"} err="failed to get container status \"ed8d5f5580de3dbeb252ed0bb25fd1f08967b60724caa0a6e6ef3f6e4c964500\": rpc error: code = NotFound desc = could not find container \"ed8d5f5580de3dbeb252ed0bb25fd1f08967b60724caa0a6e6ef3f6e4c964500\": container with ID starting with ed8d5f5580de3dbeb252ed0bb25fd1f08967b60724caa0a6e6ef3f6e4c964500 not found: ID does not exist" Oct 30 09:29:32.908701 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:32.908675 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw"] Oct 30 09:29:32.921638 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:32.921611 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-rm2bw"] Oct 30 09:29:33.874797 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:33.874758 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ba1f223-8fd5-499b-ac09-73fc50031a7d" path="/var/lib/kubelet/pods/0ba1f223-8fd5-499b-ac09-73fc50031a7d/volumes" Oct 30 09:29:42.517525 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:42.517484 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-15e1cc-645f5cbbc4-j4j7f"] Oct 30 09:29:42.573283 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:42.573250 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-15e1cc-674c7798f9-9vpwd"] Oct 30 09:29:43.653971 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:43.653930 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-51e091-5ddbc647c9-rpb26"] Oct 30 09:29:43.654461 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:43.654277 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-51e091-5ddbc647c9-rpb26" podUID="5149468c-48b5-43ee-9eba-b3b43f41af6d" containerName="maas-api" containerID="cri-o://799122ba9c236f204c0702b715ddc327b30d2516497c4a2ad8d12666e8a66d82" gracePeriod=30 Oct 30 09:29:43.891985 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:43.891950 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-51e091-769f5d944b-dgc2p"] Oct 30 09:29:43.892178 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:43.892153 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-51e091-769f5d944b-dgc2p" podUID="94ef492d-56fb-4e63-9044-063e1ffe8b3a" containerName="payload-processing" containerID="cri-o://089f06d3527363b1194ddecfba25ca5d7e05451c89891aba781e367f57faac8e" gracePeriod=30 Oct 30 09:29:43.909471 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:43.909448 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-51e091-54f95fbdcf-lpl7d"] Oct 30 09:29:43.909844 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:43.909817 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-51e091-54f95fbdcf-lpl7d" podUID="3e873180-8cd3-4f72-ad69-2290f73570c6" containerName="payload-pre-processing" containerID="cri-o://a55b8a37ea13ca0a361a5b9b2bf3e1f2844fd53e0aeb058c3f0a1bf5868218b1" gracePeriod=30 Oct 30 09:29:43.926048 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:43.926024 4562 generic.go:356] "Generic (PLEG): container finished" podID="5149468c-48b5-43ee-9eba-b3b43f41af6d" containerID="799122ba9c236f204c0702b715ddc327b30d2516497c4a2ad8d12666e8a66d82" exitCode=0 Oct 30 09:29:43.926151 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:43.926092 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-51e091-5ddbc647c9-rpb26" event={"ID":"5149468c-48b5-43ee-9eba-b3b43f41af6d","Type":"ContainerDied","Data":"799122ba9c236f204c0702b715ddc327b30d2516497c4a2ad8d12666e8a66d82"} Oct 30 09:29:43.926151 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:43.926146 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-51e091-5ddbc647c9-rpb26" event={"ID":"5149468c-48b5-43ee-9eba-b3b43f41af6d","Type":"ContainerDied","Data":"fd39808cc8001c772c88155a52bc2e95567c171ff534af56b7fec18f6cb74fc9"} Oct 30 09:29:43.926224 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:43.926163 4562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd39808cc8001c772c88155a52bc2e95567c171ff534af56b7fec18f6cb74fc9" Oct 30 09:29:44.071896 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:44.069690 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-51e091-5ddbc647c9-rpb26" Oct 30 09:29:44.129564 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:44.129489 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-51e091-54f95fbdcf-lpl7d" podUID="3e873180-8cd3-4f72-ad69-2290f73570c6" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.154:9004: connect: connection refused" Oct 30 09:29:44.196776 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:44.196738 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-51e091-769f5d944b-dgc2p" Oct 30 09:29:44.221280 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:44.221249 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5149468c-48b5-43ee-9eba-b3b43f41af6d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5149468c-48b5-43ee-9eba-b3b43f41af6d-metrics-certs\") pod \"5149468c-48b5-43ee-9eba-b3b43f41af6d\" (UID: \"5149468c-48b5-43ee-9eba-b3b43f41af6d\") " Oct 30 09:29:44.221461 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:44.221309 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5149468c-48b5-43ee-9eba-b3b43f41af6d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5149468c-48b5-43ee-9eba-b3b43f41af6d-maas-api-tls\") pod \"5149468c-48b5-43ee-9eba-b3b43f41af6d\" (UID: \"5149468c-48b5-43ee-9eba-b3b43f41af6d\") " Oct 30 09:29:44.221544 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:44.221525 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5149468c-48b5-43ee-9eba-b3b43f41af6d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5149468c-48b5-43ee-9eba-b3b43f41af6d-openshift-service-ca\") pod \"5149468c-48b5-43ee-9eba-b3b43f41af6d\" (UID: \"5149468c-48b5-43ee-9eba-b3b43f41af6d\") " Oct 30 09:29:44.222021 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:44.221981 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5149468c-48b5-43ee-9eba-b3b43f41af6d-openshift-service-ca" pod "5149468c-48b5-43ee-9eba-b3b43f41af6d" (UID: "5149468c-48b5-43ee-9eba-b3b43f41af6d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:29:44.222344 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:44.222322 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5149468c-48b5-43ee-9eba-b3b43f41af6d-kube-api-access-d8lmj\" (UniqueName: \"kubernetes.io/projected/5149468c-48b5-43ee-9eba-b3b43f41af6d-kube-api-access-d8lmj\") pod \"5149468c-48b5-43ee-9eba-b3b43f41af6d\" (UID: \"5149468c-48b5-43ee-9eba-b3b43f41af6d\") " Oct 30 09:29:44.223441 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:44.223422 4562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5149468c-48b5-43ee-9eba-b3b43f41af6d-openshift-service-ca\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:29:44.223999 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:44.223974 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5149468c-48b5-43ee-9eba-b3b43f41af6d-maas-api-tls" pod "5149468c-48b5-43ee-9eba-b3b43f41af6d" (UID: "5149468c-48b5-43ee-9eba-b3b43f41af6d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:29:44.225009 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:44.224973 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5149468c-48b5-43ee-9eba-b3b43f41af6d-kube-api-access-d8lmj" pod "5149468c-48b5-43ee-9eba-b3b43f41af6d" (UID: "5149468c-48b5-43ee-9eba-b3b43f41af6d"). InnerVolumeSpecName "kube-api-access-d8lmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:29:44.234882 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:44.226399 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5149468c-48b5-43ee-9eba-b3b43f41af6d-metrics-certs" pod "5149468c-48b5-43ee-9eba-b3b43f41af6d" (UID: "5149468c-48b5-43ee-9eba-b3b43f41af6d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:29:44.349503 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:44.345948 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94ef492d-56fb-4e63-9044-063e1ffe8b3a-kube-api-access-plgvm\" (UniqueName: \"kubernetes.io/projected/94ef492d-56fb-4e63-9044-063e1ffe8b3a-kube-api-access-plgvm\") pod \"94ef492d-56fb-4e63-9044-063e1ffe8b3a\" (UID: \"94ef492d-56fb-4e63-9044-063e1ffe8b3a\") " Oct 30 09:29:44.349503 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:44.346104 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94ef492d-56fb-4e63-9044-063e1ffe8b3a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94ef492d-56fb-4e63-9044-063e1ffe8b3a-plugins-config-volume\") pod \"94ef492d-56fb-4e63-9044-063e1ffe8b3a\" (UID: \"94ef492d-56fb-4e63-9044-063e1ffe8b3a\") " Oct 30 09:29:44.349503 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:44.346653 4562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5149468c-48b5-43ee-9eba-b3b43f41af6d-maas-api-tls\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:29:44.349503 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:44.346675 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8lmj\" (UniqueName: \"kubernetes.io/projected/5149468c-48b5-43ee-9eba-b3b43f41af6d-kube-api-access-d8lmj\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:29:44.349503 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:44.346690 4562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5149468c-48b5-43ee-9eba-b3b43f41af6d-metrics-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:29:44.350290 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:44.350233 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94ef492d-56fb-4e63-9044-063e1ffe8b3a-plugins-config-volume" pod "94ef492d-56fb-4e63-9044-063e1ffe8b3a" (UID: "94ef492d-56fb-4e63-9044-063e1ffe8b3a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:29:44.381603 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:44.381532 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94ef492d-56fb-4e63-9044-063e1ffe8b3a-kube-api-access-plgvm" pod "94ef492d-56fb-4e63-9044-063e1ffe8b3a" (UID: "94ef492d-56fb-4e63-9044-063e1ffe8b3a"). InnerVolumeSpecName "kube-api-access-plgvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:29:44.395595 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:44.395255 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-51e091-54f95fbdcf-lpl7d" Oct 30 09:29:44.448448 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:44.448375 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94ef492d-56fb-4e63-9044-063e1ffe8b3a-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:29:44.448652 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:44.448451 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plgvm\" (UniqueName: \"kubernetes.io/projected/94ef492d-56fb-4e63-9044-063e1ffe8b3a-kube-api-access-plgvm\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:29:44.548874 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:44.548844 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e873180-8cd3-4f72-ad69-2290f73570c6-kube-api-access-h9ddd\" (UniqueName: \"kubernetes.io/projected/3e873180-8cd3-4f72-ad69-2290f73570c6-kube-api-access-h9ddd\") pod \"3e873180-8cd3-4f72-ad69-2290f73570c6\" (UID: \"3e873180-8cd3-4f72-ad69-2290f73570c6\") " Oct 30 09:29:44.549076 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:44.549051 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e873180-8cd3-4f72-ad69-2290f73570c6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e873180-8cd3-4f72-ad69-2290f73570c6-plugins-config-volume\") pod \"3e873180-8cd3-4f72-ad69-2290f73570c6\" (UID: \"3e873180-8cd3-4f72-ad69-2290f73570c6\") " Oct 30 09:29:44.549455 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:44.549431 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e873180-8cd3-4f72-ad69-2290f73570c6-plugins-config-volume" pod "3e873180-8cd3-4f72-ad69-2290f73570c6" (UID: "3e873180-8cd3-4f72-ad69-2290f73570c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:29:44.550973 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:44.550951 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e873180-8cd3-4f72-ad69-2290f73570c6-kube-api-access-h9ddd" pod "3e873180-8cd3-4f72-ad69-2290f73570c6" (UID: "3e873180-8cd3-4f72-ad69-2290f73570c6"). InnerVolumeSpecName "kube-api-access-h9ddd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:29:44.655993 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:44.655746 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e873180-8cd3-4f72-ad69-2290f73570c6-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:29:44.655993 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:44.655789 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9ddd\" (UniqueName: \"kubernetes.io/projected/3e873180-8cd3-4f72-ad69-2290f73570c6-kube-api-access-h9ddd\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:29:44.933614 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:44.933572 4562 generic.go:356] "Generic (PLEG): container finished" podID="3e873180-8cd3-4f72-ad69-2290f73570c6" containerID="a55b8a37ea13ca0a361a5b9b2bf3e1f2844fd53e0aeb058c3f0a1bf5868218b1" exitCode=0 Oct 30 09:29:44.933848 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:44.933629 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-51e091-54f95fbdcf-lpl7d" Oct 30 09:29:44.933848 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:44.933664 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-51e091-54f95fbdcf-lpl7d" event={"ID":"3e873180-8cd3-4f72-ad69-2290f73570c6","Type":"ContainerDied","Data":"a55b8a37ea13ca0a361a5b9b2bf3e1f2844fd53e0aeb058c3f0a1bf5868218b1"} Oct 30 09:29:44.933848 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:44.933718 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-51e091-54f95fbdcf-lpl7d" event={"ID":"3e873180-8cd3-4f72-ad69-2290f73570c6","Type":"ContainerDied","Data":"8eca257af17627590f758453758fd84848f63319cfe90be4fa9f8c1e9a25270c"} Oct 30 09:29:44.933848 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:44.933739 4562 scope.go:122] "RemoveContainer" containerID="a55b8a37ea13ca0a361a5b9b2bf3e1f2844fd53e0aeb058c3f0a1bf5868218b1" Oct 30 09:29:44.935149 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:44.935122 4562 generic.go:356] "Generic (PLEG): container finished" podID="94ef492d-56fb-4e63-9044-063e1ffe8b3a" containerID="089f06d3527363b1194ddecfba25ca5d7e05451c89891aba781e367f57faac8e" exitCode=0 Oct 30 09:29:44.935245 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:44.935189 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-51e091-769f5d944b-dgc2p" Oct 30 09:29:44.935245 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:44.935223 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-51e091-769f5d944b-dgc2p" event={"ID":"94ef492d-56fb-4e63-9044-063e1ffe8b3a","Type":"ContainerDied","Data":"089f06d3527363b1194ddecfba25ca5d7e05451c89891aba781e367f57faac8e"} Oct 30 09:29:44.935347 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:44.935274 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-51e091-769f5d944b-dgc2p" event={"ID":"94ef492d-56fb-4e63-9044-063e1ffe8b3a","Type":"ContainerDied","Data":"8f37892e1c11edeba0bfa9a50e6f6b62799c0b94241ae0a7df17bbf7d3eac751"} Oct 30 09:29:44.935418 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:44.935398 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-51e091-5ddbc647c9-rpb26" Oct 30 09:29:44.945486 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:44.945467 4562 scope.go:122] "RemoveContainer" containerID="a55b8a37ea13ca0a361a5b9b2bf3e1f2844fd53e0aeb058c3f0a1bf5868218b1" Oct 30 09:29:44.945788 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:29:44.945763 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a55b8a37ea13ca0a361a5b9b2bf3e1f2844fd53e0aeb058c3f0a1bf5868218b1\": container with ID starting with a55b8a37ea13ca0a361a5b9b2bf3e1f2844fd53e0aeb058c3f0a1bf5868218b1 not found: ID does not exist" containerID="a55b8a37ea13ca0a361a5b9b2bf3e1f2844fd53e0aeb058c3f0a1bf5868218b1" Oct 30 09:29:44.945888 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:44.945797 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a55b8a37ea13ca0a361a5b9b2bf3e1f2844fd53e0aeb058c3f0a1bf5868218b1"} err="failed to get container status \"a55b8a37ea13ca0a361a5b9b2bf3e1f2844fd53e0aeb058c3f0a1bf5868218b1\": rpc error: code = NotFound desc = could not find container \"a55b8a37ea13ca0a361a5b9b2bf3e1f2844fd53e0aeb058c3f0a1bf5868218b1\": container with ID starting with a55b8a37ea13ca0a361a5b9b2bf3e1f2844fd53e0aeb058c3f0a1bf5868218b1 not found: ID does not exist" Oct 30 09:29:44.945888 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:44.945819 4562 scope.go:122] "RemoveContainer" containerID="089f06d3527363b1194ddecfba25ca5d7e05451c89891aba781e367f57faac8e" Oct 30 09:29:44.955526 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:44.955510 4562 scope.go:122] "RemoveContainer" containerID="089f06d3527363b1194ddecfba25ca5d7e05451c89891aba781e367f57faac8e" Oct 30 09:29:44.955805 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:29:44.955781 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"089f06d3527363b1194ddecfba25ca5d7e05451c89891aba781e367f57faac8e\": container with ID starting with 089f06d3527363b1194ddecfba25ca5d7e05451c89891aba781e367f57faac8e not found: ID does not exist" containerID="089f06d3527363b1194ddecfba25ca5d7e05451c89891aba781e367f57faac8e" Oct 30 09:29:44.955891 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:44.955810 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"089f06d3527363b1194ddecfba25ca5d7e05451c89891aba781e367f57faac8e"} err="failed to get container status \"089f06d3527363b1194ddecfba25ca5d7e05451c89891aba781e367f57faac8e\": rpc error: code = NotFound desc = could not find container \"089f06d3527363b1194ddecfba25ca5d7e05451c89891aba781e367f57faac8e\": container with ID starting with 089f06d3527363b1194ddecfba25ca5d7e05451c89891aba781e367f57faac8e not found: ID does not exist" Oct 30 09:29:45.019128 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:45.019047 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-51e091-5ddbc647c9-rpb26"] Oct 30 09:29:45.034298 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:45.034268 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-51e091-5ddbc647c9-rpb26"] Oct 30 09:29:45.059425 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:45.058782 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-51e091-769f5d944b-dgc2p"] Oct 30 09:29:45.063602 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:45.063577 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-51e091-769f5d944b-dgc2p"] Oct 30 09:29:45.098072 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:45.098042 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-51e091-54f95fbdcf-lpl7d"] Oct 30 09:29:45.101837 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:45.101810 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-51e091-54f95fbdcf-lpl7d"] Oct 30 09:29:45.130958 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:45.130906 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-b-51e091-769f5d944b-dgc2p" podUID="94ef492d-56fb-4e63-9044-063e1ffe8b3a" containerName="payload-processing" probeResult="failure" output="dial tcp 10.133.0.153:9004: i/o timeout" Oct 30 09:29:45.909418 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:45.902367 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e873180-8cd3-4f72-ad69-2290f73570c6" path="/var/lib/kubelet/pods/3e873180-8cd3-4f72-ad69-2290f73570c6/volumes" Oct 30 09:29:45.909418 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:45.903028 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5149468c-48b5-43ee-9eba-b3b43f41af6d" path="/var/lib/kubelet/pods/5149468c-48b5-43ee-9eba-b3b43f41af6d/volumes" Oct 30 09:29:45.909418 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:45.903545 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94ef492d-56fb-4e63-9044-063e1ffe8b3a" path="/var/lib/kubelet/pods/94ef492d-56fb-4e63-9044-063e1ffe8b3a/volumes" Oct 30 09:29:47.518024 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:47.517984 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-15e1cc-645f5cbbc4-j4j7f" podUID="8cab9c95-15d2-48f5-99a1-1d6df3fd2333" containerName="payload-processing" containerID="cri-o://682090c047dddac9e841455437acca838bc793f79bfbff46dc30913d27d9c91f" gracePeriod=25 Oct 30 09:29:47.574040 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:47.574004 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-15e1cc-674c7798f9-9vpwd" podUID="9597695b-a089-4c8f-8a65-2ec83df78302" containerName="payload-pre-processing" containerID="cri-o://2f5b1d301ee52170d64f75523e1a14dc634edbbe7fe57cebc51ab5f26b437b1d" gracePeriod=25 Oct 30 09:29:47.891431 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:47.891099 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-15e1cc-645f5cbbc4-j4j7f" Oct 30 09:29:47.920665 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:47.920644 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-15e1cc-674c7798f9-9vpwd" Oct 30 09:29:47.965343 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:47.965314 4562 generic.go:356] "Generic (PLEG): container finished" podID="8cab9c95-15d2-48f5-99a1-1d6df3fd2333" containerID="682090c047dddac9e841455437acca838bc793f79bfbff46dc30913d27d9c91f" exitCode=0 Oct 30 09:29:47.965517 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:47.965359 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-15e1cc-645f5cbbc4-j4j7f" Oct 30 09:29:47.965517 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:47.965423 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-15e1cc-645f5cbbc4-j4j7f" event={"ID":"8cab9c95-15d2-48f5-99a1-1d6df3fd2333","Type":"ContainerDied","Data":"682090c047dddac9e841455437acca838bc793f79bfbff46dc30913d27d9c91f"} Oct 30 09:29:47.965517 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:47.965475 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-15e1cc-645f5cbbc4-j4j7f" event={"ID":"8cab9c95-15d2-48f5-99a1-1d6df3fd2333","Type":"ContainerDied","Data":"5903d12aa2e2cb4cbc1f4f39455a93c4cee577f4dafc546d4739f8e073b4424a"} Oct 30 09:29:47.965517 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:47.965499 4562 scope.go:122] "RemoveContainer" containerID="682090c047dddac9e841455437acca838bc793f79bfbff46dc30913d27d9c91f" Oct 30 09:29:47.966883 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:47.966858 4562 generic.go:356] "Generic (PLEG): container finished" podID="9597695b-a089-4c8f-8a65-2ec83df78302" containerID="2f5b1d301ee52170d64f75523e1a14dc634edbbe7fe57cebc51ab5f26b437b1d" exitCode=0 Oct 30 09:29:47.966986 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:47.966923 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-15e1cc-674c7798f9-9vpwd" event={"ID":"9597695b-a089-4c8f-8a65-2ec83df78302","Type":"ContainerDied","Data":"2f5b1d301ee52170d64f75523e1a14dc634edbbe7fe57cebc51ab5f26b437b1d"} Oct 30 09:29:47.966986 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:47.966974 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-15e1cc-674c7798f9-9vpwd" event={"ID":"9597695b-a089-4c8f-8a65-2ec83df78302","Type":"ContainerDied","Data":"f26f341488fd76e0d73246ee7069535612a8f84198351a5b8395c7645127a695"} Oct 30 09:29:47.967109 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:47.967094 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-15e1cc-674c7798f9-9vpwd" Oct 30 09:29:47.977069 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:47.977044 4562 scope.go:122] "RemoveContainer" containerID="682090c047dddac9e841455437acca838bc793f79bfbff46dc30913d27d9c91f" Oct 30 09:29:47.977342 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:29:47.977308 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"682090c047dddac9e841455437acca838bc793f79bfbff46dc30913d27d9c91f\": container with ID starting with 682090c047dddac9e841455437acca838bc793f79bfbff46dc30913d27d9c91f not found: ID does not exist" containerID="682090c047dddac9e841455437acca838bc793f79bfbff46dc30913d27d9c91f" Oct 30 09:29:47.977529 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:47.977349 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"682090c047dddac9e841455437acca838bc793f79bfbff46dc30913d27d9c91f"} err="failed to get container status \"682090c047dddac9e841455437acca838bc793f79bfbff46dc30913d27d9c91f\": rpc error: code = NotFound desc = could not find container \"682090c047dddac9e841455437acca838bc793f79bfbff46dc30913d27d9c91f\": container with ID starting with 682090c047dddac9e841455437acca838bc793f79bfbff46dc30913d27d9c91f not found: ID does not exist" Oct 30 09:29:47.977529 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:47.977367 4562 scope.go:122] "RemoveContainer" containerID="2f5b1d301ee52170d64f75523e1a14dc634edbbe7fe57cebc51ab5f26b437b1d" Oct 30 09:29:47.992621 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:47.992601 4562 scope.go:122] "RemoveContainer" containerID="2f5b1d301ee52170d64f75523e1a14dc634edbbe7fe57cebc51ab5f26b437b1d" Oct 30 09:29:47.992878 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:29:47.992860 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f5b1d301ee52170d64f75523e1a14dc634edbbe7fe57cebc51ab5f26b437b1d\": container with ID starting with 2f5b1d301ee52170d64f75523e1a14dc634edbbe7fe57cebc51ab5f26b437b1d not found: ID does not exist" containerID="2f5b1d301ee52170d64f75523e1a14dc634edbbe7fe57cebc51ab5f26b437b1d" Oct 30 09:29:47.992959 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:47.992886 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f5b1d301ee52170d64f75523e1a14dc634edbbe7fe57cebc51ab5f26b437b1d"} err="failed to get container status \"2f5b1d301ee52170d64f75523e1a14dc634edbbe7fe57cebc51ab5f26b437b1d\": rpc error: code = NotFound desc = could not find container \"2f5b1d301ee52170d64f75523e1a14dc634edbbe7fe57cebc51ab5f26b437b1d\": container with ID starting with 2f5b1d301ee52170d64f75523e1a14dc634edbbe7fe57cebc51ab5f26b437b1d not found: ID does not exist" Oct 30 09:29:47.998565 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:47.998545 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9597695b-a089-4c8f-8a65-2ec83df78302-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9597695b-a089-4c8f-8a65-2ec83df78302-plugins-config-volume\") pod \"9597695b-a089-4c8f-8a65-2ec83df78302\" (UID: \"9597695b-a089-4c8f-8a65-2ec83df78302\") " Oct 30 09:29:47.998670 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:47.998651 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cab9c95-15d2-48f5-99a1-1d6df3fd2333-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cab9c95-15d2-48f5-99a1-1d6df3fd2333-plugins-config-volume\") pod \"8cab9c95-15d2-48f5-99a1-1d6df3fd2333\" (UID: \"8cab9c95-15d2-48f5-99a1-1d6df3fd2333\") " Oct 30 09:29:47.998795 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:47.998776 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9597695b-a089-4c8f-8a65-2ec83df78302-kube-api-access-tbjrg\" (UniqueName: \"kubernetes.io/projected/9597695b-a089-4c8f-8a65-2ec83df78302-kube-api-access-tbjrg\") pod \"9597695b-a089-4c8f-8a65-2ec83df78302\" (UID: \"9597695b-a089-4c8f-8a65-2ec83df78302\") " Oct 30 09:29:47.998867 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:47.998821 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cab9c95-15d2-48f5-99a1-1d6df3fd2333-kube-api-access-z7z78\" (UniqueName: \"kubernetes.io/projected/8cab9c95-15d2-48f5-99a1-1d6df3fd2333-kube-api-access-z7z78\") pod \"8cab9c95-15d2-48f5-99a1-1d6df3fd2333\" (UID: \"8cab9c95-15d2-48f5-99a1-1d6df3fd2333\") " Oct 30 09:29:47.998960 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:47.998941 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9597695b-a089-4c8f-8a65-2ec83df78302-plugins-config-volume" pod "9597695b-a089-4c8f-8a65-2ec83df78302" (UID: "9597695b-a089-4c8f-8a65-2ec83df78302"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:29:47.999036 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:47.999012 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cab9c95-15d2-48f5-99a1-1d6df3fd2333-plugins-config-volume" pod "8cab9c95-15d2-48f5-99a1-1d6df3fd2333" (UID: "8cab9c95-15d2-48f5-99a1-1d6df3fd2333"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:29:47.999593 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:47.999568 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9597695b-a089-4c8f-8a65-2ec83df78302-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:29:47.999693 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:47.999598 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cab9c95-15d2-48f5-99a1-1d6df3fd2333-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:29:48.001278 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:48.001251 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9597695b-a089-4c8f-8a65-2ec83df78302-kube-api-access-tbjrg" pod "9597695b-a089-4c8f-8a65-2ec83df78302" (UID: "9597695b-a089-4c8f-8a65-2ec83df78302"). InnerVolumeSpecName "kube-api-access-tbjrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:29:48.001481 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:48.001458 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cab9c95-15d2-48f5-99a1-1d6df3fd2333-kube-api-access-z7z78" pod "8cab9c95-15d2-48f5-99a1-1d6df3fd2333" (UID: "8cab9c95-15d2-48f5-99a1-1d6df3fd2333"). InnerVolumeSpecName "kube-api-access-z7z78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:29:48.101279 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:48.101208 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbjrg\" (UniqueName: \"kubernetes.io/projected/9597695b-a089-4c8f-8a65-2ec83df78302-kube-api-access-tbjrg\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:29:48.101279 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:48.101236 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7z78\" (UniqueName: \"kubernetes.io/projected/8cab9c95-15d2-48f5-99a1-1d6df3fd2333-kube-api-access-z7z78\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:29:48.319046 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:48.319016 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-15e1cc-645f5cbbc4-j4j7f"] Oct 30 09:29:48.323191 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:48.323162 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-15e1cc-645f5cbbc4-j4j7f"] Oct 30 09:29:48.357405 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:48.357326 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-15e1cc-674c7798f9-9vpwd"] Oct 30 09:29:48.363424 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:48.363398 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-15e1cc-674c7798f9-9vpwd"] Oct 30 09:29:49.585121 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.585087 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c"] Oct 30 09:29:49.585903 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.585865 4562 cpu_manager.go:414] "containerMap: removing container" podUID="56e6a564-249b-463f-a531-e1177d1c8150" containerName="maas-api" Oct 30 09:29:49.585903 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.585893 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56e6a564-249b-463f-a531-e1177d1c8150" containerName="maas-api" Oct 30 09:29:49.586040 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.585911 4562 cpu_manager.go:414] "containerMap: removing container" podUID="9597695b-a089-4c8f-8a65-2ec83df78302" containerName="payload-pre-processing" Oct 30 09:29:49.586040 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.585917 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9597695b-a089-4c8f-8a65-2ec83df78302" containerName="payload-pre-processing" Oct 30 09:29:49.586040 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.585932 4562 cpu_manager.go:414] "containerMap: removing container" podUID="0141a7df-acc1-4bb4-9cd8-c66e44ee4eb4" containerName="payload-processing" Oct 30 09:29:49.586040 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.585936 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0141a7df-acc1-4bb4-9cd8-c66e44ee4eb4" containerName="payload-processing" Oct 30 09:29:49.586040 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.585956 4562 cpu_manager.go:414] "containerMap: removing container" podUID="3e873180-8cd3-4f72-ad69-2290f73570c6" containerName="payload-pre-processing" Oct 30 09:29:49.586040 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.585962 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e873180-8cd3-4f72-ad69-2290f73570c6" containerName="payload-pre-processing" Oct 30 09:29:49.586040 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.585974 4562 cpu_manager.go:414] "containerMap: removing container" podUID="0ba1f223-8fd5-499b-ac09-73fc50031a7d" containerName="istio-proxy" Oct 30 09:29:49.586040 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.585981 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ba1f223-8fd5-499b-ac09-73fc50031a7d" containerName="istio-proxy" Oct 30 09:29:49.586040 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.585988 4562 cpu_manager.go:414] "containerMap: removing container" podUID="1eeb2baa-0243-4032-94c4-30be54e2cef5" containerName="istio-proxy" Oct 30 09:29:49.586040 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.585992 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1eeb2baa-0243-4032-94c4-30be54e2cef5" containerName="istio-proxy" Oct 30 09:29:49.586040 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.586003 4562 cpu_manager.go:414] "containerMap: removing container" podUID="94ef492d-56fb-4e63-9044-063e1ffe8b3a" containerName="payload-processing" Oct 30 09:29:49.586040 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.586007 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94ef492d-56fb-4e63-9044-063e1ffe8b3a" containerName="payload-processing" Oct 30 09:29:49.586040 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.586015 4562 cpu_manager.go:414] "containerMap: removing container" podUID="80a75bd3-f6d2-4a1b-ab28-d7111f61f5e0" containerName="payload-pre-processing" Oct 30 09:29:49.586040 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.586020 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80a75bd3-f6d2-4a1b-ab28-d7111f61f5e0" containerName="payload-pre-processing" Oct 30 09:29:49.586040 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.586029 4562 cpu_manager.go:414] "containerMap: removing container" podUID="8cab9c95-15d2-48f5-99a1-1d6df3fd2333" containerName="payload-processing" Oct 30 09:29:49.586040 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.586035 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cab9c95-15d2-48f5-99a1-1d6df3fd2333" containerName="payload-processing" Oct 30 09:29:49.586560 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.586053 4562 cpu_manager.go:414] "containerMap: removing container" podUID="5149468c-48b5-43ee-9eba-b3b43f41af6d" containerName="maas-api" Oct 30 09:29:49.586560 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.586057 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5149468c-48b5-43ee-9eba-b3b43f41af6d" containerName="maas-api" Oct 30 09:29:49.586560 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.586065 4562 cpu_manager.go:414] "containerMap: removing container" podUID="c482f1ed-23a5-4cbb-a077-c907faac860a" containerName="istio-proxy" Oct 30 09:29:49.586560 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.586069 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c482f1ed-23a5-4cbb-a077-c907faac860a" containerName="istio-proxy" Oct 30 09:29:49.586560 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.586151 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e873180-8cd3-4f72-ad69-2290f73570c6" containerName="payload-pre-processing" Oct 30 09:29:49.586560 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.586162 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ba1f223-8fd5-499b-ac09-73fc50031a7d" containerName="istio-proxy" Oct 30 09:29:49.586560 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.586167 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="1eeb2baa-0243-4032-94c4-30be54e2cef5" containerName="istio-proxy" Oct 30 09:29:49.586560 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.586175 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ba1f223-8fd5-499b-ac09-73fc50031a7d" containerName="istio-proxy" Oct 30 09:29:49.586560 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.586180 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="94ef492d-56fb-4e63-9044-063e1ffe8b3a" containerName="payload-processing" Oct 30 09:29:49.586560 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.586186 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="80a75bd3-f6d2-4a1b-ab28-d7111f61f5e0" containerName="payload-pre-processing" Oct 30 09:29:49.586560 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.586191 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cab9c95-15d2-48f5-99a1-1d6df3fd2333" containerName="payload-processing" Oct 30 09:29:49.586560 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.586198 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="5149468c-48b5-43ee-9eba-b3b43f41af6d" containerName="maas-api" Oct 30 09:29:49.586560 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.586204 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="c482f1ed-23a5-4cbb-a077-c907faac860a" containerName="istio-proxy" Oct 30 09:29:49.586560 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.586209 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="56e6a564-249b-463f-a531-e1177d1c8150" containerName="maas-api" Oct 30 09:29:49.586560 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.586213 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="9597695b-a089-4c8f-8a65-2ec83df78302" containerName="payload-pre-processing" Oct 30 09:29:49.586560 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.586219 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="0141a7df-acc1-4bb4-9cd8-c66e44ee4eb4" containerName="payload-processing" Oct 30 09:29:49.591362 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.591341 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c" Oct 30 09:29:49.609710 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.609687 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-c9232f-openshift-default-dockercfg-w7jlr\"" Oct 30 09:29:49.627985 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.627960 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c"] Oct 30 09:29:49.632629 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.632604 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh"] Oct 30 09:29:49.633513 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.633488 4562 cpu_manager.go:414] "containerMap: removing container" podUID="0ba1f223-8fd5-499b-ac09-73fc50031a7d" containerName="istio-proxy" Oct 30 09:29:49.633513 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.633513 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ba1f223-8fd5-499b-ac09-73fc50031a7d" containerName="istio-proxy" Oct 30 09:29:49.640689 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.638713 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh" Oct 30 09:29:49.671654 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.671608 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh"] Oct 30 09:29:49.722118 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.722086 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/028fea98-1060-44f3-887a-bf0a119b428b-workload-socket\") pod \"e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c\" (UID: \"028fea98-1060-44f3-887a-bf0a119b428b\") " pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c" Oct 30 09:29:49.722290 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.722133 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p57pv\" (UniqueName: \"kubernetes.io/projected/028fea98-1060-44f3-887a-bf0a119b428b-kube-api-access-p57pv\") pod \"e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c\" (UID: \"028fea98-1060-44f3-887a-bf0a119b428b\") " pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c" Oct 30 09:29:49.722290 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.722213 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/028fea98-1060-44f3-887a-bf0a119b428b-istio-podinfo\") pod \"e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c\" (UID: \"028fea98-1060-44f3-887a-bf0a119b428b\") " pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c" Oct 30 09:29:49.722290 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.722278 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/028fea98-1060-44f3-887a-bf0a119b428b-istio-data\") pod \"e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c\" (UID: \"028fea98-1060-44f3-887a-bf0a119b428b\") " pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c" Oct 30 09:29:49.722432 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.722305 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/028fea98-1060-44f3-887a-bf0a119b428b-workload-certs\") pod \"e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c\" (UID: \"028fea98-1060-44f3-887a-bf0a119b428b\") " pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c" Oct 30 09:29:49.722432 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.722334 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/028fea98-1060-44f3-887a-bf0a119b428b-istio-envoy\") pod \"e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c\" (UID: \"028fea98-1060-44f3-887a-bf0a119b428b\") " pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c" Oct 30 09:29:49.722565 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.722542 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/028fea98-1060-44f3-887a-bf0a119b428b-istio-token\") pod \"e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c\" (UID: \"028fea98-1060-44f3-887a-bf0a119b428b\") " pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c" Oct 30 09:29:49.722616 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.722600 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/028fea98-1060-44f3-887a-bf0a119b428b-istiod-ca-cert\") pod \"e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c\" (UID: \"028fea98-1060-44f3-887a-bf0a119b428b\") " pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c" Oct 30 09:29:49.722652 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.722639 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/028fea98-1060-44f3-887a-bf0a119b428b-credential-socket\") pod \"e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c\" (UID: \"028fea98-1060-44f3-887a-bf0a119b428b\") " pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c" Oct 30 09:29:49.823930 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.823892 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p57pv\" (UniqueName: \"kubernetes.io/projected/028fea98-1060-44f3-887a-bf0a119b428b-kube-api-access-p57pv\") pod \"e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c\" (UID: \"028fea98-1060-44f3-887a-bf0a119b428b\") " pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c" Oct 30 09:29:49.824124 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.824044 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/028fea98-1060-44f3-887a-bf0a119b428b-istio-podinfo\") pod \"e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c\" (UID: \"028fea98-1060-44f3-887a-bf0a119b428b\") " pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c" Oct 30 09:29:49.824124 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.824104 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/028fea98-1060-44f3-887a-bf0a119b428b-istio-data\") pod \"e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c\" (UID: \"028fea98-1060-44f3-887a-bf0a119b428b\") " pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c" Oct 30 09:29:49.824278 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.824139 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/028fea98-1060-44f3-887a-bf0a119b428b-workload-certs\") pod \"e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c\" (UID: \"028fea98-1060-44f3-887a-bf0a119b428b\") " pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c" Oct 30 09:29:49.824278 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.824184 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-292fn\" (UniqueName: \"kubernetes.io/projected/7a745f00-5456-4a63-86f9-19c03a808c05-kube-api-access-292fn\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh\" (UID: \"7a745f00-5456-4a63-86f9-19c03a808c05\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh" Oct 30 09:29:49.824278 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.824213 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/028fea98-1060-44f3-887a-bf0a119b428b-istio-envoy\") pod \"e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c\" (UID: \"028fea98-1060-44f3-887a-bf0a119b428b\") " pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c" Oct 30 09:29:49.824477 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.824287 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a745f00-5456-4a63-86f9-19c03a808c05-istiod-ca-cert\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh\" (UID: \"7a745f00-5456-4a63-86f9-19c03a808c05\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh" Oct 30 09:29:49.824539 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.824506 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a745f00-5456-4a63-86f9-19c03a808c05-istio-podinfo\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh\" (UID: \"7a745f00-5456-4a63-86f9-19c03a808c05\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh" Oct 30 09:29:49.824595 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.824537 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a745f00-5456-4a63-86f9-19c03a808c05-istio-token\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh\" (UID: \"7a745f00-5456-4a63-86f9-19c03a808c05\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh" Oct 30 09:29:49.824654 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.824610 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/028fea98-1060-44f3-887a-bf0a119b428b-istio-data\") pod \"e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c\" (UID: \"028fea98-1060-44f3-887a-bf0a119b428b\") " pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c" Oct 30 09:29:49.824654 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.824624 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/028fea98-1060-44f3-887a-bf0a119b428b-workload-certs\") pod \"e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c\" (UID: \"028fea98-1060-44f3-887a-bf0a119b428b\") " pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c" Oct 30 09:29:49.824772 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.824692 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a745f00-5456-4a63-86f9-19c03a808c05-credential-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh\" (UID: \"7a745f00-5456-4a63-86f9-19c03a808c05\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh" Oct 30 09:29:49.824772 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.824734 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a745f00-5456-4a63-86f9-19c03a808c05-istio-envoy\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh\" (UID: \"7a745f00-5456-4a63-86f9-19c03a808c05\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh" Oct 30 09:29:49.824874 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.824807 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/028fea98-1060-44f3-887a-bf0a119b428b-istio-token\") pod \"e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c\" (UID: \"028fea98-1060-44f3-887a-bf0a119b428b\") " pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c" Oct 30 09:29:49.824928 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.824870 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/028fea98-1060-44f3-887a-bf0a119b428b-istiod-ca-cert\") pod \"e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c\" (UID: \"028fea98-1060-44f3-887a-bf0a119b428b\") " pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c" Oct 30 09:29:49.824928 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.824906 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a745f00-5456-4a63-86f9-19c03a808c05-istio-data\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh\" (UID: \"7a745f00-5456-4a63-86f9-19c03a808c05\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh" Oct 30 09:29:49.825032 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.824948 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a745f00-5456-4a63-86f9-19c03a808c05-workload-certs\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh\" (UID: \"7a745f00-5456-4a63-86f9-19c03a808c05\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh" Oct 30 09:29:49.825032 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.824977 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/028fea98-1060-44f3-887a-bf0a119b428b-credential-socket\") pod \"e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c\" (UID: \"028fea98-1060-44f3-887a-bf0a119b428b\") " pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c" Oct 30 09:29:49.825149 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.825030 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a745f00-5456-4a63-86f9-19c03a808c05-workload-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh\" (UID: \"7a745f00-5456-4a63-86f9-19c03a808c05\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh" Oct 30 09:29:49.825149 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.825093 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/028fea98-1060-44f3-887a-bf0a119b428b-workload-socket\") pod \"e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c\" (UID: \"028fea98-1060-44f3-887a-bf0a119b428b\") " pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c" Oct 30 09:29:49.825509 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.825483 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/028fea98-1060-44f3-887a-bf0a119b428b-workload-socket\") pod \"e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c\" (UID: \"028fea98-1060-44f3-887a-bf0a119b428b\") " pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c" Oct 30 09:29:49.825728 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.825710 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/028fea98-1060-44f3-887a-bf0a119b428b-credential-socket\") pod \"e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c\" (UID: \"028fea98-1060-44f3-887a-bf0a119b428b\") " pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c" Oct 30 09:29:49.825910 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.825887 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/028fea98-1060-44f3-887a-bf0a119b428b-istiod-ca-cert\") pod \"e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c\" (UID: \"028fea98-1060-44f3-887a-bf0a119b428b\") " pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c" Oct 30 09:29:49.826854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.826826 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/028fea98-1060-44f3-887a-bf0a119b428b-istio-envoy\") pod \"e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c\" (UID: \"028fea98-1060-44f3-887a-bf0a119b428b\") " pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c" Oct 30 09:29:49.827001 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.826978 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/028fea98-1060-44f3-887a-bf0a119b428b-istio-podinfo\") pod \"e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c\" (UID: \"028fea98-1060-44f3-887a-bf0a119b428b\") " pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c" Oct 30 09:29:49.843823 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.843763 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p57pv\" (UniqueName: \"kubernetes.io/projected/028fea98-1060-44f3-887a-bf0a119b428b-kube-api-access-p57pv\") pod \"e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c\" (UID: \"028fea98-1060-44f3-887a-bf0a119b428b\") " pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c" Oct 30 09:29:49.844016 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.843998 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/028fea98-1060-44f3-887a-bf0a119b428b-istio-token\") pod \"e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c\" (UID: \"028fea98-1060-44f3-887a-bf0a119b428b\") " pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c" Oct 30 09:29:49.905402 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.905355 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c" Oct 30 09:29:49.916135 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.916095 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cab9c95-15d2-48f5-99a1-1d6df3fd2333" path="/var/lib/kubelet/pods/8cab9c95-15d2-48f5-99a1-1d6df3fd2333/volumes" Oct 30 09:29:49.916791 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.916751 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9597695b-a089-4c8f-8a65-2ec83df78302" path="/var/lib/kubelet/pods/9597695b-a089-4c8f-8a65-2ec83df78302/volumes" Oct 30 09:29:49.928229 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.928199 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a745f00-5456-4a63-86f9-19c03a808c05-istio-data\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh\" (UID: \"7a745f00-5456-4a63-86f9-19c03a808c05\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh" Oct 30 09:29:49.928367 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.928242 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a745f00-5456-4a63-86f9-19c03a808c05-workload-certs\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh\" (UID: \"7a745f00-5456-4a63-86f9-19c03a808c05\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh" Oct 30 09:29:49.928489 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.928457 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a745f00-5456-4a63-86f9-19c03a808c05-workload-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh\" (UID: \"7a745f00-5456-4a63-86f9-19c03a808c05\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh" Oct 30 09:29:49.928686 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.928654 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a745f00-5456-4a63-86f9-19c03a808c05-workload-certs\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh\" (UID: \"7a745f00-5456-4a63-86f9-19c03a808c05\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh" Oct 30 09:29:49.928686 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.928677 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-292fn\" (UniqueName: \"kubernetes.io/projected/7a745f00-5456-4a63-86f9-19c03a808c05-kube-api-access-292fn\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh\" (UID: \"7a745f00-5456-4a63-86f9-19c03a808c05\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh" Oct 30 09:29:49.928797 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.928681 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a745f00-5456-4a63-86f9-19c03a808c05-istio-data\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh\" (UID: \"7a745f00-5456-4a63-86f9-19c03a808c05\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh" Oct 30 09:29:49.928797 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.928713 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a745f00-5456-4a63-86f9-19c03a808c05-workload-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh\" (UID: \"7a745f00-5456-4a63-86f9-19c03a808c05\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh" Oct 30 09:29:49.928797 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.928747 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a745f00-5456-4a63-86f9-19c03a808c05-istiod-ca-cert\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh\" (UID: \"7a745f00-5456-4a63-86f9-19c03a808c05\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh" Oct 30 09:29:49.928797 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.928789 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a745f00-5456-4a63-86f9-19c03a808c05-istio-podinfo\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh\" (UID: \"7a745f00-5456-4a63-86f9-19c03a808c05\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh" Oct 30 09:29:49.928935 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.928810 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a745f00-5456-4a63-86f9-19c03a808c05-istio-token\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh\" (UID: \"7a745f00-5456-4a63-86f9-19c03a808c05\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh" Oct 30 09:29:49.928935 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.928874 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a745f00-5456-4a63-86f9-19c03a808c05-credential-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh\" (UID: \"7a745f00-5456-4a63-86f9-19c03a808c05\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh" Oct 30 09:29:49.928935 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.928904 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a745f00-5456-4a63-86f9-19c03a808c05-istio-envoy\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh\" (UID: \"7a745f00-5456-4a63-86f9-19c03a808c05\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh" Oct 30 09:29:49.929238 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.929214 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a745f00-5456-4a63-86f9-19c03a808c05-credential-socket\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh\" (UID: \"7a745f00-5456-4a63-86f9-19c03a808c05\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh" Oct 30 09:29:49.929369 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.929343 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a745f00-5456-4a63-86f9-19c03a808c05-istiod-ca-cert\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh\" (UID: \"7a745f00-5456-4a63-86f9-19c03a808c05\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh" Oct 30 09:29:49.931113 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.931093 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a745f00-5456-4a63-86f9-19c03a808c05-istio-envoy\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh\" (UID: \"7a745f00-5456-4a63-86f9-19c03a808c05\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh" Oct 30 09:29:49.931620 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.931600 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a745f00-5456-4a63-86f9-19c03a808c05-istio-podinfo\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh\" (UID: \"7a745f00-5456-4a63-86f9-19c03a808c05\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh" Oct 30 09:29:49.948898 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.948871 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a745f00-5456-4a63-86f9-19c03a808c05-istio-token\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh\" (UID: \"7a745f00-5456-4a63-86f9-19c03a808c05\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh" Oct 30 09:29:49.949583 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.949559 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-292fn\" (UniqueName: \"kubernetes.io/projected/7a745f00-5456-4a63-86f9-19c03a808c05-kube-api-access-292fn\") pod \"e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh\" (UID: \"7a745f00-5456-4a63-86f9-19c03a808c05\") " pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh" Oct 30 09:29:49.976266 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:49.976236 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh" Oct 30 09:29:50.233888 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:29:50.233859 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod028fea98_1060_44f3_887a_bf0a119b428b.slice/crio-c45862428c577580f2789dadbdc2c2df3857ccf9a14ca10cd2313735be8a4779 WatchSource:0}: Error finding container c45862428c577580f2789dadbdc2c2df3857ccf9a14ca10cd2313735be8a4779: Status 404 returned error can't find the container with id c45862428c577580f2789dadbdc2c2df3857ccf9a14ca10cd2313735be8a4779 Oct 30 09:29:50.234410 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:50.234345 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c"] Oct 30 09:29:50.236274 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:50.236242 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:29:50.236360 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:50.236308 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:29:50.236360 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:50.236335 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:29:50.342658 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:29:50.342629 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a745f00_5456_4a63_86f9_19c03a808c05.slice/crio-6f902f3daafe3b5ecf7414602636267e9daeb73a21a7da4a473987cad0317a80 WatchSource:0}: Error finding container 6f902f3daafe3b5ecf7414602636267e9daeb73a21a7da4a473987cad0317a80: Status 404 returned error can't find the container with id 6f902f3daafe3b5ecf7414602636267e9daeb73a21a7da4a473987cad0317a80 Oct 30 09:29:50.345358 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:50.345333 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh"] Oct 30 09:29:50.345694 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:50.345667 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:29:50.345744 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:50.345729 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:29:50.345781 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:50.345758 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:29:50.993201 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:50.993163 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh" event={"ID":"7a745f00-5456-4a63-86f9-19c03a808c05","Type":"ContainerStarted","Data":"c58ae56cc4d8ae95c0a4519182658ff9f2c15c0321ecb20fc7dcd0aab8ba6cad"} Oct 30 09:29:50.993723 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:50.993230 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh" event={"ID":"7a745f00-5456-4a63-86f9-19c03a808c05","Type":"ContainerStarted","Data":"6f902f3daafe3b5ecf7414602636267e9daeb73a21a7da4a473987cad0317a80"} Oct 30 09:29:50.994956 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:50.994925 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c" event={"ID":"028fea98-1060-44f3-887a-bf0a119b428b","Type":"ContainerStarted","Data":"9ac99bca87d8ea2f341ef3d64bbad45ce02d15c7aa1ed8209c238257a3e2f55f"} Oct 30 09:29:50.995069 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:50.994971 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c" event={"ID":"028fea98-1060-44f3-887a-bf0a119b428b","Type":"ContainerStarted","Data":"c45862428c577580f2789dadbdc2c2df3857ccf9a14ca10cd2313735be8a4779"} Oct 30 09:29:51.124320 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:51.124131 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c" podStartSLOduration=2.124117259 podStartE2EDuration="2.124117259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:29:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:29:51.12190367 +0000 UTC m=+2151.383464722" watchObservedRunningTime="2026-10-30 09:29:51.124117259 +0000 UTC m=+2151.385678321" Oct 30 09:29:51.124622 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:51.124584 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh" podStartSLOduration=2.124576511 podStartE2EDuration="2.124576511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:29:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:29:51.074688077 +0000 UTC m=+2151.336249066" watchObservedRunningTime="2026-10-30 09:29:51.124576511 +0000 UTC m=+2151.386137499" Oct 30 09:29:51.264470 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:51.264149 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5"] Oct 30 09:29:51.264470 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:51.264428 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5" podUID="4c0ac365-a552-4514-a497-748845006a2e" containerName="istio-proxy" containerID="cri-o://74e7d1511b8a823cd994d874dd42544ce8e579f0b80778677098c7ecaaac1e64" gracePeriod=30 Oct 30 09:29:52.127411 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:52.115511 4562 patch_prober.go:29] interesting pod/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Oct 30 09:29:52.127411 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:52.115585 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c" podUID="028fea98-1060-44f3-887a-bf0a119b428b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Oct 30 09:29:52.130452 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:52.129702 4562 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 30 09:29:52.130452 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:52.129776 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh" podUID="7a745f00-5456-4a63-86f9-19c03a808c05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 30 09:29:52.213432 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:52.210716 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c" Oct 30 09:29:52.213432 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:52.210787 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh" Oct 30 09:29:52.908422 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:52.907239 4562 patch_prober.go:29] interesting pod/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Oct 30 09:29:52.908422 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:52.907300 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c" podUID="028fea98-1060-44f3-887a-bf0a119b428b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Oct 30 09:29:53.001423 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:53.000913 4562 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 30 09:29:53.001423 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:53.000975 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh" podUID="7a745f00-5456-4a63-86f9-19c03a808c05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 30 09:29:53.908053 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:53.908017 4562 patch_prober.go:29] interesting pod/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Oct 30 09:29:53.908536 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:53.908068 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c" podUID="028fea98-1060-44f3-887a-bf0a119b428b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Oct 30 09:29:53.986895 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:53.978744 4562 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 30 09:29:53.986895 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:53.978833 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh" podUID="7a745f00-5456-4a63-86f9-19c03a808c05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 30 09:29:54.104421 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:54.103668 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj"] Oct 30 09:29:54.136662 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:54.136632 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj" Oct 30 09:29:54.159465 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:54.157590 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-eff9fd-openshift-default-dockercfg-6fspq\"" Oct 30 09:29:54.218403 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:54.213057 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64-istiod-ca-cert\") pod \"e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj\" (UID: \"f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64\") " pod="openshift-ingress/e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj" Oct 30 09:29:54.218403 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:54.213132 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64-istio-token\") pod \"e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj\" (UID: \"f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64\") " pod="openshift-ingress/e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj" Oct 30 09:29:54.218403 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:54.213171 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64-istio-envoy\") pod \"e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj\" (UID: \"f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64\") " pod="openshift-ingress/e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj" Oct 30 09:29:54.218403 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:54.213283 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64-istio-podinfo\") pod \"e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj\" (UID: \"f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64\") " pod="openshift-ingress/e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj" Oct 30 09:29:54.218403 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:54.213326 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64-credential-socket\") pod \"e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj\" (UID: \"f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64\") " pod="openshift-ingress/e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj" Oct 30 09:29:54.218403 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:54.213420 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj7wd\" (UniqueName: \"kubernetes.io/projected/f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64-kube-api-access-mj7wd\") pod \"e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj\" (UID: \"f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64\") " pod="openshift-ingress/e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj" Oct 30 09:29:54.218403 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:54.213452 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64-workload-socket\") pod \"e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj\" (UID: \"f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64\") " pod="openshift-ingress/e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj" Oct 30 09:29:54.218403 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:54.213497 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64-workload-certs\") pod \"e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj\" (UID: \"f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64\") " pod="openshift-ingress/e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj" Oct 30 09:29:54.218403 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:54.213745 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64-istio-data\") pod \"e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj\" (UID: \"f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64\") " pod="openshift-ingress/e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj" Oct 30 09:29:54.338497 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:54.336263 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj7wd\" (UniqueName: \"kubernetes.io/projected/f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64-kube-api-access-mj7wd\") pod \"e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj\" (UID: \"f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64\") " pod="openshift-ingress/e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj" Oct 30 09:29:54.338497 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:54.336329 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64-workload-socket\") pod \"e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj\" (UID: \"f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64\") " pod="openshift-ingress/e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj" Oct 30 09:29:54.338497 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:54.336375 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64-workload-certs\") pod \"e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj\" (UID: \"f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64\") " pod="openshift-ingress/e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj" Oct 30 09:29:54.338497 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:54.336457 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64-istio-data\") pod \"e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj\" (UID: \"f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64\") " pod="openshift-ingress/e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj" Oct 30 09:29:54.338497 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:54.336483 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64-istiod-ca-cert\") pod \"e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj\" (UID: \"f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64\") " pod="openshift-ingress/e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj" Oct 30 09:29:54.338497 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:54.336536 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64-istio-token\") pod \"e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj\" (UID: \"f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64\") " pod="openshift-ingress/e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj" Oct 30 09:29:54.338497 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:54.336575 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64-istio-envoy\") pod \"e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj\" (UID: \"f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64\") " pod="openshift-ingress/e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj" Oct 30 09:29:54.338497 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:54.336707 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64-istio-podinfo\") pod \"e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj\" (UID: \"f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64\") " pod="openshift-ingress/e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj" Oct 30 09:29:54.338497 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:54.336751 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64-credential-socket\") pod \"e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj\" (UID: \"f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64\") " pod="openshift-ingress/e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj" Oct 30 09:29:54.338497 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:54.337403 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64-credential-socket\") pod \"e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj\" (UID: \"f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64\") " pod="openshift-ingress/e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj" Oct 30 09:29:54.338497 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:54.338221 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64-workload-socket\") pod \"e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj\" (UID: \"f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64\") " pod="openshift-ingress/e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj" Oct 30 09:29:54.338497 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:54.338441 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64-workload-certs\") pod \"e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj\" (UID: \"f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64\") " pod="openshift-ingress/e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj" Oct 30 09:29:54.339297 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:54.338647 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64-istio-data\") pod \"e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj\" (UID: \"f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64\") " pod="openshift-ingress/e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj" Oct 30 09:29:54.339297 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:54.339197 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64-istiod-ca-cert\") pod \"e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj\" (UID: \"f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64\") " pod="openshift-ingress/e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj" Oct 30 09:29:54.401407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:54.354681 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64-istio-envoy\") pod \"e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj\" (UID: \"f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64\") " pod="openshift-ingress/e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj" Oct 30 09:29:54.471578 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:54.448020 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj"] Oct 30 09:29:54.471578 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:54.452459 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64-istio-podinfo\") pod \"e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj\" (UID: \"f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64\") " pod="openshift-ingress/e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj" Oct 30 09:29:54.554910 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:54.554861 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj7wd\" (UniqueName: \"kubernetes.io/projected/f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64-kube-api-access-mj7wd\") pod \"e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj\" (UID: \"f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64\") " pod="openshift-ingress/e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj" Oct 30 09:29:54.609980 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:54.609943 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64-istio-token\") pod \"e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj\" (UID: \"f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64\") " pod="openshift-ingress/e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj" Oct 30 09:29:54.649510 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:54.641116 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj" Oct 30 09:29:54.909604 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:54.909571 4562 patch_prober.go:29] interesting pod/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Oct 30 09:29:54.910085 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:54.909623 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c" podUID="028fea98-1060-44f3-887a-bf0a119b428b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Oct 30 09:29:55.035403 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:55.035296 4562 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 30 09:29:55.035403 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:55.035353 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh" podUID="7a745f00-5456-4a63-86f9-19c03a808c05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 30 09:29:55.109571 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:55.109535 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj"] Oct 30 09:29:55.110287 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:55.110247 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:29:55.110417 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:55.110331 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:29:55.110417 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:55.110375 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:29:55.404953 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:55.404865 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj" event={"ID":"f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64","Type":"ContainerStarted","Data":"399e769bcd0cff3a38573e1a9c7d015ea6a1098c2ab43b44cb47e566bed33985"} Oct 30 09:29:55.405174 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:55.405158 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj" event={"ID":"f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64","Type":"ContainerStarted","Data":"dcf85573a1f48e421e41cd740b1d15d4b1bb9d384899f3ff41f7d4fe34889b09"} Oct 30 09:29:55.473407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:55.468189 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj" podStartSLOduration=1.468167936 podStartE2EDuration="1.468167936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:29:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:29:55.453742157 +0000 UTC m=+2155.715303145" watchObservedRunningTime="2026-10-30 09:29:55.468167936 +0000 UTC m=+2155.729728926" Oct 30 09:29:55.653278 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:55.653234 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj" Oct 30 09:29:55.908180 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:55.908148 4562 patch_prober.go:29] interesting pod/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Oct 30 09:29:55.908333 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:55.908198 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c" podUID="028fea98-1060-44f3-887a-bf0a119b428b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Oct 30 09:29:55.979407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:55.978787 4562 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 30 09:29:55.979407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:55.978889 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh" podUID="7a745f00-5456-4a63-86f9-19c03a808c05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 30 09:29:56.413919 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:56.413883 4562 generic.go:356] "Generic (PLEG): container finished" podID="4c0ac365-a552-4514-a497-748845006a2e" containerID="74e7d1511b8a823cd994d874dd42544ce8e579f0b80778677098c7ecaaac1e64" exitCode=0 Oct 30 09:29:56.414145 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:56.413976 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5" event={"ID":"4c0ac365-a552-4514-a497-748845006a2e","Type":"ContainerDied","Data":"74e7d1511b8a823cd994d874dd42544ce8e579f0b80778677098c7ecaaac1e64"} Oct 30 09:29:56.712815 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:56.710275 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb"] Oct 30 09:29:56.712815 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:56.710531 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb" podUID="28431c22-78da-4145-9842-e7d92b360558" containerName="istio-proxy" containerID="cri-o://a245dbbb93cdaea99d02ce7029d9269ce0cdaddc9f2e9de5aeddce8277d4d83d" gracePeriod=30 Oct 30 09:29:56.713441 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:56.713120 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj" Oct 30 09:29:56.966934 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:56.963999 4562 patch_prober.go:29] interesting pod/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Oct 30 09:29:56.967295 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:56.967264 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c" podUID="028fea98-1060-44f3-887a-bf0a119b428b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Oct 30 09:29:56.967455 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:56.967208 4562 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 09:29:56.967578 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:56.967559 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5" podUID="4c0ac365-a552-4514-a497-748845006a2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 09:29:56.980806 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:56.980774 4562 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 30 09:29:56.999422 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:56.992819 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh" podUID="7a745f00-5456-4a63-86f9-19c03a808c05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 30 09:29:57.015038 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:57.011820 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-15e1cc-c988685bd-6z7kz"] Oct 30 09:29:57.019146 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:57.017133 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-15e1cc-c988685bd-6z7kz" Oct 30 09:29:57.024225 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:57.024107 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-15e1cc-c988685bd-6z7kz"] Oct 30 09:29:57.046211 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:57.042123 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-15e1cc\"" Oct 30 09:29:57.046211 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:57.042900 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-15e1cc-dockercfg-rhbrg\"" Oct 30 09:29:57.119163 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:57.119121 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-15e1cc-79d76cd75-hkf2r"] Oct 30 09:29:57.125011 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:57.124980 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-15e1cc-79d76cd75-hkf2r" Oct 30 09:29:57.147547 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:57.147498 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62afe007-634a-41ab-9a9b-b10b4578452e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-15e1cc-c988685bd-6z7kz\" (UID: \"62afe007-634a-41ab-9a9b-b10b4578452e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-15e1cc-c988685bd-6z7kz" Oct 30 09:29:57.147710 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:57.147562 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10e23f86-d880-4196-b329-175b8629d542-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-15e1cc-79d76cd75-hkf2r\" (UID: \"10e23f86-d880-4196-b329-175b8629d542\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-15e1cc-79d76cd75-hkf2r" Oct 30 09:29:57.147710 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:57.147610 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq5br\" (UniqueName: \"kubernetes.io/projected/62afe007-634a-41ab-9a9b-b10b4578452e-kube-api-access-lq5br\") pod \"payload-processing-e2e-ipp-b-15e1cc-c988685bd-6z7kz\" (UID: \"62afe007-634a-41ab-9a9b-b10b4578452e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-15e1cc-c988685bd-6z7kz" Oct 30 09:29:57.147840 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:57.147730 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szxjr\" (UniqueName: \"kubernetes.io/projected/10e23f86-d880-4196-b329-175b8629d542-kube-api-access-szxjr\") pod \"payload-pre-processing-e2e-ipp-b-15e1cc-79d76cd75-hkf2r\" (UID: \"10e23f86-d880-4196-b329-175b8629d542\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-15e1cc-79d76cd75-hkf2r" Oct 30 09:29:57.149294 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:57.149262 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-15e1cc-79d76cd75-hkf2r"] Oct 30 09:29:57.234124 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:57.234051 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5" Oct 30 09:29:57.248514 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:57.248479 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c0ac365-a552-4514-a497-748845006a2e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c0ac365-a552-4514-a497-748845006a2e-credential-socket\") pod \"4c0ac365-a552-4514-a497-748845006a2e\" (UID: \"4c0ac365-a552-4514-a497-748845006a2e\") " Oct 30 09:29:57.248662 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:57.248520 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c0ac365-a552-4514-a497-748845006a2e-istio-token\" (UniqueName: \"kubernetes.io/projected/4c0ac365-a552-4514-a497-748845006a2e-istio-token\") pod \"4c0ac365-a552-4514-a497-748845006a2e\" (UID: \"4c0ac365-a552-4514-a497-748845006a2e\") " Oct 30 09:29:57.248662 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:57.248558 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4c0ac365-a552-4514-a497-748845006a2e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c0ac365-a552-4514-a497-748845006a2e-istio-podinfo\") pod \"4c0ac365-a552-4514-a497-748845006a2e\" (UID: \"4c0ac365-a552-4514-a497-748845006a2e\") " Oct 30 09:29:57.248662 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:57.248632 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c0ac365-a552-4514-a497-748845006a2e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c0ac365-a552-4514-a497-748845006a2e-istio-data\") pod \"4c0ac365-a552-4514-a497-748845006a2e\" (UID: \"4c0ac365-a552-4514-a497-748845006a2e\") " Oct 30 09:29:57.248828 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:57.248693 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c0ac365-a552-4514-a497-748845006a2e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c0ac365-a552-4514-a497-748845006a2e-workload-certs\") pod \"4c0ac365-a552-4514-a497-748845006a2e\" (UID: \"4c0ac365-a552-4514-a497-748845006a2e\") " Oct 30 09:29:57.248828 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:57.248769 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c0ac365-a552-4514-a497-748845006a2e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c0ac365-a552-4514-a497-748845006a2e-istio-envoy\") pod \"4c0ac365-a552-4514-a497-748845006a2e\" (UID: \"4c0ac365-a552-4514-a497-748845006a2e\") " Oct 30 09:29:57.248828 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:57.248801 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c0ac365-a552-4514-a497-748845006a2e-credential-socket" pod "4c0ac365-a552-4514-a497-748845006a2e" (UID: "4c0ac365-a552-4514-a497-748845006a2e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:29:57.248989 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:57.248959 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c0ac365-a552-4514-a497-748845006a2e-istio-data" pod "4c0ac365-a552-4514-a497-748845006a2e" (UID: "4c0ac365-a552-4514-a497-748845006a2e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:29:57.248989 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:57.248978 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c0ac365-a552-4514-a497-748845006a2e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c0ac365-a552-4514-a497-748845006a2e-istiod-ca-cert\") pod \"4c0ac365-a552-4514-a497-748845006a2e\" (UID: \"4c0ac365-a552-4514-a497-748845006a2e\") " Oct 30 09:29:57.249098 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:57.249010 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c0ac365-a552-4514-a497-748845006a2e-kube-api-access-rkfzz\" (UniqueName: \"kubernetes.io/projected/4c0ac365-a552-4514-a497-748845006a2e-kube-api-access-rkfzz\") pod \"4c0ac365-a552-4514-a497-748845006a2e\" (UID: \"4c0ac365-a552-4514-a497-748845006a2e\") " Oct 30 09:29:57.249098 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:57.249042 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c0ac365-a552-4514-a497-748845006a2e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c0ac365-a552-4514-a497-748845006a2e-workload-socket\") pod \"4c0ac365-a552-4514-a497-748845006a2e\" (UID: \"4c0ac365-a552-4514-a497-748845006a2e\") " Oct 30 09:29:57.249098 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:57.249066 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c0ac365-a552-4514-a497-748845006a2e-workload-certs" pod "4c0ac365-a552-4514-a497-748845006a2e" (UID: "4c0ac365-a552-4514-a497-748845006a2e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:29:57.249362 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:57.249339 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c0ac365-a552-4514-a497-748845006a2e-istiod-ca-cert" pod "4c0ac365-a552-4514-a497-748845006a2e" (UID: "4c0ac365-a552-4514-a497-748845006a2e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:29:57.249462 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:57.249367 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62afe007-634a-41ab-9a9b-b10b4578452e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-15e1cc-c988685bd-6z7kz\" (UID: \"62afe007-634a-41ab-9a9b-b10b4578452e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-15e1cc-c988685bd-6z7kz" Oct 30 09:29:57.249462 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:57.249434 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10e23f86-d880-4196-b329-175b8629d542-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-15e1cc-79d76cd75-hkf2r\" (UID: \"10e23f86-d880-4196-b329-175b8629d542\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-15e1cc-79d76cd75-hkf2r" Oct 30 09:29:57.249562 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:57.249470 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c0ac365-a552-4514-a497-748845006a2e-workload-socket" pod "4c0ac365-a552-4514-a497-748845006a2e" (UID: "4c0ac365-a552-4514-a497-748845006a2e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:29:57.249562 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:57.249528 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq5br\" (UniqueName: \"kubernetes.io/projected/62afe007-634a-41ab-9a9b-b10b4578452e-kube-api-access-lq5br\") pod \"payload-processing-e2e-ipp-b-15e1cc-c988685bd-6z7kz\" (UID: \"62afe007-634a-41ab-9a9b-b10b4578452e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-15e1cc-c988685bd-6z7kz" Oct 30 09:29:57.249686 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:57.249666 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szxjr\" (UniqueName: \"kubernetes.io/projected/10e23f86-d880-4196-b329-175b8629d542-kube-api-access-szxjr\") pod \"payload-pre-processing-e2e-ipp-b-15e1cc-79d76cd75-hkf2r\" (UID: \"10e23f86-d880-4196-b329-175b8629d542\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-15e1cc-79d76cd75-hkf2r" Oct 30 09:29:57.249759 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:57.249742 4562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c0ac365-a552-4514-a497-748845006a2e-istio-data\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:29:57.249819 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:57.249762 4562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c0ac365-a552-4514-a497-748845006a2e-workload-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:29:57.249819 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:57.249776 4562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c0ac365-a552-4514-a497-748845006a2e-istiod-ca-cert\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:29:57.249819 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:57.249789 4562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c0ac365-a552-4514-a497-748845006a2e-workload-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:29:57.249819 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:57.249802 4562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c0ac365-a552-4514-a497-748845006a2e-credential-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:29:57.250357 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:57.250327 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62afe007-634a-41ab-9a9b-b10b4578452e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-15e1cc-c988685bd-6z7kz\" (UID: \"62afe007-634a-41ab-9a9b-b10b4578452e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-15e1cc-c988685bd-6z7kz" Oct 30 09:29:57.250661 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:57.250630 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10e23f86-d880-4196-b329-175b8629d542-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-15e1cc-79d76cd75-hkf2r\" (UID: \"10e23f86-d880-4196-b329-175b8629d542\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-15e1cc-79d76cd75-hkf2r" Oct 30 09:29:57.252480 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:57.252159 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c0ac365-a552-4514-a497-748845006a2e-kube-api-access-rkfzz" pod "4c0ac365-a552-4514-a497-748845006a2e" (UID: "4c0ac365-a552-4514-a497-748845006a2e"). InnerVolumeSpecName "kube-api-access-rkfzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:29:57.252598 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:57.252549 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c0ac365-a552-4514-a497-748845006a2e-istio-token" pod "4c0ac365-a552-4514-a497-748845006a2e" (UID: "4c0ac365-a552-4514-a497-748845006a2e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:29:57.252875 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:57.252849 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c0ac365-a552-4514-a497-748845006a2e-istio-envoy" pod "4c0ac365-a552-4514-a497-748845006a2e" (UID: "4c0ac365-a552-4514-a497-748845006a2e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:29:57.253170 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:57.253150 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c0ac365-a552-4514-a497-748845006a2e-istio-podinfo" pod "4c0ac365-a552-4514-a497-748845006a2e" (UID: "4c0ac365-a552-4514-a497-748845006a2e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:29:57.270084 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:57.270063 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq5br\" (UniqueName: \"kubernetes.io/projected/62afe007-634a-41ab-9a9b-b10b4578452e-kube-api-access-lq5br\") pod \"payload-processing-e2e-ipp-b-15e1cc-c988685bd-6z7kz\" (UID: \"62afe007-634a-41ab-9a9b-b10b4578452e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-15e1cc-c988685bd-6z7kz" Oct 30 09:29:57.270217 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:57.270193 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szxjr\" (UniqueName: \"kubernetes.io/projected/10e23f86-d880-4196-b329-175b8629d542-kube-api-access-szxjr\") pod \"payload-pre-processing-e2e-ipp-b-15e1cc-79d76cd75-hkf2r\" (UID: \"10e23f86-d880-4196-b329-175b8629d542\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-15e1cc-79d76cd75-hkf2r" Oct 30 09:29:57.337894 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:57.337836 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-15e1cc-c988685bd-6z7kz" Oct 30 09:29:57.351716 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:57.351688 4562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c0ac365-a552-4514-a497-748845006a2e-istio-envoy\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:29:57.351841 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:57.351716 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkfzz\" (UniqueName: \"kubernetes.io/projected/4c0ac365-a552-4514-a497-748845006a2e-kube-api-access-rkfzz\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:29:57.351841 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:57.351732 4562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c0ac365-a552-4514-a497-748845006a2e-istio-token\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:29:57.351841 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:57.351746 4562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c0ac365-a552-4514-a497-748845006a2e-istio-podinfo\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:29:57.439218 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:57.438583 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5" event={"ID":"4c0ac365-a552-4514-a497-748845006a2e","Type":"ContainerDied","Data":"34bf5cf7a25fd9b826b51372f079c5d5a597d2b16096d70e88baa62e90220045"} Oct 30 09:29:57.439218 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:57.438636 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5" Oct 30 09:29:57.439218 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:57.438661 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj" Oct 30 09:29:57.439218 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:57.438697 4562 scope.go:122] "RemoveContainer" containerID="74e7d1511b8a823cd994d874dd42544ce8e579f0b80778677098c7ecaaac1e64" Oct 30 09:29:57.440629 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:57.440575 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj" Oct 30 09:29:57.465478 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:57.465436 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-15e1cc-79d76cd75-hkf2r" Oct 30 09:29:57.541074 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:57.541018 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5"] Oct 30 09:29:57.546271 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:57.546234 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-dmmw5"] Oct 30 09:29:57.691009 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:57.690816 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-15e1cc-c988685bd-6z7kz"] Oct 30 09:29:57.691268 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:29:57.691224 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62afe007_634a_41ab_9a9b_b10b4578452e.slice/crio-a3d6a0c1d8c5974cc98f8501c0f31bf50b701392baaf1574fce58080be43c4cc WatchSource:0}: Error finding container a3d6a0c1d8c5974cc98f8501c0f31bf50b701392baaf1574fce58080be43c4cc: Status 404 returned error can't find the container with id a3d6a0c1d8c5974cc98f8501c0f31bf50b701392baaf1574fce58080be43c4cc Oct 30 09:29:57.854395 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:57.854354 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-15e1cc-79d76cd75-hkf2r"] Oct 30 09:29:57.876685 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:57.876659 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c0ac365-a552-4514-a497-748845006a2e" path="/var/lib/kubelet/pods/4c0ac365-a552-4514-a497-748845006a2e/volumes" Oct 30 09:29:57.915828 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:57.915801 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c" Oct 30 09:29:57.916486 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:57.916464 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c" Oct 30 09:29:57.917291 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:57.917271 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c" Oct 30 09:29:57.977212 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:57.977165 4562 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 30 09:29:57.977398 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:57.977214 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh" podUID="7a745f00-5456-4a63-86f9-19c03a808c05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 30 09:29:58.461085 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:58.461044 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-15e1cc-c988685bd-6z7kz" event={"ID":"62afe007-634a-41ab-9a9b-b10b4578452e","Type":"ContainerStarted","Data":"b26cfa46daaf6d09b79191fa68295550341d62e66ac7de354de0e208cf9869e7"} Oct 30 09:29:58.461605 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:58.461111 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-15e1cc-c988685bd-6z7kz" event={"ID":"62afe007-634a-41ab-9a9b-b10b4578452e","Type":"ContainerStarted","Data":"a3d6a0c1d8c5974cc98f8501c0f31bf50b701392baaf1574fce58080be43c4cc"} Oct 30 09:29:58.462008 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:58.461956 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-15e1cc-c988685bd-6z7kz" Oct 30 09:29:58.470405 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:58.470359 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-15e1cc-79d76cd75-hkf2r" event={"ID":"10e23f86-d880-4196-b329-175b8629d542","Type":"ContainerStarted","Data":"da3911b4a92a556c23130fbfcfc5cb525dfedd5abf19c62d6546765b80236e5a"} Oct 30 09:29:58.470512 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:58.470420 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-15e1cc-79d76cd75-hkf2r" event={"ID":"10e23f86-d880-4196-b329-175b8629d542","Type":"ContainerStarted","Data":"45c04102f7b0920ce77652acff2780b18191052dfc1e3c1f64a9225c837a5fd2"} Oct 30 09:29:58.470512 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:58.470448 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-15e1cc-79d76cd75-hkf2r" Oct 30 09:29:58.643408 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:58.636138 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-15e1cc-c988685bd-6z7kz" podStartSLOduration=2.6361170879999998 podStartE2EDuration="2.636117088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:29:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:29:58.534798216 +0000 UTC m=+2158.796359206" watchObservedRunningTime="2026-10-30 09:29:58.636117088 +0000 UTC m=+2158.897678076" Oct 30 09:29:58.670783 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:58.670710 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-15e1cc-79d76cd75-hkf2r" podStartSLOduration=1.67068953 podStartE2EDuration="1.67068953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:29:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:29:58.594946045 +0000 UTC m=+2158.856507034" watchObservedRunningTime="2026-10-30 09:29:58.67068953 +0000 UTC m=+2158.932250519" Oct 30 09:29:58.830209 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:29:58.830120 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:29:59.381407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:59.380758 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh" Oct 30 09:29:59.385403 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:59.382308 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh" Oct 30 09:29:59.394649 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:29:59.394621 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh" Oct 30 09:30:00.857267 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:00.857221 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55" podUID="88be34c3-a9e9-4489-a449-206a99ef1663" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:30:00.989410 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:30:00.961498 4562 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.023s" Oct 30 09:30:00.989410 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:00.961539 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d95d35-29889210-rf62t"] Oct 30 09:30:01.017408 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:00.989856 4562 cpu_manager.go:414] "containerMap: removing container" podUID="4c0ac365-a552-4514-a497-748845006a2e" containerName="istio-proxy" Oct 30 09:30:01.017408 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:00.989885 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c0ac365-a552-4514-a497-748845006a2e" containerName="istio-proxy" Oct 30 09:30:01.017408 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:00.990141 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c0ac365-a552-4514-a497-748845006a2e" containerName="istio-proxy" Oct 30 09:30:01.310440 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:01.295280 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889210-7c9t8"] Oct 30 09:30:01.314079 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:01.314017 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb" podUID="a540c86f-c856-42a8-9f73-b5b9b690f42b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Oct 30 09:30:01.349848 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:01.349812 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d95d35-29889210-rf62t" Oct 30 09:30:01.538816 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:01.538590 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d95d35-29889210-rf62t"] Oct 30 09:30:01.538816 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:01.538655 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889210-7c9t8"] Oct 30 09:30:01.538816 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:01.538730 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889210-7c9t8" Oct 30 09:30:01.539145 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:01.539023 4562 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.033390393s: [/var/lib/containers/storage/overlay/4fa991d6ce3f2292a259e975b535550dec0db4ed665c5319ee1cccbf60087d94/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6q7n_1f51db5a-5720-4dea-ad02-f336b35d43b1/kube-rbac-proxy-ovn-metrics/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 09:30:01.608472 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:01.559061 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkcpg\" (UniqueName: \"kubernetes.io/projected/57b6fb70-7cbb-46ee-ac05-be848af5349f-kube-api-access-zkcpg\") pod \"maas-api-key-cleanup-e2e-praxis-b-d95d35-29889210-rf62t\" (UID: \"57b6fb70-7cbb-46ee-ac05-be848af5349f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d95d35-29889210-rf62t" Oct 30 09:30:01.622893 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:01.614642 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e525d80e-1056-4e3b-9ded-485e2f174a78-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889210-7c9t8\" (UID: \"e525d80e-1056-4e3b-9ded-485e2f174a78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889210-7c9t8" Oct 30 09:30:01.623187 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:01.623157 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n7fc\" (UniqueName: \"kubernetes.io/projected/e525d80e-1056-4e3b-9ded-485e2f174a78-kube-api-access-4n7fc\") pod \"maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889210-7c9t8\" (UID: \"e525d80e-1056-4e3b-9ded-485e2f174a78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889210-7c9t8" Oct 30 09:30:01.631526 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:01.623539 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57b6fb70-7cbb-46ee-ac05-be848af5349f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-d95d35-29889210-rf62t\" (UID: \"57b6fb70-7cbb-46ee-ac05-be848af5349f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d95d35-29889210-rf62t" Oct 30 09:30:01.631861 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:01.609091 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-eff9fd-847fb7448c-5n9rh"] Oct 30 09:30:01.636008 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:01.571198 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:30:01.642597 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:01.642567 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-eff9fd-847fb7448c-5n9rh"] Oct 30 09:30:01.642875 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:01.642857 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eff9fd-847fb7448c-5n9rh" Oct 30 09:30:01.664875 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:01.664844 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-eff9fd-dockercfg-pzr76\"" Oct 30 09:30:01.691407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:01.665688 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-eff9fd\"" Oct 30 09:30:01.749016 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:01.748978 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkcpg\" (UniqueName: \"kubernetes.io/projected/57b6fb70-7cbb-46ee-ac05-be848af5349f-kube-api-access-zkcpg\") pod \"maas-api-key-cleanup-e2e-praxis-b-d95d35-29889210-rf62t\" (UID: \"57b6fb70-7cbb-46ee-ac05-be848af5349f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d95d35-29889210-rf62t" Oct 30 09:30:01.749780 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:01.749751 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22x2s\" (UniqueName: \"kubernetes.io/projected/c8b60df6-3c80-4c90-9960-0af22206d277-kube-api-access-22x2s\") pod \"payload-pre-processing-e2e-praxis-a-eff9fd-847fb7448c-5n9rh\" (UID: \"c8b60df6-3c80-4c90-9960-0af22206d277\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eff9fd-847fb7448c-5n9rh" Oct 30 09:30:01.750130 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:01.750107 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e525d80e-1056-4e3b-9ded-485e2f174a78-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889210-7c9t8\" (UID: \"e525d80e-1056-4e3b-9ded-485e2f174a78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889210-7c9t8" Oct 30 09:30:01.760918 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:01.760887 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n7fc\" (UniqueName: \"kubernetes.io/projected/e525d80e-1056-4e3b-9ded-485e2f174a78-kube-api-access-4n7fc\") pod \"maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889210-7c9t8\" (UID: \"e525d80e-1056-4e3b-9ded-485e2f174a78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889210-7c9t8" Oct 30 09:30:01.764616 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:01.750998 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e525d80e-1056-4e3b-9ded-485e2f174a78-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889210-7c9t8\" (UID: \"e525d80e-1056-4e3b-9ded-485e2f174a78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889210-7c9t8" Oct 30 09:30:01.776411 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:01.776130 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57b6fb70-7cbb-46ee-ac05-be848af5349f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-d95d35-29889210-rf62t\" (UID: \"57b6fb70-7cbb-46ee-ac05-be848af5349f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d95d35-29889210-rf62t" Oct 30 09:30:01.776411 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:01.776315 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8b60df6-3c80-4c90-9960-0af22206d277-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-eff9fd-847fb7448c-5n9rh\" (UID: \"c8b60df6-3c80-4c90-9960-0af22206d277\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eff9fd-847fb7448c-5n9rh" Oct 30 09:30:01.790420 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:01.778376 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57b6fb70-7cbb-46ee-ac05-be848af5349f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-d95d35-29889210-rf62t\" (UID: \"57b6fb70-7cbb-46ee-ac05-be848af5349f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d95d35-29889210-rf62t" Oct 30 09:30:01.874853 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:01.846331 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-eff9fd-6b784dd94d-lrzsf"] Oct 30 09:30:01.874853 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:01.863217 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-eff9fd-6b784dd94d-lrzsf"] Oct 30 09:30:01.874853 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:01.863349 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-eff9fd-6b784dd94d-lrzsf" Oct 30 09:30:02.019592 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:02.015913 4562 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 09:30:02.019592 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:02.015977 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh" podUID="9491d089-4927-4270-82f2-fecd0f108311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 09:30:02.029648 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:02.022807 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8b60df6-3c80-4c90-9960-0af22206d277-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-eff9fd-847fb7448c-5n9rh\" (UID: \"c8b60df6-3c80-4c90-9960-0af22206d277\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eff9fd-847fb7448c-5n9rh" Oct 30 09:30:02.029648 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:02.022953 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22x2s\" (UniqueName: \"kubernetes.io/projected/c8b60df6-3c80-4c90-9960-0af22206d277-kube-api-access-22x2s\") pod \"payload-pre-processing-e2e-praxis-a-eff9fd-847fb7448c-5n9rh\" (UID: \"c8b60df6-3c80-4c90-9960-0af22206d277\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eff9fd-847fb7448c-5n9rh" Oct 30 09:30:02.029648 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:02.027553 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8b60df6-3c80-4c90-9960-0af22206d277-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-eff9fd-847fb7448c-5n9rh\" (UID: \"c8b60df6-3c80-4c90-9960-0af22206d277\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eff9fd-847fb7448c-5n9rh" Oct 30 09:30:02.107110 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:02.107076 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkcpg\" (UniqueName: \"kubernetes.io/projected/57b6fb70-7cbb-46ee-ac05-be848af5349f-kube-api-access-zkcpg\") pod \"maas-api-key-cleanup-e2e-praxis-b-d95d35-29889210-rf62t\" (UID: \"57b6fb70-7cbb-46ee-ac05-be848af5349f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d95d35-29889210-rf62t" Oct 30 09:30:02.136733 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:02.136658 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n7fc\" (UniqueName: \"kubernetes.io/projected/e525d80e-1056-4e3b-9ded-485e2f174a78-kube-api-access-4n7fc\") pod \"maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889210-7c9t8\" (UID: \"e525d80e-1056-4e3b-9ded-485e2f174a78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889210-7c9t8" Oct 30 09:30:02.254575 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:02.254543 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22x2s\" (UniqueName: \"kubernetes.io/projected/c8b60df6-3c80-4c90-9960-0af22206d277-kube-api-access-22x2s\") pod \"payload-pre-processing-e2e-praxis-a-eff9fd-847fb7448c-5n9rh\" (UID: \"c8b60df6-3c80-4c90-9960-0af22206d277\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eff9fd-847fb7448c-5n9rh" Oct 30 09:30:02.278459 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:02.256069 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmz7s\" (UniqueName: \"kubernetes.io/projected/2198e031-552b-46f4-adc2-bfbf3bb08dfd-kube-api-access-rmz7s\") pod \"payload-processing-e2e-praxis-a-eff9fd-6b784dd94d-lrzsf\" (UID: \"2198e031-552b-46f4-adc2-bfbf3bb08dfd\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-eff9fd-6b784dd94d-lrzsf" Oct 30 09:30:02.278721 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:02.278688 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2198e031-552b-46f4-adc2-bfbf3bb08dfd-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-eff9fd-6b784dd94d-lrzsf\" (UID: \"2198e031-552b-46f4-adc2-bfbf3bb08dfd\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-eff9fd-6b784dd94d-lrzsf" Oct 30 09:30:02.297735 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:02.297520 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d95d35-29889210-rf62t" Oct 30 09:30:02.312777 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:02.312750 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eff9fd-847fb7448c-5n9rh" Oct 30 09:30:02.343760 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:02.343729 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889210-7c9t8" Oct 30 09:30:02.388021 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:02.387951 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmz7s\" (UniqueName: \"kubernetes.io/projected/2198e031-552b-46f4-adc2-bfbf3bb08dfd-kube-api-access-rmz7s\") pod \"payload-processing-e2e-praxis-a-eff9fd-6b784dd94d-lrzsf\" (UID: \"2198e031-552b-46f4-adc2-bfbf3bb08dfd\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-eff9fd-6b784dd94d-lrzsf" Oct 30 09:30:02.388239 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:02.388219 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2198e031-552b-46f4-adc2-bfbf3bb08dfd-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-eff9fd-6b784dd94d-lrzsf\" (UID: \"2198e031-552b-46f4-adc2-bfbf3bb08dfd\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-eff9fd-6b784dd94d-lrzsf" Oct 30 09:30:02.389445 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:02.389422 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2198e031-552b-46f4-adc2-bfbf3bb08dfd-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-eff9fd-6b784dd94d-lrzsf\" (UID: \"2198e031-552b-46f4-adc2-bfbf3bb08dfd\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-eff9fd-6b784dd94d-lrzsf" Oct 30 09:30:02.423626 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:02.416428 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmz7s\" (UniqueName: \"kubernetes.io/projected/2198e031-552b-46f4-adc2-bfbf3bb08dfd-kube-api-access-rmz7s\") pod \"payload-processing-e2e-praxis-a-eff9fd-6b784dd94d-lrzsf\" (UID: \"2198e031-552b-46f4-adc2-bfbf3bb08dfd\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-eff9fd-6b784dd94d-lrzsf" Oct 30 09:30:02.609733 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:02.587070 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-eff9fd-6b784dd94d-lrzsf" Oct 30 09:30:02.747677 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:02.747652 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb" Oct 30 09:30:02.840209 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:02.840173 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28431c22-78da-4145-9842-e7d92b360558-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28431c22-78da-4145-9842-e7d92b360558-istiod-ca-cert\") pod \"28431c22-78da-4145-9842-e7d92b360558\" (UID: \"28431c22-78da-4145-9842-e7d92b360558\") " Oct 30 09:30:02.840430 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:02.840255 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28431c22-78da-4145-9842-e7d92b360558-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28431c22-78da-4145-9842-e7d92b360558-credential-socket\") pod \"28431c22-78da-4145-9842-e7d92b360558\" (UID: \"28431c22-78da-4145-9842-e7d92b360558\") " Oct 30 09:30:02.840430 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:02.840318 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/28431c22-78da-4145-9842-e7d92b360558-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28431c22-78da-4145-9842-e7d92b360558-istio-podinfo\") pod \"28431c22-78da-4145-9842-e7d92b360558\" (UID: \"28431c22-78da-4145-9842-e7d92b360558\") " Oct 30 09:30:02.840430 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:02.840351 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28431c22-78da-4145-9842-e7d92b360558-kube-api-access-h77tv\" (UniqueName: \"kubernetes.io/projected/28431c22-78da-4145-9842-e7d92b360558-kube-api-access-h77tv\") pod \"28431c22-78da-4145-9842-e7d92b360558\" (UID: \"28431c22-78da-4145-9842-e7d92b360558\") " Oct 30 09:30:02.840430 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:02.840419 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28431c22-78da-4145-9842-e7d92b360558-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28431c22-78da-4145-9842-e7d92b360558-workload-socket\") pod \"28431c22-78da-4145-9842-e7d92b360558\" (UID: \"28431c22-78da-4145-9842-e7d92b360558\") " Oct 30 09:30:02.840670 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:02.840454 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28431c22-78da-4145-9842-e7d92b360558-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28431c22-78da-4145-9842-e7d92b360558-istio-data\") pod \"28431c22-78da-4145-9842-e7d92b360558\" (UID: \"28431c22-78da-4145-9842-e7d92b360558\") " Oct 30 09:30:02.840670 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:02.840568 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28431c22-78da-4145-9842-e7d92b360558-istiod-ca-cert" pod "28431c22-78da-4145-9842-e7d92b360558" (UID: "28431c22-78da-4145-9842-e7d92b360558"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:30:02.840670 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:02.840597 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28431c22-78da-4145-9842-e7d92b360558-credential-socket" pod "28431c22-78da-4145-9842-e7d92b360558" (UID: "28431c22-78da-4145-9842-e7d92b360558"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:30:02.840826 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:02.840685 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28431c22-78da-4145-9842-e7d92b360558-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28431c22-78da-4145-9842-e7d92b360558-workload-certs\") pod \"28431c22-78da-4145-9842-e7d92b360558\" (UID: \"28431c22-78da-4145-9842-e7d92b360558\") " Oct 30 09:30:02.840826 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:02.840719 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28431c22-78da-4145-9842-e7d92b360558-istio-token\" (UniqueName: \"kubernetes.io/projected/28431c22-78da-4145-9842-e7d92b360558-istio-token\") pod \"28431c22-78da-4145-9842-e7d92b360558\" (UID: \"28431c22-78da-4145-9842-e7d92b360558\") " Oct 30 09:30:02.840826 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:02.840804 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28431c22-78da-4145-9842-e7d92b360558-istio-data" pod "28431c22-78da-4145-9842-e7d92b360558" (UID: "28431c22-78da-4145-9842-e7d92b360558"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:30:02.840961 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:02.840878 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28431c22-78da-4145-9842-e7d92b360558-workload-socket" pod "28431c22-78da-4145-9842-e7d92b360558" (UID: "28431c22-78da-4145-9842-e7d92b360558"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:30:02.840961 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:02.840880 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28431c22-78da-4145-9842-e7d92b360558-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28431c22-78da-4145-9842-e7d92b360558-istio-envoy\") pod \"28431c22-78da-4145-9842-e7d92b360558\" (UID: \"28431c22-78da-4145-9842-e7d92b360558\") " Oct 30 09:30:02.841025 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:02.840988 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28431c22-78da-4145-9842-e7d92b360558-workload-certs" pod "28431c22-78da-4145-9842-e7d92b360558" (UID: "28431c22-78da-4145-9842-e7d92b360558"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:30:02.841583 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:02.841559 4562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28431c22-78da-4145-9842-e7d92b360558-istiod-ca-cert\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:30:02.841752 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:02.841736 4562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28431c22-78da-4145-9842-e7d92b360558-credential-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:30:02.841867 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:02.841851 4562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28431c22-78da-4145-9842-e7d92b360558-workload-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:30:02.841962 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:02.841950 4562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28431c22-78da-4145-9842-e7d92b360558-istio-data\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:30:02.842052 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:02.842042 4562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28431c22-78da-4145-9842-e7d92b360558-workload-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:30:02.843195 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:02.843169 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28431c22-78da-4145-9842-e7d92b360558-istio-token" pod "28431c22-78da-4145-9842-e7d92b360558" (UID: "28431c22-78da-4145-9842-e7d92b360558"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:30:02.843418 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:02.843358 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28431c22-78da-4145-9842-e7d92b360558-istio-envoy" pod "28431c22-78da-4145-9842-e7d92b360558" (UID: "28431c22-78da-4145-9842-e7d92b360558"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:30:02.843520 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:02.843477 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/28431c22-78da-4145-9842-e7d92b360558-istio-podinfo" pod "28431c22-78da-4145-9842-e7d92b360558" (UID: "28431c22-78da-4145-9842-e7d92b360558"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:30:02.843602 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:02.843576 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28431c22-78da-4145-9842-e7d92b360558-kube-api-access-h77tv" pod "28431c22-78da-4145-9842-e7d92b360558" (UID: "28431c22-78da-4145-9842-e7d92b360558"). InnerVolumeSpecName "kube-api-access-h77tv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:30:02.868732 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:30:02.868702 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode525d80e_1056_4e3b_9ded_485e2f174a78.slice/crio-b3533d385c29aec4264184fd638f2073c1d94dd2deea400871d242eda1207c9c WatchSource:0}: Error finding container b3533d385c29aec4264184fd638f2073c1d94dd2deea400871d242eda1207c9c: Status 404 returned error can't find the container with id b3533d385c29aec4264184fd638f2073c1d94dd2deea400871d242eda1207c9c Oct 30 09:30:02.874053 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:02.874031 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889210-7c9t8"] Oct 30 09:30:02.913930 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:30:02.913897 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57b6fb70_7cbb_46ee_ac05_be848af5349f.slice/crio-358a079d924109d75fcc6e74fcc883332e16f3e127042a9be85cdc5e52e4e0d6 WatchSource:0}: Error finding container 358a079d924109d75fcc6e74fcc883332e16f3e127042a9be85cdc5e52e4e0d6: Status 404 returned error can't find the container with id 358a079d924109d75fcc6e74fcc883332e16f3e127042a9be85cdc5e52e4e0d6 Oct 30 09:30:02.916124 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:02.916093 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d95d35-29889210-rf62t"] Oct 30 09:30:02.943693 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:02.943655 4562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28431c22-78da-4145-9842-e7d92b360558-istio-token\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:30:02.943693 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:02.943683 4562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28431c22-78da-4145-9842-e7d92b360558-istio-envoy\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:30:02.943693 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:02.943696 4562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28431c22-78da-4145-9842-e7d92b360558-istio-podinfo\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:30:02.943937 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:02.943707 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h77tv\" (UniqueName: \"kubernetes.io/projected/28431c22-78da-4145-9842-e7d92b360558-kube-api-access-h77tv\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:30:02.958564 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:30:02.958536 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2198e031_552b_46f4_adc2_bfbf3bb08dfd.slice/crio-a1b7d4111220339dc9fea60ed5502ae9ed15b81f4cd86c4ec609195da8e646c5 WatchSource:0}: Error finding container a1b7d4111220339dc9fea60ed5502ae9ed15b81f4cd86c4ec609195da8e646c5: Status 404 returned error can't find the container with id a1b7d4111220339dc9fea60ed5502ae9ed15b81f4cd86c4ec609195da8e646c5 Oct 30 09:30:02.960422 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:02.960366 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-eff9fd-6b784dd94d-lrzsf"] Oct 30 09:30:03.007612 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:03.007588 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-eff9fd-847fb7448c-5n9rh"] Oct 30 09:30:03.011699 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:30:03.011675 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8b60df6_3c80_4c90_9960_0af22206d277.slice/crio-2c82463ff68f0897664fdf3e14c456f195b5e59f3b35a6b8e9b3478cdf9a656f WatchSource:0}: Error finding container 2c82463ff68f0897664fdf3e14c456f195b5e59f3b35a6b8e9b3478cdf9a656f: Status 404 returned error can't find the container with id 2c82463ff68f0897664fdf3e14c456f195b5e59f3b35a6b8e9b3478cdf9a656f Oct 30 09:30:03.197459 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:03.196914 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eff9fd-847fb7448c-5n9rh" event={"ID":"c8b60df6-3c80-4c90-9960-0af22206d277","Type":"ContainerStarted","Data":"e13ac977f6e2fa1b1da1d44eb3973dc8615145942af3fbf2f0ff55b45d584c7e"} Oct 30 09:30:03.197716 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:03.197487 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eff9fd-847fb7448c-5n9rh" event={"ID":"c8b60df6-3c80-4c90-9960-0af22206d277","Type":"ContainerStarted","Data":"2c82463ff68f0897664fdf3e14c456f195b5e59f3b35a6b8e9b3478cdf9a656f"} Oct 30 09:30:03.197716 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:03.197521 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eff9fd-847fb7448c-5n9rh" Oct 30 09:30:03.198538 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:03.198506 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d95d35-29889210-rf62t" event={"ID":"57b6fb70-7cbb-46ee-ac05-be848af5349f","Type":"ContainerStarted","Data":"358a079d924109d75fcc6e74fcc883332e16f3e127042a9be85cdc5e52e4e0d6"} Oct 30 09:30:03.199880 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:03.199855 4562 generic.go:356] "Generic (PLEG): container finished" podID="28431c22-78da-4145-9842-e7d92b360558" containerID="a245dbbb93cdaea99d02ce7029d9269ce0cdaddc9f2e9de5aeddce8277d4d83d" exitCode=0 Oct 30 09:30:03.199988 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:03.199906 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb" Oct 30 09:30:03.199988 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:03.199926 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb" event={"ID":"28431c22-78da-4145-9842-e7d92b360558","Type":"ContainerDied","Data":"a245dbbb93cdaea99d02ce7029d9269ce0cdaddc9f2e9de5aeddce8277d4d83d"} Oct 30 09:30:03.199988 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:03.199966 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb" event={"ID":"28431c22-78da-4145-9842-e7d92b360558","Type":"ContainerDied","Data":"52545ea1acdef5b53f73ef2f9646595ffdd6a4450fe38f0256b7f7bd3cfb1d9c"} Oct 30 09:30:03.199988 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:03.199987 4562 scope.go:122] "RemoveContainer" containerID="a245dbbb93cdaea99d02ce7029d9269ce0cdaddc9f2e9de5aeddce8277d4d83d" Oct 30 09:30:03.201734 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:03.201708 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889210-7c9t8" event={"ID":"e525d80e-1056-4e3b-9ded-485e2f174a78","Type":"ContainerStarted","Data":"b3533d385c29aec4264184fd638f2073c1d94dd2deea400871d242eda1207c9c"} Oct 30 09:30:03.203345 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:03.203310 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-eff9fd-6b784dd94d-lrzsf" event={"ID":"2198e031-552b-46f4-adc2-bfbf3bb08dfd","Type":"ContainerStarted","Data":"01f5816caf5272858c6ad8aa76fa122503121bd7d7f2584d011eccb03cff7f92"} Oct 30 09:30:03.203477 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:03.203351 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-eff9fd-6b784dd94d-lrzsf" event={"ID":"2198e031-552b-46f4-adc2-bfbf3bb08dfd","Type":"ContainerStarted","Data":"a1b7d4111220339dc9fea60ed5502ae9ed15b81f4cd86c4ec609195da8e646c5"} Oct 30 09:30:03.203477 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:03.203447 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-eff9fd-6b784dd94d-lrzsf" Oct 30 09:30:03.210353 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:03.210339 4562 scope.go:122] "RemoveContainer" containerID="a245dbbb93cdaea99d02ce7029d9269ce0cdaddc9f2e9de5aeddce8277d4d83d" Oct 30 09:30:03.210681 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:30:03.210661 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a245dbbb93cdaea99d02ce7029d9269ce0cdaddc9f2e9de5aeddce8277d4d83d\": container with ID starting with a245dbbb93cdaea99d02ce7029d9269ce0cdaddc9f2e9de5aeddce8277d4d83d not found: ID does not exist" containerID="a245dbbb93cdaea99d02ce7029d9269ce0cdaddc9f2e9de5aeddce8277d4d83d" Oct 30 09:30:03.210724 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:03.210688 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a245dbbb93cdaea99d02ce7029d9269ce0cdaddc9f2e9de5aeddce8277d4d83d"} err="failed to get container status \"a245dbbb93cdaea99d02ce7029d9269ce0cdaddc9f2e9de5aeddce8277d4d83d\": rpc error: code = NotFound desc = could not find container \"a245dbbb93cdaea99d02ce7029d9269ce0cdaddc9f2e9de5aeddce8277d4d83d\": container with ID starting with a245dbbb93cdaea99d02ce7029d9269ce0cdaddc9f2e9de5aeddce8277d4d83d not found: ID does not exist" Oct 30 09:30:03.242514 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:03.242441 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eff9fd-847fb7448c-5n9rh" podStartSLOduration=3.242427722 podStartE2EDuration="3.242427722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:30:03.238951162 +0000 UTC m=+2163.500512174" watchObservedRunningTime="2026-10-30 09:30:03.242427722 +0000 UTC m=+2163.503988710" Oct 30 09:30:03.278952 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:03.278889 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-eff9fd-6b784dd94d-lrzsf" podStartSLOduration=3.278871763 podStartE2EDuration="3.278871763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:30:03.278293027 +0000 UTC m=+2163.539854018" watchObservedRunningTime="2026-10-30 09:30:03.278871763 +0000 UTC m=+2163.540432751" Oct 30 09:30:03.316849 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:03.316816 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb"] Oct 30 09:30:03.320251 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:03.320221 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-kn2tb"] Oct 30 09:30:03.879485 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:03.879445 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28431c22-78da-4145-9842-e7d92b360558" path="/var/lib/kubelet/pods/28431c22-78da-4145-9842-e7d92b360558/volumes" Oct 30 09:30:06.230321 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:06.230206 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d95d35-29889210-rf62t" event={"ID":"57b6fb70-7cbb-46ee-ac05-be848af5349f","Type":"ContainerStarted","Data":"7825e8887badc8033b3f475c1db6bdd3daa4c4e1d19bde41b6eb8cdf034d8edb"} Oct 30 09:30:06.271204 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:06.271156 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d95d35-29889210-rf62t" podStartSLOduration=3.552300307 podStartE2EDuration="6.271143265s" totalImagesPullingTime="2.718842958s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:30:06.26789303 +0000 UTC m=+2166.529454017" watchObservedRunningTime="2026-10-30 09:30:06.271143265 +0000 UTC m=+2166.532704285" Oct 30 09:30:07.239486 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:07.239450 4562 generic.go:356] "Generic (PLEG): container finished" podID="57b6fb70-7cbb-46ee-ac05-be848af5349f" containerID="7825e8887badc8033b3f475c1db6bdd3daa4c4e1d19bde41b6eb8cdf034d8edb" exitCode=60 Oct 30 09:30:07.239963 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:07.239528 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d95d35-29889210-rf62t" event={"ID":"57b6fb70-7cbb-46ee-ac05-be848af5349f","Type":"ContainerDied","Data":"7825e8887badc8033b3f475c1db6bdd3daa4c4e1d19bde41b6eb8cdf034d8edb"} Oct 30 09:30:07.239963 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:07.239785 4562 scope.go:122] "RemoveContainer" containerID="7825e8887badc8033b3f475c1db6bdd3daa4c4e1d19bde41b6eb8cdf034d8edb" Oct 30 09:30:07.241192 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:07.241167 4562 generic.go:356] "Generic (PLEG): container finished" podID="e525d80e-1056-4e3b-9ded-485e2f174a78" containerID="73bf4f2f048f60012b03918080fe46197812709387a7f60d00f9176e895b2ef4" exitCode=60 Oct 30 09:30:07.241304 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:07.241251 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889210-7c9t8" event={"ID":"e525d80e-1056-4e3b-9ded-485e2f174a78","Type":"ContainerDied","Data":"73bf4f2f048f60012b03918080fe46197812709387a7f60d00f9176e895b2ef4"} Oct 30 09:30:07.241516 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:07.241496 4562 scope.go:122] "RemoveContainer" containerID="73bf4f2f048f60012b03918080fe46197812709387a7f60d00f9176e895b2ef4" Oct 30 09:30:08.251410 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:08.251366 4562 generic.go:356] "Generic (PLEG): container finished" podID="57b6fb70-7cbb-46ee-ac05-be848af5349f" containerID="3ff47c18992f4c84c846bacf5509c9ca03710d20f44b2a8a5be4f7f9f135b021" exitCode=60 Oct 30 09:30:08.251804 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:08.251413 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d95d35-29889210-rf62t" event={"ID":"57b6fb70-7cbb-46ee-ac05-be848af5349f","Type":"ContainerDied","Data":"3ff47c18992f4c84c846bacf5509c9ca03710d20f44b2a8a5be4f7f9f135b021"} Oct 30 09:30:08.251804 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:08.251479 4562 scope.go:122] "RemoveContainer" containerID="7825e8887badc8033b3f475c1db6bdd3daa4c4e1d19bde41b6eb8cdf034d8edb" Oct 30 09:30:08.251905 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:08.251833 4562 scope.go:122] "RemoveContainer" containerID="3ff47c18992f4c84c846bacf5509c9ca03710d20f44b2a8a5be4f7f9f135b021" Oct 30 09:30:08.252296 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:30:08.252213 4562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-d95d35-29889210-rf62t_odh-ai-gateway-infra(57b6fb70-7cbb-46ee-ac05-be848af5349f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d95d35-29889210-rf62t" podUID="57b6fb70-7cbb-46ee-ac05-be848af5349f" Oct 30 09:30:08.255677 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:08.255650 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889210-7c9t8" event={"ID":"e525d80e-1056-4e3b-9ded-485e2f174a78","Type":"ContainerStarted","Data":"d9568bc938411bf76970c1b37d05860c3c84e5d3a9897ef4cab3e887ed1f94cf"} Oct 30 09:30:08.350455 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:08.335868 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889210-7c9t8" podStartSLOduration=5.476103823 podStartE2EDuration="8.335846145s" totalImagesPullingTime="2.859742322s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:30:08.33307804 +0000 UTC m=+2168.594639031" watchObservedRunningTime="2026-10-30 09:30:08.335846145 +0000 UTC m=+2168.597407134" Oct 30 09:30:09.264329 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:09.264300 4562 scope.go:122] "RemoveContainer" containerID="3ff47c18992f4c84c846bacf5509c9ca03710d20f44b2a8a5be4f7f9f135b021" Oct 30 09:30:09.264836 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:30:09.264661 4562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-d95d35-29889210-rf62t_odh-ai-gateway-infra(57b6fb70-7cbb-46ee-ac05-be848af5349f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d95d35-29889210-rf62t" podUID="57b6fb70-7cbb-46ee-ac05-be848af5349f" Oct 30 09:30:09.266194 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:09.266165 4562 generic.go:356] "Generic (PLEG): container finished" podID="e525d80e-1056-4e3b-9ded-485e2f174a78" containerID="d9568bc938411bf76970c1b37d05860c3c84e5d3a9897ef4cab3e887ed1f94cf" exitCode=60 Oct 30 09:30:09.266320 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:09.266250 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889210-7c9t8" event={"ID":"e525d80e-1056-4e3b-9ded-485e2f174a78","Type":"ContainerDied","Data":"d9568bc938411bf76970c1b37d05860c3c84e5d3a9897ef4cab3e887ed1f94cf"} Oct 30 09:30:09.266364 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:09.266336 4562 scope.go:122] "RemoveContainer" containerID="73bf4f2f048f60012b03918080fe46197812709387a7f60d00f9176e895b2ef4" Oct 30 09:30:09.266512 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:09.266495 4562 scope.go:122] "RemoveContainer" containerID="d9568bc938411bf76970c1b37d05860c3c84e5d3a9897ef4cab3e887ed1f94cf" Oct 30 09:30:09.266804 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:30:09.266783 4562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889210-7c9t8_odh-ai-gateway-infra(e525d80e-1056-4e3b-9ded-485e2f174a78)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889210-7c9t8" podUID="e525d80e-1056-4e3b-9ded-485e2f174a78" Oct 30 09:30:09.567158 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:09.567075 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-15e1cc-79d76cd75-hkf2r" Oct 30 09:30:09.569487 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:09.569449 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-15e1cc-c988685bd-6z7kz" Oct 30 09:30:10.274062 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:10.274034 4562 scope.go:122] "RemoveContainer" containerID="d9568bc938411bf76970c1b37d05860c3c84e5d3a9897ef4cab3e887ed1f94cf" Oct 30 09:30:10.274454 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:30:10.274283 4562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889210-7c9t8_odh-ai-gateway-infra(e525d80e-1056-4e3b-9ded-485e2f174a78)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889210-7c9t8" podUID="e525d80e-1056-4e3b-9ded-485e2f174a78" Oct 30 09:30:11.155060 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:11.155028 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c421e892-9pkx9"] Oct 30 09:30:11.155686 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:11.155671 4562 cpu_manager.go:414] "containerMap: removing container" podUID="28431c22-78da-4145-9842-e7d92b360558" containerName="istio-proxy" Oct 30 09:30:11.155686 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:11.155686 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28431c22-78da-4145-9842-e7d92b360558" containerName="istio-proxy" Oct 30 09:30:11.155802 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:11.155775 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="28431c22-78da-4145-9842-e7d92b360558" containerName="istio-proxy" Oct 30 09:30:11.160809 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:11.160787 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c421e892-9pkx9" Oct 30 09:30:11.176413 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:11.176375 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c421e892-9pkx9"] Oct 30 09:30:11.184750 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:11.184726 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:30:11.265874 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:11.265839 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecab733b-751b-4448-a556-4a70875db2e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c421e892-9pkx9\" (UID: \"ecab733b-751b-4448-a556-4a70875db2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c421e892-9pkx9" Oct 30 09:30:11.266034 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:11.265882 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frwrn\" (UniqueName: \"kubernetes.io/projected/ecab733b-751b-4448-a556-4a70875db2e5-kube-api-access-frwrn\") pod \"maas-api-revoke-sub-c421e892-9pkx9\" (UID: \"ecab733b-751b-4448-a556-4a70875db2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c421e892-9pkx9" Oct 30 09:30:11.266077 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:11.266029 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecab733b-751b-4448-a556-4a70875db2e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c421e892-9pkx9\" (UID: \"ecab733b-751b-4448-a556-4a70875db2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c421e892-9pkx9" Oct 30 09:30:11.367658 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:11.367618 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frwrn\" (UniqueName: \"kubernetes.io/projected/ecab733b-751b-4448-a556-4a70875db2e5-kube-api-access-frwrn\") pod \"maas-api-revoke-sub-c421e892-9pkx9\" (UID: \"ecab733b-751b-4448-a556-4a70875db2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c421e892-9pkx9" Oct 30 09:30:11.368040 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:11.367714 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecab733b-751b-4448-a556-4a70875db2e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c421e892-9pkx9\" (UID: \"ecab733b-751b-4448-a556-4a70875db2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c421e892-9pkx9" Oct 30 09:30:11.368040 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:11.367887 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecab733b-751b-4448-a556-4a70875db2e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c421e892-9pkx9\" (UID: \"ecab733b-751b-4448-a556-4a70875db2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c421e892-9pkx9" Oct 30 09:30:11.368438 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:11.368372 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecab733b-751b-4448-a556-4a70875db2e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c421e892-9pkx9\" (UID: \"ecab733b-751b-4448-a556-4a70875db2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c421e892-9pkx9" Oct 30 09:30:11.370179 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:11.370159 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecab733b-751b-4448-a556-4a70875db2e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c421e892-9pkx9\" (UID: \"ecab733b-751b-4448-a556-4a70875db2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c421e892-9pkx9" Oct 30 09:30:11.388557 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:11.388530 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frwrn\" (UniqueName: \"kubernetes.io/projected/ecab733b-751b-4448-a556-4a70875db2e5-kube-api-access-frwrn\") pod \"maas-api-revoke-sub-c421e892-9pkx9\" (UID: \"ecab733b-751b-4448-a556-4a70875db2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c421e892-9pkx9" Oct 30 09:30:11.475959 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:11.475933 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c421e892-9pkx9" Oct 30 09:30:11.775063 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:11.774891 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c421e892-9pkx9"] Oct 30 09:30:12.288978 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:12.288937 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c421e892-9pkx9" event={"ID":"ecab733b-751b-4448-a556-4a70875db2e5","Type":"ContainerStarted","Data":"390867dc8e77f223d0ad16c2e13467809a7ff6279076ed3e5cff3cd4704ede34"} Oct 30 09:30:12.289158 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:12.289003 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c421e892-9pkx9" event={"ID":"ecab733b-751b-4448-a556-4a70875db2e5","Type":"ContainerStarted","Data":"fd107476e0bd474acbf79ee9629e9ae57431da5acfbf6d21b947b1d1fcc39b0c"} Oct 30 09:30:12.329972 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:12.329874 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c421e892-9pkx9" podStartSLOduration=1.3298598990000001 podStartE2EDuration="1.329859899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:30:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:30:12.329012142 +0000 UTC m=+2172.590573140" watchObservedRunningTime="2026-10-30 09:30:12.329859899 +0000 UTC m=+2172.591420896" Oct 30 09:30:13.297267 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:13.297226 4562 generic.go:356] "Generic (PLEG): container finished" podID="ecab733b-751b-4448-a556-4a70875db2e5" containerID="390867dc8e77f223d0ad16c2e13467809a7ff6279076ed3e5cff3cd4704ede34" exitCode=0 Oct 30 09:30:13.297685 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:13.297315 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c421e892-9pkx9" event={"ID":"ecab733b-751b-4448-a556-4a70875db2e5","Type":"ContainerDied","Data":"390867dc8e77f223d0ad16c2e13467809a7ff6279076ed3e5cff3cd4704ede34"} Oct 30 09:30:14.218499 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:14.218467 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-eff9fd-6b784dd94d-lrzsf" Oct 30 09:30:14.218701 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:14.218537 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eff9fd-847fb7448c-5n9rh" Oct 30 09:30:14.471339 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:14.471278 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c421e892-9pkx9" Oct 30 09:30:14.519740 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:14.519711 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecab733b-751b-4448-a556-4a70875db2e5-kube-api-access-frwrn\" (UniqueName: \"kubernetes.io/projected/ecab733b-751b-4448-a556-4a70875db2e5-kube-api-access-frwrn\") pod \"ecab733b-751b-4448-a556-4a70875db2e5\" (UID: \"ecab733b-751b-4448-a556-4a70875db2e5\") " Oct 30 09:30:14.519960 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:14.519943 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecab733b-751b-4448-a556-4a70875db2e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecab733b-751b-4448-a556-4a70875db2e5-maas-api-xks-ca\") pod \"ecab733b-751b-4448-a556-4a70875db2e5\" (UID: \"ecab733b-751b-4448-a556-4a70875db2e5\") " Oct 30 09:30:14.520031 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:14.519990 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecab733b-751b-4448-a556-4a70875db2e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecab733b-751b-4448-a556-4a70875db2e5-maas-api-openshift-ca\") pod \"ecab733b-751b-4448-a556-4a70875db2e5\" (UID: \"ecab733b-751b-4448-a556-4a70875db2e5\") " Oct 30 09:30:14.520788 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:14.520755 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecab733b-751b-4448-a556-4a70875db2e5-maas-api-openshift-ca" pod "ecab733b-751b-4448-a556-4a70875db2e5" (UID: "ecab733b-751b-4448-a556-4a70875db2e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:30:14.522079 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:14.522048 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecab733b-751b-4448-a556-4a70875db2e5-maas-api-xks-ca" pod "ecab733b-751b-4448-a556-4a70875db2e5" (UID: "ecab733b-751b-4448-a556-4a70875db2e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:30:14.529940 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:14.529912 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecab733b-751b-4448-a556-4a70875db2e5-kube-api-access-frwrn" pod "ecab733b-751b-4448-a556-4a70875db2e5" (UID: "ecab733b-751b-4448-a556-4a70875db2e5"). InnerVolumeSpecName "kube-api-access-frwrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:30:14.621759 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:14.621724 4562 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecab733b-751b-4448-a556-4a70875db2e5-maas-api-xks-ca\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:30:14.621759 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:14.621757 4562 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecab733b-751b-4448-a556-4a70875db2e5-maas-api-openshift-ca\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:30:14.621759 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:14.621768 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frwrn\" (UniqueName: \"kubernetes.io/projected/ecab733b-751b-4448-a556-4a70875db2e5-kube-api-access-frwrn\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:30:15.317485 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:15.317442 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c421e892-9pkx9" event={"ID":"ecab733b-751b-4448-a556-4a70875db2e5","Type":"ContainerDied","Data":"fd107476e0bd474acbf79ee9629e9ae57431da5acfbf6d21b947b1d1fcc39b0c"} Oct 30 09:30:15.317637 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:15.317508 4562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd107476e0bd474acbf79ee9629e9ae57431da5acfbf6d21b947b1d1fcc39b0c" Oct 30 09:30:15.317637 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:15.317519 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c421e892-9pkx9" Oct 30 09:30:15.444371 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:15.444338 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-eff9fd-b94d9594f-z9mbl"] Oct 30 09:30:15.445034 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:15.445019 4562 cpu_manager.go:414] "containerMap: removing container" podUID="ecab733b-751b-4448-a556-4a70875db2e5" containerName="revoke-keys" Oct 30 09:30:15.445086 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:15.445035 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecab733b-751b-4448-a556-4a70875db2e5" containerName="revoke-keys" Oct 30 09:30:15.445134 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:15.445125 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecab733b-751b-4448-a556-4a70875db2e5" containerName="revoke-keys" Oct 30 09:30:15.448683 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:15.448652 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eff9fd-b94d9594f-z9mbl" Oct 30 09:30:15.463884 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:15.463852 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-eff9fd-serving-cert\"" Oct 30 09:30:15.467309 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:15.467281 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-eff9fd-b94d9594f-z9mbl"] Oct 30 09:30:15.532746 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:15.532715 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fb8f8cd-509a-49e4-abcf-82095b5a9731-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-eff9fd-b94d9594f-z9mbl\" (UID: \"3fb8f8cd-509a-49e4-abcf-82095b5a9731\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eff9fd-b94d9594f-z9mbl" Oct 30 09:30:15.533219 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:15.532861 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4psg5\" (UniqueName: \"kubernetes.io/projected/3fb8f8cd-509a-49e4-abcf-82095b5a9731-kube-api-access-4psg5\") pod \"maas-api-e2e-praxis-a-eff9fd-b94d9594f-z9mbl\" (UID: \"3fb8f8cd-509a-49e4-abcf-82095b5a9731\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eff9fd-b94d9594f-z9mbl" Oct 30 09:30:15.533219 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:15.533072 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fb8f8cd-509a-49e4-abcf-82095b5a9731-maas-api-tls\") pod \"maas-api-e2e-praxis-a-eff9fd-b94d9594f-z9mbl\" (UID: \"3fb8f8cd-509a-49e4-abcf-82095b5a9731\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eff9fd-b94d9594f-z9mbl" Oct 30 09:30:15.533219 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:15.533106 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fb8f8cd-509a-49e4-abcf-82095b5a9731-metrics-certs\") pod \"maas-api-e2e-praxis-a-eff9fd-b94d9594f-z9mbl\" (UID: \"3fb8f8cd-509a-49e4-abcf-82095b5a9731\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eff9fd-b94d9594f-z9mbl" Oct 30 09:30:15.634973 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:15.634887 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fb8f8cd-509a-49e4-abcf-82095b5a9731-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-eff9fd-b94d9594f-z9mbl\" (UID: \"3fb8f8cd-509a-49e4-abcf-82095b5a9731\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eff9fd-b94d9594f-z9mbl" Oct 30 09:30:15.634973 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:15.634971 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4psg5\" (UniqueName: \"kubernetes.io/projected/3fb8f8cd-509a-49e4-abcf-82095b5a9731-kube-api-access-4psg5\") pod \"maas-api-e2e-praxis-a-eff9fd-b94d9594f-z9mbl\" (UID: \"3fb8f8cd-509a-49e4-abcf-82095b5a9731\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eff9fd-b94d9594f-z9mbl" Oct 30 09:30:15.635174 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:15.635097 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fb8f8cd-509a-49e4-abcf-82095b5a9731-maas-api-tls\") pod \"maas-api-e2e-praxis-a-eff9fd-b94d9594f-z9mbl\" (UID: \"3fb8f8cd-509a-49e4-abcf-82095b5a9731\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eff9fd-b94d9594f-z9mbl" Oct 30 09:30:15.635174 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:15.635116 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fb8f8cd-509a-49e4-abcf-82095b5a9731-metrics-certs\") pod \"maas-api-e2e-praxis-a-eff9fd-b94d9594f-z9mbl\" (UID: \"3fb8f8cd-509a-49e4-abcf-82095b5a9731\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eff9fd-b94d9594f-z9mbl" Oct 30 09:30:15.635537 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:15.635513 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fb8f8cd-509a-49e4-abcf-82095b5a9731-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-eff9fd-b94d9594f-z9mbl\" (UID: \"3fb8f8cd-509a-49e4-abcf-82095b5a9731\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eff9fd-b94d9594f-z9mbl" Oct 30 09:30:15.637629 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:15.637607 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fb8f8cd-509a-49e4-abcf-82095b5a9731-maas-api-tls\") pod \"maas-api-e2e-praxis-a-eff9fd-b94d9594f-z9mbl\" (UID: \"3fb8f8cd-509a-49e4-abcf-82095b5a9731\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eff9fd-b94d9594f-z9mbl" Oct 30 09:30:15.637755 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:15.637637 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fb8f8cd-509a-49e4-abcf-82095b5a9731-metrics-certs\") pod \"maas-api-e2e-praxis-a-eff9fd-b94d9594f-z9mbl\" (UID: \"3fb8f8cd-509a-49e4-abcf-82095b5a9731\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eff9fd-b94d9594f-z9mbl" Oct 30 09:30:15.654659 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:15.654635 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4psg5\" (UniqueName: \"kubernetes.io/projected/3fb8f8cd-509a-49e4-abcf-82095b5a9731-kube-api-access-4psg5\") pod \"maas-api-e2e-praxis-a-eff9fd-b94d9594f-z9mbl\" (UID: \"3fb8f8cd-509a-49e4-abcf-82095b5a9731\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eff9fd-b94d9594f-z9mbl" Oct 30 09:30:15.706593 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:15.706561 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-d87c85/ipp-route-d87c85-kserve-7cc5cb6f7c-qk4q7"] Oct 30 09:30:15.713028 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:15.713005 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-d87c85/ipp-route-d87c85-kserve-7cc5cb6f7c-qk4q7" Oct 30 09:30:15.728628 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:15.728596 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-d87c85\"/\"openshift-service-ca.crt\"" Oct 30 09:30:15.728805 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:15.728794 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-d87c85\"/\"ipp-route-d87c85-kserve-self-signed-certs\"" Oct 30 09:30:15.728893 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:15.728876 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-d87c85\"/\"kube-root-ca.crt\"" Oct 30 09:30:15.729251 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:15.729226 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-d87c85\"/\"default-dockercfg-xxjbk\"" Oct 30 09:30:15.729960 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:15.729934 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-d87c85/ipp-route-d87c85-kserve-7cc5cb6f7c-qk4q7"] Oct 30 09:30:15.762326 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:15.762297 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eff9fd-b94d9594f-z9mbl" Oct 30 09:30:15.839267 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:15.838460 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bfc138b-8938-473c-adfe-052d9a722233-tmp-dir\") pod \"ipp-route-d87c85-kserve-7cc5cb6f7c-qk4q7\" (UID: \"9bfc138b-8938-473c-adfe-052d9a722233\") " pod="ai-tenant-e2e-ipp-a-d87c85/ipp-route-d87c85-kserve-7cc5cb6f7c-qk4q7" Oct 30 09:30:15.839267 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:15.838564 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bfc138b-8938-473c-adfe-052d9a722233-home\") pod \"ipp-route-d87c85-kserve-7cc5cb6f7c-qk4q7\" (UID: \"9bfc138b-8938-473c-adfe-052d9a722233\") " pod="ai-tenant-e2e-ipp-a-d87c85/ipp-route-d87c85-kserve-7cc5cb6f7c-qk4q7" Oct 30 09:30:15.839267 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:15.838599 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bfc138b-8938-473c-adfe-052d9a722233-dshm\") pod \"ipp-route-d87c85-kserve-7cc5cb6f7c-qk4q7\" (UID: \"9bfc138b-8938-473c-adfe-052d9a722233\") " pod="ai-tenant-e2e-ipp-a-d87c85/ipp-route-d87c85-kserve-7cc5cb6f7c-qk4q7" Oct 30 09:30:15.839267 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:15.838624 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bfc138b-8938-473c-adfe-052d9a722233-model-cache\") pod \"ipp-route-d87c85-kserve-7cc5cb6f7c-qk4q7\" (UID: \"9bfc138b-8938-473c-adfe-052d9a722233\") " pod="ai-tenant-e2e-ipp-a-d87c85/ipp-route-d87c85-kserve-7cc5cb6f7c-qk4q7" Oct 30 09:30:15.839267 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:15.838644 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bfc138b-8938-473c-adfe-052d9a722233-tls-certs\") pod \"ipp-route-d87c85-kserve-7cc5cb6f7c-qk4q7\" (UID: \"9bfc138b-8938-473c-adfe-052d9a722233\") " pod="ai-tenant-e2e-ipp-a-d87c85/ipp-route-d87c85-kserve-7cc5cb6f7c-qk4q7" Oct 30 09:30:15.839267 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:15.838782 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mdp8\" (UniqueName: \"kubernetes.io/projected/9bfc138b-8938-473c-adfe-052d9a722233-kube-api-access-2mdp8\") pod \"ipp-route-d87c85-kserve-7cc5cb6f7c-qk4q7\" (UID: \"9bfc138b-8938-473c-adfe-052d9a722233\") " pod="ai-tenant-e2e-ipp-a-d87c85/ipp-route-d87c85-kserve-7cc5cb6f7c-qk4q7" Oct 30 09:30:15.977407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:15.943063 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mdp8\" (UniqueName: \"kubernetes.io/projected/9bfc138b-8938-473c-adfe-052d9a722233-kube-api-access-2mdp8\") pod \"ipp-route-d87c85-kserve-7cc5cb6f7c-qk4q7\" (UID: \"9bfc138b-8938-473c-adfe-052d9a722233\") " pod="ai-tenant-e2e-ipp-a-d87c85/ipp-route-d87c85-kserve-7cc5cb6f7c-qk4q7" Oct 30 09:30:15.989576 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:15.989541 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bfc138b-8938-473c-adfe-052d9a722233-tmp-dir\") pod \"ipp-route-d87c85-kserve-7cc5cb6f7c-qk4q7\" (UID: \"9bfc138b-8938-473c-adfe-052d9a722233\") " pod="ai-tenant-e2e-ipp-a-d87c85/ipp-route-d87c85-kserve-7cc5cb6f7c-qk4q7" Oct 30 09:30:15.991272 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:15.991245 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bfc138b-8938-473c-adfe-052d9a722233-home\") pod \"ipp-route-d87c85-kserve-7cc5cb6f7c-qk4q7\" (UID: \"9bfc138b-8938-473c-adfe-052d9a722233\") " pod="ai-tenant-e2e-ipp-a-d87c85/ipp-route-d87c85-kserve-7cc5cb6f7c-qk4q7" Oct 30 09:30:15.992544 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:15.992522 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bfc138b-8938-473c-adfe-052d9a722233-dshm\") pod \"ipp-route-d87c85-kserve-7cc5cb6f7c-qk4q7\" (UID: \"9bfc138b-8938-473c-adfe-052d9a722233\") " pod="ai-tenant-e2e-ipp-a-d87c85/ipp-route-d87c85-kserve-7cc5cb6f7c-qk4q7" Oct 30 09:30:15.993038 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:15.993018 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bfc138b-8938-473c-adfe-052d9a722233-model-cache\") pod \"ipp-route-d87c85-kserve-7cc5cb6f7c-qk4q7\" (UID: \"9bfc138b-8938-473c-adfe-052d9a722233\") " pod="ai-tenant-e2e-ipp-a-d87c85/ipp-route-d87c85-kserve-7cc5cb6f7c-qk4q7" Oct 30 09:30:15.993478 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:15.993459 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bfc138b-8938-473c-adfe-052d9a722233-tls-certs\") pod \"ipp-route-d87c85-kserve-7cc5cb6f7c-qk4q7\" (UID: \"9bfc138b-8938-473c-adfe-052d9a722233\") " pod="ai-tenant-e2e-ipp-a-d87c85/ipp-route-d87c85-kserve-7cc5cb6f7c-qk4q7" Oct 30 09:30:15.994446 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:15.993414 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bfc138b-8938-473c-adfe-052d9a722233-model-cache\") pod \"ipp-route-d87c85-kserve-7cc5cb6f7c-qk4q7\" (UID: \"9bfc138b-8938-473c-adfe-052d9a722233\") " pod="ai-tenant-e2e-ipp-a-d87c85/ipp-route-d87c85-kserve-7cc5cb6f7c-qk4q7" Oct 30 09:30:15.994593 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:15.990912 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bfc138b-8938-473c-adfe-052d9a722233-tmp-dir\") pod \"ipp-route-d87c85-kserve-7cc5cb6f7c-qk4q7\" (UID: \"9bfc138b-8938-473c-adfe-052d9a722233\") " pod="ai-tenant-e2e-ipp-a-d87c85/ipp-route-d87c85-kserve-7cc5cb6f7c-qk4q7" Oct 30 09:30:15.994746 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:15.992039 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bfc138b-8938-473c-adfe-052d9a722233-home\") pod \"ipp-route-d87c85-kserve-7cc5cb6f7c-qk4q7\" (UID: \"9bfc138b-8938-473c-adfe-052d9a722233\") " pod="ai-tenant-e2e-ipp-a-d87c85/ipp-route-d87c85-kserve-7cc5cb6f7c-qk4q7" Oct 30 09:30:16.002241 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:16.002174 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bfc138b-8938-473c-adfe-052d9a722233-tls-certs\") pod \"ipp-route-d87c85-kserve-7cc5cb6f7c-qk4q7\" (UID: \"9bfc138b-8938-473c-adfe-052d9a722233\") " pod="ai-tenant-e2e-ipp-a-d87c85/ipp-route-d87c85-kserve-7cc5cb6f7c-qk4q7" Oct 30 09:30:16.003174 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:16.003123 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bfc138b-8938-473c-adfe-052d9a722233-dshm\") pod \"ipp-route-d87c85-kserve-7cc5cb6f7c-qk4q7\" (UID: \"9bfc138b-8938-473c-adfe-052d9a722233\") " pod="ai-tenant-e2e-ipp-a-d87c85/ipp-route-d87c85-kserve-7cc5cb6f7c-qk4q7" Oct 30 09:30:16.018240 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:16.018177 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mdp8\" (UniqueName: \"kubernetes.io/projected/9bfc138b-8938-473c-adfe-052d9a722233-kube-api-access-2mdp8\") pod \"ipp-route-d87c85-kserve-7cc5cb6f7c-qk4q7\" (UID: \"9bfc138b-8938-473c-adfe-052d9a722233\") " pod="ai-tenant-e2e-ipp-a-d87c85/ipp-route-d87c85-kserve-7cc5cb6f7c-qk4q7" Oct 30 09:30:16.030992 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:16.030656 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-d87c85/ipp-route-d87c85-kserve-7cc5cb6f7c-qk4q7" Oct 30 09:30:16.082637 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:16.082608 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-eff9fd-b94d9594f-z9mbl"] Oct 30 09:30:16.083042 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:30:16.083009 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fb8f8cd_509a_49e4_abcf_82095b5a9731.slice/crio-bba653ece5fe4154af56026a52f3b0a2c2a6c05fcf1d60a03c52a304962769fb WatchSource:0}: Error finding container bba653ece5fe4154af56026a52f3b0a2c2a6c05fcf1d60a03c52a304962769fb: Status 404 returned error can't find the container with id bba653ece5fe4154af56026a52f3b0a2c2a6c05fcf1d60a03c52a304962769fb Oct 30 09:30:16.325616 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:16.325537 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eff9fd-b94d9594f-z9mbl" event={"ID":"3fb8f8cd-509a-49e4-abcf-82095b5a9731","Type":"ContainerStarted","Data":"bba653ece5fe4154af56026a52f3b0a2c2a6c05fcf1d60a03c52a304962769fb"} Oct 30 09:30:16.345555 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:30:16.345521 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9bfc138b_8938_473c_adfe_052d9a722233.slice/crio-e8065c2a4f305b2f784a60ae2e7cc0c55cb3bc4f5c99ceebe967e6fe986f123e WatchSource:0}: Error finding container e8065c2a4f305b2f784a60ae2e7cc0c55cb3bc4f5c99ceebe967e6fe986f123e: Status 404 returned error can't find the container with id e8065c2a4f305b2f784a60ae2e7cc0c55cb3bc4f5c99ceebe967e6fe986f123e Oct 30 09:30:16.347027 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:16.346999 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-d87c85/ipp-route-d87c85-kserve-7cc5cb6f7c-qk4q7"] Oct 30 09:30:17.333917 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:17.333875 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eff9fd-b94d9594f-z9mbl" event={"ID":"3fb8f8cd-509a-49e4-abcf-82095b5a9731","Type":"ContainerStarted","Data":"010aee899fb1c91c00491e97836c2582c2b9d8d5db57a3221fc98e72cf17bc29"} Oct 30 09:30:17.334365 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:17.334064 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eff9fd-b94d9594f-z9mbl" Oct 30 09:30:17.335609 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:17.335579 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-d87c85/ipp-route-d87c85-kserve-7cc5cb6f7c-qk4q7" event={"ID":"9bfc138b-8938-473c-adfe-052d9a722233","Type":"ContainerStarted","Data":"b6b5cb03d0e4f89c74177e9a690093f3b032a244561b16401f52900e43de085f"} Oct 30 09:30:17.335718 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:17.335615 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-d87c85/ipp-route-d87c85-kserve-7cc5cb6f7c-qk4q7" event={"ID":"9bfc138b-8938-473c-adfe-052d9a722233","Type":"ContainerStarted","Data":"e8065c2a4f305b2f784a60ae2e7cc0c55cb3bc4f5c99ceebe967e6fe986f123e"} Oct 30 09:30:17.376997 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:17.376943 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eff9fd-b94d9594f-z9mbl" podStartSLOduration=2.024539779 podStartE2EDuration="2.376929793s" totalImagesPullingTime="352.390014ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:30:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:30:17.375862641 +0000 UTC m=+2177.637423653" watchObservedRunningTime="2026-10-30 09:30:17.376929793 +0000 UTC m=+2177.638490781" Oct 30 09:30:17.420284 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:17.420233 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-d87c85/ipp-route-d87c85-kserve-7cc5cb6f7c-qk4q7" podStartSLOduration=2.42021866 podStartE2EDuration="2.42021866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:30:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:30:17.418423293 +0000 UTC m=+2177.679984281" watchObservedRunningTime="2026-10-30 09:30:17.42021866 +0000 UTC m=+2177.681779648" Oct 30 09:30:18.031321 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:18.031286 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-d87c85/ipp-route-d87c85-kserve-7cc5cb6f7c-qk4q7" Oct 30 09:30:19.876428 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:19.876362 4562 scope.go:122] "RemoveContainer" containerID="d9568bc938411bf76970c1b37d05860c3c84e5d3a9897ef4cab3e887ed1f94cf" Oct 30 09:30:20.356332 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:20.356300 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889210-7c9t8" event={"ID":"e525d80e-1056-4e3b-9ded-485e2f174a78","Type":"ContainerStarted","Data":"d5817fb399ccedf29511fc9020d09523e66b1a6a652d56cec7752c4df0971268"} Oct 30 09:30:20.867889 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:20.867856 4562 scope.go:122] "RemoveContainer" containerID="3ff47c18992f4c84c846bacf5509c9ca03710d20f44b2a8a5be4f7f9f135b021" Oct 30 09:30:21.365449 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:21.365412 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d95d35-29889210-rf62t" event={"ID":"57b6fb70-7cbb-46ee-ac05-be848af5349f","Type":"ContainerStarted","Data":"ed23989427edda3b7d6d35019507c14827e40826adc79b08f1cc5c9158372fe4"} Oct 30 09:30:21.367798 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:21.367772 4562 generic.go:356] "Generic (PLEG): container finished" podID="e525d80e-1056-4e3b-9ded-485e2f174a78" containerID="d5817fb399ccedf29511fc9020d09523e66b1a6a652d56cec7752c4df0971268" exitCode=60 Oct 30 09:30:21.367969 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:21.367842 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889210-7c9t8" event={"ID":"e525d80e-1056-4e3b-9ded-485e2f174a78","Type":"ContainerDied","Data":"d5817fb399ccedf29511fc9020d09523e66b1a6a652d56cec7752c4df0971268"} Oct 30 09:30:21.367969 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:21.367940 4562 scope.go:122] "RemoveContainer" containerID="d9568bc938411bf76970c1b37d05860c3c84e5d3a9897ef4cab3e887ed1f94cf" Oct 30 09:30:21.368268 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:21.368250 4562 scope.go:122] "RemoveContainer" containerID="d5817fb399ccedf29511fc9020d09523e66b1a6a652d56cec7752c4df0971268" Oct 30 09:30:21.368635 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:30:21.368604 4562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889210-7c9t8_odh-ai-gateway-infra(e525d80e-1056-4e3b-9ded-485e2f174a78)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889210-7c9t8" podUID="e525d80e-1056-4e3b-9ded-485e2f174a78" Oct 30 09:30:21.427247 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:21.427217 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889210-7c9t8"] Oct 30 09:30:22.376303 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:22.376269 4562 generic.go:356] "Generic (PLEG): container finished" podID="57b6fb70-7cbb-46ee-ac05-be848af5349f" containerID="ed23989427edda3b7d6d35019507c14827e40826adc79b08f1cc5c9158372fe4" exitCode=60 Oct 30 09:30:22.376745 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:22.376349 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d95d35-29889210-rf62t" event={"ID":"57b6fb70-7cbb-46ee-ac05-be848af5349f","Type":"ContainerDied","Data":"ed23989427edda3b7d6d35019507c14827e40826adc79b08f1cc5c9158372fe4"} Oct 30 09:30:22.376745 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:22.376453 4562 scope.go:122] "RemoveContainer" containerID="3ff47c18992f4c84c846bacf5509c9ca03710d20f44b2a8a5be4f7f9f135b021" Oct 30 09:30:22.376902 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:22.376885 4562 scope.go:122] "RemoveContainer" containerID="ed23989427edda3b7d6d35019507c14827e40826adc79b08f1cc5c9158372fe4" Oct 30 09:30:22.377287 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:30:22.377264 4562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-d95d35-29889210-rf62t_odh-ai-gateway-infra(57b6fb70-7cbb-46ee-ac05-be848af5349f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d95d35-29889210-rf62t" podUID="57b6fb70-7cbb-46ee-ac05-be848af5349f" Oct 30 09:30:22.418323 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:22.418288 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d95d35-29889210-rf62t"] Oct 30 09:30:22.539144 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:22.539111 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889210-7c9t8" Oct 30 09:30:22.628693 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:22.628588 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e525d80e-1056-4e3b-9ded-485e2f174a78-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e525d80e-1056-4e3b-9ded-485e2f174a78-maas-api-service-ca\") pod \"e525d80e-1056-4e3b-9ded-485e2f174a78\" (UID: \"e525d80e-1056-4e3b-9ded-485e2f174a78\") " Oct 30 09:30:22.628876 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:22.628718 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e525d80e-1056-4e3b-9ded-485e2f174a78-kube-api-access-4n7fc\" (UniqueName: \"kubernetes.io/projected/e525d80e-1056-4e3b-9ded-485e2f174a78-kube-api-access-4n7fc\") pod \"e525d80e-1056-4e3b-9ded-485e2f174a78\" (UID: \"e525d80e-1056-4e3b-9ded-485e2f174a78\") " Oct 30 09:30:22.629030 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:22.629006 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e525d80e-1056-4e3b-9ded-485e2f174a78-maas-api-service-ca" pod "e525d80e-1056-4e3b-9ded-485e2f174a78" (UID: "e525d80e-1056-4e3b-9ded-485e2f174a78"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:30:22.629319 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:22.629302 4562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e525d80e-1056-4e3b-9ded-485e2f174a78-maas-api-service-ca\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:30:22.630972 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:22.630948 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e525d80e-1056-4e3b-9ded-485e2f174a78-kube-api-access-4n7fc" pod "e525d80e-1056-4e3b-9ded-485e2f174a78" (UID: "e525d80e-1056-4e3b-9ded-485e2f174a78"). InnerVolumeSpecName "kube-api-access-4n7fc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:30:22.730646 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:22.730608 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4n7fc\" (UniqueName: \"kubernetes.io/projected/e525d80e-1056-4e3b-9ded-485e2f174a78-kube-api-access-4n7fc\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:30:23.349442 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:23.349414 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eff9fd-b94d9594f-z9mbl" Oct 30 09:30:23.390578 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:23.390549 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889210-7c9t8" Oct 30 09:30:23.391022 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:23.390604 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889210-7c9t8" event={"ID":"e525d80e-1056-4e3b-9ded-485e2f174a78","Type":"ContainerDied","Data":"b3533d385c29aec4264184fd638f2073c1d94dd2deea400871d242eda1207c9c"} Oct 30 09:30:23.391022 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:23.390686 4562 scope.go:122] "RemoveContainer" containerID="d5817fb399ccedf29511fc9020d09523e66b1a6a652d56cec7752c4df0971268" Oct 30 09:30:23.501499 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:23.501451 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889210-7c9t8"] Oct 30 09:30:23.506916 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:23.506819 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889210-7c9t8"] Oct 30 09:30:23.692441 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:23.691248 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d95d35-29889210-rf62t" Oct 30 09:30:23.771088 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:23.771049 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57b6fb70-7cbb-46ee-ac05-be848af5349f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57b6fb70-7cbb-46ee-ac05-be848af5349f-maas-api-service-ca\") pod \"57b6fb70-7cbb-46ee-ac05-be848af5349f\" (UID: \"57b6fb70-7cbb-46ee-ac05-be848af5349f\") " Oct 30 09:30:23.771770 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:23.771740 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57b6fb70-7cbb-46ee-ac05-be848af5349f-kube-api-access-zkcpg\" (UniqueName: \"kubernetes.io/projected/57b6fb70-7cbb-46ee-ac05-be848af5349f-kube-api-access-zkcpg\") pod \"57b6fb70-7cbb-46ee-ac05-be848af5349f\" (UID: \"57b6fb70-7cbb-46ee-ac05-be848af5349f\") " Oct 30 09:30:23.772113 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:23.771748 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57b6fb70-7cbb-46ee-ac05-be848af5349f-maas-api-service-ca" pod "57b6fb70-7cbb-46ee-ac05-be848af5349f" (UID: "57b6fb70-7cbb-46ee-ac05-be848af5349f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:30:23.772861 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:23.772839 4562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57b6fb70-7cbb-46ee-ac05-be848af5349f-maas-api-service-ca\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:30:23.831423 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:23.827751 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57b6fb70-7cbb-46ee-ac05-be848af5349f-kube-api-access-zkcpg" pod "57b6fb70-7cbb-46ee-ac05-be848af5349f" (UID: "57b6fb70-7cbb-46ee-ac05-be848af5349f"). InnerVolumeSpecName "kube-api-access-zkcpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:30:23.880837 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:23.880747 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e525d80e-1056-4e3b-9ded-485e2f174a78" path="/var/lib/kubelet/pods/e525d80e-1056-4e3b-9ded-485e2f174a78/volumes" Oct 30 09:30:23.881816 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:23.881786 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkcpg\" (UniqueName: \"kubernetes.io/projected/57b6fb70-7cbb-46ee-ac05-be848af5349f-kube-api-access-zkcpg\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:30:24.404407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:24.403853 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d95d35-29889210-rf62t" event={"ID":"57b6fb70-7cbb-46ee-ac05-be848af5349f","Type":"ContainerDied","Data":"358a079d924109d75fcc6e74fcc883332e16f3e127042a9be85cdc5e52e4e0d6"} Oct 30 09:30:24.404407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:24.403964 4562 scope.go:122] "RemoveContainer" containerID="ed23989427edda3b7d6d35019507c14827e40826adc79b08f1cc5c9158372fe4" Oct 30 09:30:24.404407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:24.404112 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d95d35-29889210-rf62t" Oct 30 09:30:24.477631 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:24.477595 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d95d35-29889210-rf62t"] Oct 30 09:30:24.483451 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:24.483376 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d95d35-29889210-rf62t"] Oct 30 09:30:25.852408 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:25.851466 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55" podUID="88be34c3-a9e9-4489-a449-206a99ef1663" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 09:30:25.946407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:25.945758 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57b6fb70-7cbb-46ee-ac05-be848af5349f" path="/var/lib/kubelet/pods/57b6fb70-7cbb-46ee-ac05-be848af5349f/volumes" Oct 30 09:30:26.035899 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:26.035865 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-d87c85/ipp-route-d87c85-kserve-7cc5cb6f7c-qk4q7" Oct 30 09:30:26.060415 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:26.060358 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/distinct-2-w1-c5447d-kserve-676749d548-xlmsf" podUID="c9ff897f-c520-4d0f-8820-e185b1d47dac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 09:30:26.129459 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:26.120463 4562 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c5447d/distinct-2-w1-c5447d-kserve-676749d548-xlmsf" podUID="c9ff897f-c520-4d0f-8820-e185b1d47dac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 09:30:26.163710 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:26.163680 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-d87c85/ipp-route-d87c85-kserve-7cc5cb6f7c-qk4q7" Oct 30 09:30:26.654404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:26.654196 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-d87c85/ipp-route-d87c85-kserve-7cc5cb6f7c-qk4q7" Oct 30 09:30:26.737328 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:26.737290 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp"] Oct 30 09:30:26.738525 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:26.738495 4562 cpu_manager.go:414] "containerMap: removing container" podUID="57b6fb70-7cbb-46ee-ac05-be848af5349f" containerName="cleanup" Oct 30 09:30:26.738679 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:26.738664 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57b6fb70-7cbb-46ee-ac05-be848af5349f" containerName="cleanup" Oct 30 09:30:26.738817 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:26.738803 4562 cpu_manager.go:414] "containerMap: removing container" podUID="e525d80e-1056-4e3b-9ded-485e2f174a78" containerName="cleanup" Oct 30 09:30:26.738906 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:26.738895 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e525d80e-1056-4e3b-9ded-485e2f174a78" containerName="cleanup" Oct 30 09:30:26.739005 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:26.738995 4562 cpu_manager.go:414] "containerMap: removing container" podUID="e525d80e-1056-4e3b-9ded-485e2f174a78" containerName="cleanup" Oct 30 09:30:26.739085 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:26.739074 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e525d80e-1056-4e3b-9ded-485e2f174a78" containerName="cleanup" Oct 30 09:30:26.739193 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:26.739181 4562 cpu_manager.go:414] "containerMap: removing container" podUID="57b6fb70-7cbb-46ee-ac05-be848af5349f" containerName="cleanup" Oct 30 09:30:26.739281 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:26.739269 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57b6fb70-7cbb-46ee-ac05-be848af5349f" containerName="cleanup" Oct 30 09:30:26.739576 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:26.739558 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="e525d80e-1056-4e3b-9ded-485e2f174a78" containerName="cleanup" Oct 30 09:30:26.739685 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:26.739672 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="57b6fb70-7cbb-46ee-ac05-be848af5349f" containerName="cleanup" Oct 30 09:30:26.739783 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:26.739773 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="e525d80e-1056-4e3b-9ded-485e2f174a78" containerName="cleanup" Oct 30 09:30:26.739862 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:26.739852 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="e525d80e-1056-4e3b-9ded-485e2f174a78" containerName="cleanup" Oct 30 09:30:26.739951 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:26.739941 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="57b6fb70-7cbb-46ee-ac05-be848af5349f" containerName="cleanup" Oct 30 09:30:26.743395 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:26.743361 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp" Oct 30 09:30:26.761489 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:26.761455 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4406b9a5-a7b3-4469-9760-f3dcbec872a0-workload-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp\" (UID: \"4406b9a5-a7b3-4469-9760-f3dcbec872a0\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp" Oct 30 09:30:26.761743 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:26.761721 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4406b9a5-a7b3-4469-9760-f3dcbec872a0-istio-data\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp\" (UID: \"4406b9a5-a7b3-4469-9760-f3dcbec872a0\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp" Oct 30 09:30:26.761903 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:26.761886 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4406b9a5-a7b3-4469-9760-f3dcbec872a0-istiod-ca-cert\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp\" (UID: \"4406b9a5-a7b3-4469-9760-f3dcbec872a0\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp" Oct 30 09:30:26.762071 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:26.762056 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4406b9a5-a7b3-4469-9760-f3dcbec872a0-workload-certs\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp\" (UID: \"4406b9a5-a7b3-4469-9760-f3dcbec872a0\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp" Oct 30 09:30:26.762184 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:26.762169 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjbpv\" (UniqueName: \"kubernetes.io/projected/4406b9a5-a7b3-4469-9760-f3dcbec872a0-kube-api-access-mjbpv\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp\" (UID: \"4406b9a5-a7b3-4469-9760-f3dcbec872a0\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp" Oct 30 09:30:26.762499 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:26.762477 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4406b9a5-a7b3-4469-9760-f3dcbec872a0-credential-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp\" (UID: \"4406b9a5-a7b3-4469-9760-f3dcbec872a0\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp" Oct 30 09:30:26.762589 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:26.762514 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4406b9a5-a7b3-4469-9760-f3dcbec872a0-istio-podinfo\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp\" (UID: \"4406b9a5-a7b3-4469-9760-f3dcbec872a0\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp" Oct 30 09:30:26.762589 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:26.762562 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4406b9a5-a7b3-4469-9760-f3dcbec872a0-istio-envoy\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp\" (UID: \"4406b9a5-a7b3-4469-9760-f3dcbec872a0\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp" Oct 30 09:30:26.762911 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:26.762650 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4406b9a5-a7b3-4469-9760-f3dcbec872a0-istio-token\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp\" (UID: \"4406b9a5-a7b3-4469-9760-f3dcbec872a0\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp" Oct 30 09:30:26.770834 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:26.770804 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp"] Oct 30 09:30:26.864137 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:26.864104 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4406b9a5-a7b3-4469-9760-f3dcbec872a0-workload-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp\" (UID: \"4406b9a5-a7b3-4469-9760-f3dcbec872a0\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp" Oct 30 09:30:26.864609 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:26.864207 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4406b9a5-a7b3-4469-9760-f3dcbec872a0-istio-data\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp\" (UID: \"4406b9a5-a7b3-4469-9760-f3dcbec872a0\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp" Oct 30 09:30:26.864609 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:26.864248 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4406b9a5-a7b3-4469-9760-f3dcbec872a0-istiod-ca-cert\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp\" (UID: \"4406b9a5-a7b3-4469-9760-f3dcbec872a0\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp" Oct 30 09:30:26.864609 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:26.864301 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4406b9a5-a7b3-4469-9760-f3dcbec872a0-workload-certs\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp\" (UID: \"4406b9a5-a7b3-4469-9760-f3dcbec872a0\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp" Oct 30 09:30:26.864609 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:26.864329 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjbpv\" (UniqueName: \"kubernetes.io/projected/4406b9a5-a7b3-4469-9760-f3dcbec872a0-kube-api-access-mjbpv\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp\" (UID: \"4406b9a5-a7b3-4469-9760-f3dcbec872a0\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp" Oct 30 09:30:26.864609 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:26.864451 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4406b9a5-a7b3-4469-9760-f3dcbec872a0-credential-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp\" (UID: \"4406b9a5-a7b3-4469-9760-f3dcbec872a0\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp" Oct 30 09:30:26.864609 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:26.864508 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4406b9a5-a7b3-4469-9760-f3dcbec872a0-istio-podinfo\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp\" (UID: \"4406b9a5-a7b3-4469-9760-f3dcbec872a0\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp" Oct 30 09:30:26.864609 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:26.864559 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4406b9a5-a7b3-4469-9760-f3dcbec872a0-istio-envoy\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp\" (UID: \"4406b9a5-a7b3-4469-9760-f3dcbec872a0\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp" Oct 30 09:30:26.864609 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:26.864593 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4406b9a5-a7b3-4469-9760-f3dcbec872a0-workload-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp\" (UID: \"4406b9a5-a7b3-4469-9760-f3dcbec872a0\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp" Oct 30 09:30:26.864609 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:26.864609 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4406b9a5-a7b3-4469-9760-f3dcbec872a0-istio-data\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp\" (UID: \"4406b9a5-a7b3-4469-9760-f3dcbec872a0\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp" Oct 30 09:30:26.865154 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:26.864681 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4406b9a5-a7b3-4469-9760-f3dcbec872a0-istio-token\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp\" (UID: \"4406b9a5-a7b3-4469-9760-f3dcbec872a0\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp" Oct 30 09:30:26.865154 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:26.864852 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4406b9a5-a7b3-4469-9760-f3dcbec872a0-workload-certs\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp\" (UID: \"4406b9a5-a7b3-4469-9760-f3dcbec872a0\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp" Oct 30 09:30:26.865154 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:26.865061 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4406b9a5-a7b3-4469-9760-f3dcbec872a0-istiod-ca-cert\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp\" (UID: \"4406b9a5-a7b3-4469-9760-f3dcbec872a0\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp" Oct 30 09:30:26.865154 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:26.865091 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4406b9a5-a7b3-4469-9760-f3dcbec872a0-credential-socket\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp\" (UID: \"4406b9a5-a7b3-4469-9760-f3dcbec872a0\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp" Oct 30 09:30:26.866964 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:26.866939 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4406b9a5-a7b3-4469-9760-f3dcbec872a0-istio-envoy\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp\" (UID: \"4406b9a5-a7b3-4469-9760-f3dcbec872a0\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp" Oct 30 09:30:26.867246 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:26.867229 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4406b9a5-a7b3-4469-9760-f3dcbec872a0-istio-podinfo\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp\" (UID: \"4406b9a5-a7b3-4469-9760-f3dcbec872a0\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp" Oct 30 09:30:26.886833 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:26.886804 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4406b9a5-a7b3-4469-9760-f3dcbec872a0-istio-token\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp\" (UID: \"4406b9a5-a7b3-4469-9760-f3dcbec872a0\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp" Oct 30 09:30:26.887015 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:26.886994 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjbpv\" (UniqueName: \"kubernetes.io/projected/4406b9a5-a7b3-4469-9760-f3dcbec872a0-kube-api-access-mjbpv\") pod \"e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp\" (UID: \"4406b9a5-a7b3-4469-9760-f3dcbec872a0\") " pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp" Oct 30 09:30:27.071849 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:27.071819 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp" Oct 30 09:30:27.386683 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:30:27.386655 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4406b9a5_a7b3_4469_9760_f3dcbec872a0.slice/crio-681b7c8a8dcb365c527c8db896989adf75aeace13faf939a8b0d237a0203d9a6 WatchSource:0}: Error finding container 681b7c8a8dcb365c527c8db896989adf75aeace13faf939a8b0d237a0203d9a6: Status 404 returned error can't find the container with id 681b7c8a8dcb365c527c8db896989adf75aeace13faf939a8b0d237a0203d9a6 Oct 30 09:30:27.388107 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:27.388083 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp"] Oct 30 09:30:27.389136 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:27.389104 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:30:27.389221 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:27.389167 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:30:27.389221 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:27.389196 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:30:27.640662 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:27.640554 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp" event={"ID":"4406b9a5-a7b3-4469-9760-f3dcbec872a0","Type":"ContainerStarted","Data":"f99be65fe664122bff556172bc3db3958f832b2ba284800e0b8874822ae34f9a"} Oct 30 09:30:27.640662 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:27.640623 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp" event={"ID":"4406b9a5-a7b3-4469-9760-f3dcbec872a0","Type":"ContainerStarted","Data":"681b7c8a8dcb365c527c8db896989adf75aeace13faf939a8b0d237a0203d9a6"} Oct 30 09:30:27.685897 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:27.685845 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp" podStartSLOduration=1.6858312469999999 podStartE2EDuration="1.685831247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:30:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:30:27.682879789 +0000 UTC m=+2187.944440783" watchObservedRunningTime="2026-10-30 09:30:27.685831247 +0000 UTC m=+2187.947392232" Oct 30 09:30:28.072071 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:28.072038 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp" Oct 30 09:30:28.074372 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:28.074340 4562 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Oct 30 09:30:28.074577 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:28.074432 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp" podUID="4406b9a5-a7b3-4469-9760-f3dcbec872a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Oct 30 09:30:28.598346 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:30:28.598301 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:30:29.072400 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:29.072337 4562 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Oct 30 09:30:29.072888 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:29.072423 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp" podUID="4406b9a5-a7b3-4469-9760-f3dcbec872a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Oct 30 09:30:30.072360 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:30.072322 4562 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Oct 30 09:30:30.072575 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:30.072425 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp" podUID="4406b9a5-a7b3-4469-9760-f3dcbec872a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Oct 30 09:30:31.072326 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:31.072288 4562 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Oct 30 09:30:31.072540 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:31.072345 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp" podUID="4406b9a5-a7b3-4469-9760-f3dcbec872a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Oct 30 09:30:32.075403 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:32.072643 4562 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Oct 30 09:30:32.075403 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:32.072712 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp" podUID="4406b9a5-a7b3-4469-9760-f3dcbec872a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Oct 30 09:30:33.072315 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:33.072276 4562 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Oct 30 09:30:33.072513 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:33.072351 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp" podUID="4406b9a5-a7b3-4469-9760-f3dcbec872a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Oct 30 09:30:34.072840 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:34.072808 4562 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Oct 30 09:30:34.073497 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:34.072863 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp" podUID="4406b9a5-a7b3-4469-9760-f3dcbec872a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Oct 30 09:30:35.076719 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:35.076686 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp" Oct 30 09:30:35.077188 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:35.077047 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp" Oct 30 09:30:35.078116 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:35.078090 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp" Oct 30 09:30:40.187023 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:40.186982 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2"] Oct 30 09:30:40.187979 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:40.187955 4562 cpu_manager.go:414] "containerMap: removing container" podUID="e525d80e-1056-4e3b-9ded-485e2f174a78" containerName="cleanup" Oct 30 09:30:40.187979 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:40.187976 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e525d80e-1056-4e3b-9ded-485e2f174a78" containerName="cleanup" Oct 30 09:30:40.188156 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:40.188022 4562 cpu_manager.go:414] "containerMap: removing container" podUID="57b6fb70-7cbb-46ee-ac05-be848af5349f" containerName="cleanup" Oct 30 09:30:40.188156 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:40.188028 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57b6fb70-7cbb-46ee-ac05-be848af5349f" containerName="cleanup" Oct 30 09:30:40.188156 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:40.188138 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="57b6fb70-7cbb-46ee-ac05-be848af5349f" containerName="cleanup" Oct 30 09:30:40.197249 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:40.197222 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2" Oct 30 09:30:40.218410 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:40.218356 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2"] Oct 30 09:30:40.225541 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:40.225515 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-066f95-openshift-default-dockercfg-p7w6p\"" Oct 30 09:30:40.334466 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:40.314660 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/162a9d13-c5b1-4752-91e7-3b19049a4582-istio-envoy\") pod \"e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2\" (UID: \"162a9d13-c5b1-4752-91e7-3b19049a4582\") " pod="openshift-ingress/e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2" Oct 30 09:30:40.346539 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:40.346501 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/162a9d13-c5b1-4752-91e7-3b19049a4582-istio-token\") pod \"e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2\" (UID: \"162a9d13-c5b1-4752-91e7-3b19049a4582\") " pod="openshift-ingress/e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2" Oct 30 09:30:40.346853 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:40.346830 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv9zw\" (UniqueName: \"kubernetes.io/projected/162a9d13-c5b1-4752-91e7-3b19049a4582-kube-api-access-tv9zw\") pod \"e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2\" (UID: \"162a9d13-c5b1-4752-91e7-3b19049a4582\") " pod="openshift-ingress/e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2" Oct 30 09:30:40.347021 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:40.347005 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/162a9d13-c5b1-4752-91e7-3b19049a4582-credential-socket\") pod \"e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2\" (UID: \"162a9d13-c5b1-4752-91e7-3b19049a4582\") " pod="openshift-ingress/e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2" Oct 30 09:30:40.347139 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:40.347124 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/162a9d13-c5b1-4752-91e7-3b19049a4582-istiod-ca-cert\") pod \"e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2\" (UID: \"162a9d13-c5b1-4752-91e7-3b19049a4582\") " pod="openshift-ingress/e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2" Oct 30 09:30:40.354450 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:40.347253 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/162a9d13-c5b1-4752-91e7-3b19049a4582-istio-data\") pod \"e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2\" (UID: \"162a9d13-c5b1-4752-91e7-3b19049a4582\") " pod="openshift-ingress/e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2" Oct 30 09:30:40.354691 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:40.354669 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/162a9d13-c5b1-4752-91e7-3b19049a4582-workload-certs\") pod \"e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2\" (UID: \"162a9d13-c5b1-4752-91e7-3b19049a4582\") " pod="openshift-ingress/e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2" Oct 30 09:30:40.369401 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:40.361714 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/162a9d13-c5b1-4752-91e7-3b19049a4582-workload-socket\") pod \"e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2\" (UID: \"162a9d13-c5b1-4752-91e7-3b19049a4582\") " pod="openshift-ingress/e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2" Oct 30 09:30:40.369401 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:40.361772 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/162a9d13-c5b1-4752-91e7-3b19049a4582-istio-podinfo\") pod \"e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2\" (UID: \"162a9d13-c5b1-4752-91e7-3b19049a4582\") " pod="openshift-ingress/e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2" Oct 30 09:30:40.470406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:40.465292 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/162a9d13-c5b1-4752-91e7-3b19049a4582-workload-socket\") pod \"e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2\" (UID: \"162a9d13-c5b1-4752-91e7-3b19049a4582\") " pod="openshift-ingress/e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2" Oct 30 09:30:40.470406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:40.465363 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/162a9d13-c5b1-4752-91e7-3b19049a4582-istio-podinfo\") pod \"e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2\" (UID: \"162a9d13-c5b1-4752-91e7-3b19049a4582\") " pod="openshift-ingress/e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2" Oct 30 09:30:40.470406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:40.465512 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/162a9d13-c5b1-4752-91e7-3b19049a4582-istio-envoy\") pod \"e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2\" (UID: \"162a9d13-c5b1-4752-91e7-3b19049a4582\") " pod="openshift-ingress/e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2" Oct 30 09:30:40.470406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:40.465566 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/162a9d13-c5b1-4752-91e7-3b19049a4582-istio-token\") pod \"e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2\" (UID: \"162a9d13-c5b1-4752-91e7-3b19049a4582\") " pod="openshift-ingress/e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2" Oct 30 09:30:40.470406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:40.465635 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv9zw\" (UniqueName: \"kubernetes.io/projected/162a9d13-c5b1-4752-91e7-3b19049a4582-kube-api-access-tv9zw\") pod \"e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2\" (UID: \"162a9d13-c5b1-4752-91e7-3b19049a4582\") " pod="openshift-ingress/e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2" Oct 30 09:30:40.470406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:40.465673 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/162a9d13-c5b1-4752-91e7-3b19049a4582-credential-socket\") pod \"e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2\" (UID: \"162a9d13-c5b1-4752-91e7-3b19049a4582\") " pod="openshift-ingress/e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2" Oct 30 09:30:40.470406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:40.465699 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/162a9d13-c5b1-4752-91e7-3b19049a4582-istiod-ca-cert\") pod \"e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2\" (UID: \"162a9d13-c5b1-4752-91e7-3b19049a4582\") " pod="openshift-ingress/e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2" Oct 30 09:30:40.470406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:40.465734 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/162a9d13-c5b1-4752-91e7-3b19049a4582-istio-data\") pod \"e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2\" (UID: \"162a9d13-c5b1-4752-91e7-3b19049a4582\") " pod="openshift-ingress/e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2" Oct 30 09:30:40.470406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:40.465781 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/162a9d13-c5b1-4752-91e7-3b19049a4582-workload-certs\") pod \"e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2\" (UID: \"162a9d13-c5b1-4752-91e7-3b19049a4582\") " pod="openshift-ingress/e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2" Oct 30 09:30:40.470406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:40.466358 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/162a9d13-c5b1-4752-91e7-3b19049a4582-workload-certs\") pod \"e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2\" (UID: \"162a9d13-c5b1-4752-91e7-3b19049a4582\") " pod="openshift-ingress/e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2" Oct 30 09:30:40.470406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:40.466638 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/162a9d13-c5b1-4752-91e7-3b19049a4582-workload-socket\") pod \"e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2\" (UID: \"162a9d13-c5b1-4752-91e7-3b19049a4582\") " pod="openshift-ingress/e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2" Oct 30 09:30:40.472171 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:40.472150 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/162a9d13-c5b1-4752-91e7-3b19049a4582-istiod-ca-cert\") pod \"e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2\" (UID: \"162a9d13-c5b1-4752-91e7-3b19049a4582\") " pod="openshift-ingress/e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2" Oct 30 09:30:40.473027 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:40.472751 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/162a9d13-c5b1-4752-91e7-3b19049a4582-credential-socket\") pod \"e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2\" (UID: \"162a9d13-c5b1-4752-91e7-3b19049a4582\") " pod="openshift-ingress/e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2" Oct 30 09:30:40.473187 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:40.472948 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/162a9d13-c5b1-4752-91e7-3b19049a4582-istio-data\") pod \"e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2\" (UID: \"162a9d13-c5b1-4752-91e7-3b19049a4582\") " pod="openshift-ingress/e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2" Oct 30 09:30:40.485079 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:40.483966 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/162a9d13-c5b1-4752-91e7-3b19049a4582-istio-podinfo\") pod \"e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2\" (UID: \"162a9d13-c5b1-4752-91e7-3b19049a4582\") " pod="openshift-ingress/e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2" Oct 30 09:30:40.485209 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:40.485134 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/162a9d13-c5b1-4752-91e7-3b19049a4582-istio-envoy\") pod \"e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2\" (UID: \"162a9d13-c5b1-4752-91e7-3b19049a4582\") " pod="openshift-ingress/e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2" Oct 30 09:30:40.493918 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:40.493819 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv9zw\" (UniqueName: \"kubernetes.io/projected/162a9d13-c5b1-4752-91e7-3b19049a4582-kube-api-access-tv9zw\") pod \"e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2\" (UID: \"162a9d13-c5b1-4752-91e7-3b19049a4582\") " pod="openshift-ingress/e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2" Oct 30 09:30:40.495463 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:40.494104 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/162a9d13-c5b1-4752-91e7-3b19049a4582-istio-token\") pod \"e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2\" (UID: \"162a9d13-c5b1-4752-91e7-3b19049a4582\") " pod="openshift-ingress/e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2" Oct 30 09:30:40.522207 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:40.522180 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2" Oct 30 09:30:40.885564 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:30:40.885536 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod162a9d13_c5b1_4752_91e7_3b19049a4582.slice/crio-2fd1179b892bbee0b5cabc9ecab5c7447cd5bcc624eba39c7405bdfc9514e32e WatchSource:0}: Error finding container 2fd1179b892bbee0b5cabc9ecab5c7447cd5bcc624eba39c7405bdfc9514e32e: Status 404 returned error can't find the container with id 2fd1179b892bbee0b5cabc9ecab5c7447cd5bcc624eba39c7405bdfc9514e32e Oct 30 09:30:40.886860 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:40.886834 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2"] Oct 30 09:30:40.888213 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:40.888177 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:30:40.888316 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:40.888248 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:30:40.888316 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:40.888283 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:30:41.847285 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:41.847250 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2" event={"ID":"162a9d13-c5b1-4752-91e7-3b19049a4582","Type":"ContainerStarted","Data":"c32877b05aef6dfc4183c5146c9c45c43715d14a9be1b5ade2b2746c3980f2b2"} Oct 30 09:30:41.847716 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:41.847302 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2" event={"ID":"162a9d13-c5b1-4752-91e7-3b19049a4582","Type":"ContainerStarted","Data":"2fd1179b892bbee0b5cabc9ecab5c7447cd5bcc624eba39c7405bdfc9514e32e"} Oct 30 09:30:41.918491 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:41.918436 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2" podStartSLOduration=1.9184181850000002 podStartE2EDuration="1.918418185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:30:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:30:41.902529337 +0000 UTC m=+2202.164090325" watchObservedRunningTime="2026-10-30 09:30:41.918418185 +0000 UTC m=+2202.179979183" Oct 30 09:30:42.535425 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:42.528783 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2" Oct 30 09:30:42.564225 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:42.564196 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2" Oct 30 09:30:42.854167 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:42.854092 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2" Oct 30 09:30:42.855176 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:42.855158 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2" Oct 30 09:30:54.524492 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:54.524341 4562 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-176b6c/distinct-w3-176b6c-kserve-f57b7579c-k4pc5" podUID="28fd4072-399c-4be5-a0d8-1be6966da4af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": context deadline exceeded" Oct 30 09:30:54.642407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:54.639472 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-d87c85/ipp-route-d87c85-kserve-7cc5cb6f7c-qk4q7" podUID="9bfc138b-8938-473c-adfe-052d9a722233" containerName="main" probeResult="failure" output="Get \"https://10.133.0.168:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:30:54.847466 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:54.844479 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-176b6c/distinct-w3-176b6c-kserve-f57b7579c-k4pc5" podUID="28fd4072-399c-4be5-a0d8-1be6966da4af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:30:54.851406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:54.850460 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55" podUID="88be34c3-a9e9-4489-a449-206a99ef1663" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:30:55.016406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:55.015473 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c5447d/distinct-2-w1-c5447d-kserve-676749d548-xlmsf" podUID="c9ff897f-c520-4d0f-8820-e185b1d47dac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:30:55.073405 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:30:55.073188 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb" podUID="a540c86f-c856-42a8-9f73-b5b9b690f42b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:30:58.596136 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:30:58.596094 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:31:04.046695 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:04.046663 4562 generic.go:356] "Generic (PLEG): container finished" podID="f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64" containerID="399e769bcd0cff3a38573e1a9c7d015ea6a1098c2ab43b44cb47e566bed33985" exitCode=0 Oct 30 09:31:04.047144 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:04.046738 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj" event={"ID":"f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64","Type":"ContainerDied","Data":"399e769bcd0cff3a38573e1a9c7d015ea6a1098c2ab43b44cb47e566bed33985"} Oct 30 09:31:04.047229 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:04.047214 4562 scope.go:122] "RemoveContainer" containerID="399e769bcd0cff3a38573e1a9c7d015ea6a1098c2ab43b44cb47e566bed33985" Oct 30 09:31:04.048857 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:04.048829 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:31:04.048940 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:04.048900 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:31:04.048940 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:04.048934 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:31:05.077457 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:05.075409 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj" event={"ID":"f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64","Type":"ContainerStarted","Data":"5fef09b987b0725a65b240bee63a0d4a5825d8bb44ca482440308c8db0f8b0f9"} Oct 30 09:31:05.651281 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:05.650791 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj" Oct 30 09:31:05.651281 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:05.650927 4562 patch_prober.go:29] interesting pod/e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 30 09:31:05.651281 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:05.651013 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj" podUID="f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 30 09:31:06.661408 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:06.660370 4562 patch_prober.go:29] interesting pod/e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 30 09:31:06.661408 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:06.660463 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj" podUID="f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 30 09:31:07.657408 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:07.655756 4562 patch_prober.go:29] interesting pod/e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 30 09:31:07.657408 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:07.655817 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj" podUID="f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 30 09:31:08.660410 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:08.656119 4562 patch_prober.go:29] interesting pod/e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 30 09:31:08.660410 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:08.656183 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj" podUID="f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 30 09:31:09.651943 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:09.651906 4562 patch_prober.go:29] interesting pod/e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 30 09:31:09.652236 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:09.652204 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj" podUID="f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 30 09:31:10.655167 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:10.655139 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj" Oct 30 09:31:10.655639 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:10.655619 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj" Oct 30 09:31:10.656244 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:10.656228 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj" Oct 30 09:31:17.267431 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:17.266358 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c"] Oct 30 09:31:17.267431 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:17.266736 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c" podUID="028fea98-1060-44f3-887a-bf0a119b428b" containerName="istio-proxy" containerID="cri-o://9ac99bca87d8ea2f341ef3d64bbad45ce02d15c7aa1ed8209c238257a3e2f55f" gracePeriod=30 Oct 30 09:31:22.405982 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:22.405938 4562 generic.go:356] "Generic (PLEG): container finished" podID="028fea98-1060-44f3-887a-bf0a119b428b" containerID="9ac99bca87d8ea2f341ef3d64bbad45ce02d15c7aa1ed8209c238257a3e2f55f" exitCode=0 Oct 30 09:31:22.406477 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:22.406415 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c" event={"ID":"028fea98-1060-44f3-887a-bf0a119b428b","Type":"ContainerDied","Data":"9ac99bca87d8ea2f341ef3d64bbad45ce02d15c7aa1ed8209c238257a3e2f55f"} Oct 30 09:31:22.415163 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:31:22.415119 4562 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod028fea98_1060_44f3_887a_bf0a119b428b.slice/crio-9ac99bca87d8ea2f341ef3d64bbad45ce02d15c7aa1ed8209c238257a3e2f55f.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod028fea98_1060_44f3_887a_bf0a119b428b.slice/crio-conmon-9ac99bca87d8ea2f341ef3d64bbad45ce02d15c7aa1ed8209c238257a3e2f55f.scope\": RecentStats: unable to find data in memory cache]" Oct 30 09:31:23.113014 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:23.112988 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c" Oct 30 09:31:23.293951 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:23.293915 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/028fea98-1060-44f3-887a-bf0a119b428b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/028fea98-1060-44f3-887a-bf0a119b428b-credential-socket\") pod \"028fea98-1060-44f3-887a-bf0a119b428b\" (UID: \"028fea98-1060-44f3-887a-bf0a119b428b\") " Oct 30 09:31:23.294154 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:23.294024 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/028fea98-1060-44f3-887a-bf0a119b428b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/028fea98-1060-44f3-887a-bf0a119b428b-istiod-ca-cert\") pod \"028fea98-1060-44f3-887a-bf0a119b428b\" (UID: \"028fea98-1060-44f3-887a-bf0a119b428b\") " Oct 30 09:31:23.294154 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:23.294081 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/028fea98-1060-44f3-887a-bf0a119b428b-kube-api-access-p57pv\" (UniqueName: \"kubernetes.io/projected/028fea98-1060-44f3-887a-bf0a119b428b-kube-api-access-p57pv\") pod \"028fea98-1060-44f3-887a-bf0a119b428b\" (UID: \"028fea98-1060-44f3-887a-bf0a119b428b\") " Oct 30 09:31:23.294295 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:23.294156 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/028fea98-1060-44f3-887a-bf0a119b428b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/028fea98-1060-44f3-887a-bf0a119b428b-istio-podinfo\") pod \"028fea98-1060-44f3-887a-bf0a119b428b\" (UID: \"028fea98-1060-44f3-887a-bf0a119b428b\") " Oct 30 09:31:23.294295 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:23.294184 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/028fea98-1060-44f3-887a-bf0a119b428b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/028fea98-1060-44f3-887a-bf0a119b428b-istio-envoy\") pod \"028fea98-1060-44f3-887a-bf0a119b428b\" (UID: \"028fea98-1060-44f3-887a-bf0a119b428b\") " Oct 30 09:31:23.294295 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:23.294210 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/028fea98-1060-44f3-887a-bf0a119b428b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/028fea98-1060-44f3-887a-bf0a119b428b-workload-socket\") pod \"028fea98-1060-44f3-887a-bf0a119b428b\" (UID: \"028fea98-1060-44f3-887a-bf0a119b428b\") " Oct 30 09:31:23.294295 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:23.294271 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/028fea98-1060-44f3-887a-bf0a119b428b-credential-socket" pod "028fea98-1060-44f3-887a-bf0a119b428b" (UID: "028fea98-1060-44f3-887a-bf0a119b428b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:31:23.294535 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:23.294298 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/028fea98-1060-44f3-887a-bf0a119b428b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/028fea98-1060-44f3-887a-bf0a119b428b-istio-data\") pod \"028fea98-1060-44f3-887a-bf0a119b428b\" (UID: \"028fea98-1060-44f3-887a-bf0a119b428b\") " Oct 30 09:31:23.294535 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:23.294345 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/028fea98-1060-44f3-887a-bf0a119b428b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/028fea98-1060-44f3-887a-bf0a119b428b-workload-certs\") pod \"028fea98-1060-44f3-887a-bf0a119b428b\" (UID: \"028fea98-1060-44f3-887a-bf0a119b428b\") " Oct 30 09:31:23.294535 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:23.294464 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/028fea98-1060-44f3-887a-bf0a119b428b-istio-token\" (UniqueName: \"kubernetes.io/projected/028fea98-1060-44f3-887a-bf0a119b428b-istio-token\") pod \"028fea98-1060-44f3-887a-bf0a119b428b\" (UID: \"028fea98-1060-44f3-887a-bf0a119b428b\") " Oct 30 09:31:23.294683 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:23.294532 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/028fea98-1060-44f3-887a-bf0a119b428b-istiod-ca-cert" pod "028fea98-1060-44f3-887a-bf0a119b428b" (UID: "028fea98-1060-44f3-887a-bf0a119b428b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:31:23.294737 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:23.294699 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/028fea98-1060-44f3-887a-bf0a119b428b-workload-certs" pod "028fea98-1060-44f3-887a-bf0a119b428b" (UID: "028fea98-1060-44f3-887a-bf0a119b428b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:31:23.294842 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:23.294738 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/028fea98-1060-44f3-887a-bf0a119b428b-istio-data" pod "028fea98-1060-44f3-887a-bf0a119b428b" (UID: "028fea98-1060-44f3-887a-bf0a119b428b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:31:23.294980 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:23.294960 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/028fea98-1060-44f3-887a-bf0a119b428b-workload-socket" pod "028fea98-1060-44f3-887a-bf0a119b428b" (UID: "028fea98-1060-44f3-887a-bf0a119b428b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:31:23.295148 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:23.295134 4562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/028fea98-1060-44f3-887a-bf0a119b428b-istiod-ca-cert\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:31:23.295220 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:23.295151 4562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/028fea98-1060-44f3-887a-bf0a119b428b-workload-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:31:23.295220 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:23.295165 4562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/028fea98-1060-44f3-887a-bf0a119b428b-istio-data\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:31:23.295220 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:23.295179 4562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/028fea98-1060-44f3-887a-bf0a119b428b-workload-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:31:23.295220 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:23.295190 4562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/028fea98-1060-44f3-887a-bf0a119b428b-credential-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:31:23.296584 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:23.296556 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/028fea98-1060-44f3-887a-bf0a119b428b-kube-api-access-p57pv" pod "028fea98-1060-44f3-887a-bf0a119b428b" (UID: "028fea98-1060-44f3-887a-bf0a119b428b"). InnerVolumeSpecName "kube-api-access-p57pv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:31:23.296710 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:23.296660 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/028fea98-1060-44f3-887a-bf0a119b428b-istio-podinfo" pod "028fea98-1060-44f3-887a-bf0a119b428b" (UID: "028fea98-1060-44f3-887a-bf0a119b428b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:31:23.296855 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:23.296836 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/028fea98-1060-44f3-887a-bf0a119b428b-istio-envoy" pod "028fea98-1060-44f3-887a-bf0a119b428b" (UID: "028fea98-1060-44f3-887a-bf0a119b428b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:31:23.296938 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:23.296918 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/028fea98-1060-44f3-887a-bf0a119b428b-istio-token" pod "028fea98-1060-44f3-887a-bf0a119b428b" (UID: "028fea98-1060-44f3-887a-bf0a119b428b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:31:23.396495 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:23.396410 4562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/028fea98-1060-44f3-887a-bf0a119b428b-istio-podinfo\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:31:23.396495 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:23.396457 4562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/028fea98-1060-44f3-887a-bf0a119b428b-istio-envoy\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:31:23.396495 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:23.396471 4562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/028fea98-1060-44f3-887a-bf0a119b428b-istio-token\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:31:23.396495 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:23.396486 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p57pv\" (UniqueName: \"kubernetes.io/projected/028fea98-1060-44f3-887a-bf0a119b428b-kube-api-access-p57pv\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:31:23.416635 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:23.416600 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c" Oct 30 09:31:23.417076 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:23.416633 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c" event={"ID":"028fea98-1060-44f3-887a-bf0a119b428b","Type":"ContainerDied","Data":"c45862428c577580f2789dadbdc2c2df3857ccf9a14ca10cd2313735be8a4779"} Oct 30 09:31:23.417076 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:23.416698 4562 scope.go:122] "RemoveContainer" containerID="9ac99bca87d8ea2f341ef3d64bbad45ce02d15c7aa1ed8209c238257a3e2f55f" Oct 30 09:31:23.472699 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:23.472661 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c"] Oct 30 09:31:23.485984 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:23.485954 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-c9232f-openshift-default-85dfbb8d97-w9v6c"] Oct 30 09:31:23.875512 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:23.875469 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="028fea98-1060-44f3-887a-bf0a119b428b" path="/var/lib/kubelet/pods/028fea98-1060-44f3-887a-bf0a119b428b/volumes" Oct 30 09:31:27.448820 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:27.448770 4562 generic.go:356] "Generic (PLEG): container finished" podID="162a9d13-c5b1-4752-91e7-3b19049a4582" containerID="c32877b05aef6dfc4183c5146c9c45c43715d14a9be1b5ade2b2746c3980f2b2" exitCode=0 Oct 30 09:31:27.449329 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:27.448838 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2" event={"ID":"162a9d13-c5b1-4752-91e7-3b19049a4582","Type":"ContainerDied","Data":"c32877b05aef6dfc4183c5146c9c45c43715d14a9be1b5ade2b2746c3980f2b2"} Oct 30 09:31:27.449329 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:27.449252 4562 scope.go:122] "RemoveContainer" containerID="c32877b05aef6dfc4183c5146c9c45c43715d14a9be1b5ade2b2746c3980f2b2" Oct 30 09:31:27.450872 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:27.450846 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:31:27.450946 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:27.450913 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:31:27.450984 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:27.450944 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:31:27.854297 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:27.854209 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2" Oct 30 09:31:28.472365 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:28.472324 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2" event={"ID":"162a9d13-c5b1-4752-91e7-3b19049a4582","Type":"ContainerStarted","Data":"e3750ac93490e3484e232029cea39ffa2e3a02ac137b43d723cdbdbae320ca0d"} Oct 30 09:31:28.522857 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:28.522814 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2" Oct 30 09:31:28.524003 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:28.523977 4562 patch_prober.go:29] interesting pod/e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Oct 30 09:31:28.524100 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:28.524046 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2" podUID="162a9d13-c5b1-4752-91e7-3b19049a4582" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Oct 30 09:31:28.598015 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:31:28.597971 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:31:29.522648 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:29.522616 4562 patch_prober.go:29] interesting pod/e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Oct 30 09:31:29.523035 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:29.522668 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2" podUID="162a9d13-c5b1-4752-91e7-3b19049a4582" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Oct 30 09:31:30.522991 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:30.522952 4562 patch_prober.go:29] interesting pod/e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Oct 30 09:31:30.523493 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:30.523020 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2" podUID="162a9d13-c5b1-4752-91e7-3b19049a4582" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Oct 30 09:31:31.523026 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:31.522988 4562 patch_prober.go:29] interesting pod/e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Oct 30 09:31:31.523433 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:31.523059 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2" podUID="162a9d13-c5b1-4752-91e7-3b19049a4582" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Oct 30 09:31:32.478222 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:32.478180 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-d95d35-88b87698-5msxx"] Oct 30 09:31:32.523177 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:32.523140 4562 patch_prober.go:29] interesting pod/e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Oct 30 09:31:32.523689 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:32.523197 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2" podUID="162a9d13-c5b1-4752-91e7-3b19049a4582" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Oct 30 09:31:32.524182 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:32.524159 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-d95d35-f67b6f879-pdc6f"] Oct 30 09:31:33.620669 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:33.620632 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2" Oct 30 09:31:33.621122 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:33.620960 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2" Oct 30 09:31:33.622310 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:33.622287 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2" Oct 30 09:31:37.479453 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:37.479362 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-d95d35-88b87698-5msxx" podUID="49e0fd93-7200-46ea-bcf7-28d1df31d16f" containerName="payload-processing" containerID="cri-o://fd9bec4f6b5de7778bbf8df83e5fe7925f4fa7a47539cf14708f5fd81eae04c4" gracePeriod=25 Oct 30 09:31:37.524577 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:37.524526 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d95d35-f67b6f879-pdc6f" podUID="14458dfc-296c-47f3-adb5-1b28e26b4314" containerName="payload-pre-processing" containerID="cri-o://5d37a4051953b1894154c20beaa97193242252cae2df88fa2fe95cf2bdd5277a" gracePeriod=25 Oct 30 09:31:37.814574 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:37.814548 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-d95d35-88b87698-5msxx" Oct 30 09:31:37.850125 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:37.850096 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d95d35-f67b6f879-pdc6f" Oct 30 09:31:37.931595 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:37.931561 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49e0fd93-7200-46ea-bcf7-28d1df31d16f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49e0fd93-7200-46ea-bcf7-28d1df31d16f-plugins-config-volume\") pod \"49e0fd93-7200-46ea-bcf7-28d1df31d16f\" (UID: \"49e0fd93-7200-46ea-bcf7-28d1df31d16f\") " Oct 30 09:31:37.931774 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:37.931620 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14458dfc-296c-47f3-adb5-1b28e26b4314-kube-api-access-n57ll\" (UniqueName: \"kubernetes.io/projected/14458dfc-296c-47f3-adb5-1b28e26b4314-kube-api-access-n57ll\") pod \"14458dfc-296c-47f3-adb5-1b28e26b4314\" (UID: \"14458dfc-296c-47f3-adb5-1b28e26b4314\") " Oct 30 09:31:37.931849 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:37.931804 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49e0fd93-7200-46ea-bcf7-28d1df31d16f-kube-api-access-wccwh\" (UniqueName: \"kubernetes.io/projected/49e0fd93-7200-46ea-bcf7-28d1df31d16f-kube-api-access-wccwh\") pod \"49e0fd93-7200-46ea-bcf7-28d1df31d16f\" (UID: \"49e0fd93-7200-46ea-bcf7-28d1df31d16f\") " Oct 30 09:31:37.931949 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:37.931932 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14458dfc-296c-47f3-adb5-1b28e26b4314-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14458dfc-296c-47f3-adb5-1b28e26b4314-plugins-config-volume\") pod \"14458dfc-296c-47f3-adb5-1b28e26b4314\" (UID: \"14458dfc-296c-47f3-adb5-1b28e26b4314\") " Oct 30 09:31:37.932025 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:37.931961 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49e0fd93-7200-46ea-bcf7-28d1df31d16f-plugins-config-volume" pod "49e0fd93-7200-46ea-bcf7-28d1df31d16f" (UID: "49e0fd93-7200-46ea-bcf7-28d1df31d16f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:31:37.932366 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:37.932339 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14458dfc-296c-47f3-adb5-1b28e26b4314-plugins-config-volume" pod "14458dfc-296c-47f3-adb5-1b28e26b4314" (UID: "14458dfc-296c-47f3-adb5-1b28e26b4314"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:31:37.932552 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:37.932534 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49e0fd93-7200-46ea-bcf7-28d1df31d16f-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:31:37.932612 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:37.932557 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14458dfc-296c-47f3-adb5-1b28e26b4314-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:31:37.933947 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:37.933922 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49e0fd93-7200-46ea-bcf7-28d1df31d16f-kube-api-access-wccwh" pod "49e0fd93-7200-46ea-bcf7-28d1df31d16f" (UID: "49e0fd93-7200-46ea-bcf7-28d1df31d16f"). InnerVolumeSpecName "kube-api-access-wccwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:31:37.934049 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:37.933956 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14458dfc-296c-47f3-adb5-1b28e26b4314-kube-api-access-n57ll" pod "14458dfc-296c-47f3-adb5-1b28e26b4314" (UID: "14458dfc-296c-47f3-adb5-1b28e26b4314"). InnerVolumeSpecName "kube-api-access-n57ll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:31:38.034031 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:38.033958 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n57ll\" (UniqueName: \"kubernetes.io/projected/14458dfc-296c-47f3-adb5-1b28e26b4314-kube-api-access-n57ll\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:31:38.034031 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:38.033984 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wccwh\" (UniqueName: \"kubernetes.io/projected/49e0fd93-7200-46ea-bcf7-28d1df31d16f-kube-api-access-wccwh\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:31:38.536638 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:38.536604 4562 generic.go:356] "Generic (PLEG): container finished" podID="14458dfc-296c-47f3-adb5-1b28e26b4314" containerID="5d37a4051953b1894154c20beaa97193242252cae2df88fa2fe95cf2bdd5277a" exitCode=0 Oct 30 09:31:38.537059 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:38.536654 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d95d35-f67b6f879-pdc6f" Oct 30 09:31:38.537059 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:38.536692 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d95d35-f67b6f879-pdc6f" event={"ID":"14458dfc-296c-47f3-adb5-1b28e26b4314","Type":"ContainerDied","Data":"5d37a4051953b1894154c20beaa97193242252cae2df88fa2fe95cf2bdd5277a"} Oct 30 09:31:38.537059 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:38.536750 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d95d35-f67b6f879-pdc6f" event={"ID":"14458dfc-296c-47f3-adb5-1b28e26b4314","Type":"ContainerDied","Data":"9c0c8070ead25476b8d5095fab9e87ce179d317e983c65906b23e1a29aed1a87"} Oct 30 09:31:38.537059 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:38.536773 4562 scope.go:122] "RemoveContainer" containerID="5d37a4051953b1894154c20beaa97193242252cae2df88fa2fe95cf2bdd5277a" Oct 30 09:31:38.538310 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:38.538290 4562 generic.go:356] "Generic (PLEG): container finished" podID="49e0fd93-7200-46ea-bcf7-28d1df31d16f" containerID="fd9bec4f6b5de7778bbf8df83e5fe7925f4fa7a47539cf14708f5fd81eae04c4" exitCode=0 Oct 30 09:31:38.538416 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:38.538322 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-d95d35-88b87698-5msxx" event={"ID":"49e0fd93-7200-46ea-bcf7-28d1df31d16f","Type":"ContainerDied","Data":"fd9bec4f6b5de7778bbf8df83e5fe7925f4fa7a47539cf14708f5fd81eae04c4"} Oct 30 09:31:38.538416 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:38.538356 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-d95d35-88b87698-5msxx" Oct 30 09:31:38.538416 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:38.538358 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-d95d35-88b87698-5msxx" event={"ID":"49e0fd93-7200-46ea-bcf7-28d1df31d16f","Type":"ContainerDied","Data":"fa4929553356dc966c383fcade4e080065b1fab98157e606907901c02af0601e"} Oct 30 09:31:38.548324 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:38.548306 4562 scope.go:122] "RemoveContainer" containerID="5d37a4051953b1894154c20beaa97193242252cae2df88fa2fe95cf2bdd5277a" Oct 30 09:31:38.549111 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:31:38.549079 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d37a4051953b1894154c20beaa97193242252cae2df88fa2fe95cf2bdd5277a\": container with ID starting with 5d37a4051953b1894154c20beaa97193242252cae2df88fa2fe95cf2bdd5277a not found: ID does not exist" containerID="5d37a4051953b1894154c20beaa97193242252cae2df88fa2fe95cf2bdd5277a" Oct 30 09:31:38.549209 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:38.549107 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d37a4051953b1894154c20beaa97193242252cae2df88fa2fe95cf2bdd5277a"} err="failed to get container status \"5d37a4051953b1894154c20beaa97193242252cae2df88fa2fe95cf2bdd5277a\": rpc error: code = NotFound desc = could not find container \"5d37a4051953b1894154c20beaa97193242252cae2df88fa2fe95cf2bdd5277a\": container with ID starting with 5d37a4051953b1894154c20beaa97193242252cae2df88fa2fe95cf2bdd5277a not found: ID does not exist" Oct 30 09:31:38.549209 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:38.549129 4562 scope.go:122] "RemoveContainer" containerID="fd9bec4f6b5de7778bbf8df83e5fe7925f4fa7a47539cf14708f5fd81eae04c4" Oct 30 09:31:38.608688 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:38.608555 4562 scope.go:122] "RemoveContainer" containerID="fd9bec4f6b5de7778bbf8df83e5fe7925f4fa7a47539cf14708f5fd81eae04c4" Oct 30 09:31:38.619873 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:38.619833 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-d95d35-88b87698-5msxx"] Oct 30 09:31:38.626007 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:38.624557 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-d95d35-88b87698-5msxx"] Oct 30 09:31:38.626007 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:31:38.625832 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd9bec4f6b5de7778bbf8df83e5fe7925f4fa7a47539cf14708f5fd81eae04c4\": container with ID starting with fd9bec4f6b5de7778bbf8df83e5fe7925f4fa7a47539cf14708f5fd81eae04c4 not found: ID does not exist" containerID="fd9bec4f6b5de7778bbf8df83e5fe7925f4fa7a47539cf14708f5fd81eae04c4" Oct 30 09:31:38.626007 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:38.625875 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd9bec4f6b5de7778bbf8df83e5fe7925f4fa7a47539cf14708f5fd81eae04c4"} err="failed to get container status \"fd9bec4f6b5de7778bbf8df83e5fe7925f4fa7a47539cf14708f5fd81eae04c4\": rpc error: code = NotFound desc = could not find container \"fd9bec4f6b5de7778bbf8df83e5fe7925f4fa7a47539cf14708f5fd81eae04c4\": container with ID starting with fd9bec4f6b5de7778bbf8df83e5fe7925f4fa7a47539cf14708f5fd81eae04c4 not found: ID does not exist" Oct 30 09:31:38.659342 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:38.659315 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-d95d35-f67b6f879-pdc6f"] Oct 30 09:31:38.668065 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:38.668041 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-d95d35-f67b6f879-pdc6f"] Oct 30 09:31:39.875173 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:39.875139 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14458dfc-296c-47f3-adb5-1b28e26b4314" path="/var/lib/kubelet/pods/14458dfc-296c-47f3-adb5-1b28e26b4314/volumes" Oct 30 09:31:39.875576 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:39.875561 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49e0fd93-7200-46ea-bcf7-28d1df31d16f" path="/var/lib/kubelet/pods/49e0fd93-7200-46ea-bcf7-28d1df31d16f/volumes" Oct 30 09:31:42.655406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:42.654652 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-d87c85/ipp-route-d87c85-kserve-7cc5cb6f7c-qk4q7" podUID="9bfc138b-8938-473c-adfe-052d9a722233" containerName="main" probeResult="failure" output="Get \"https://10.133.0.168:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:31:42.820407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:42.817633 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-176b6c/distinct-w3-176b6c-kserve-f57b7579c-k4pc5" podUID="28fd4072-399c-4be5-a0d8-1be6966da4af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:31:42.858406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:42.857510 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55" podUID="88be34c3-a9e9-4489-a449-206a99ef1663" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:31:43.076406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:43.073400 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb" podUID="a540c86f-c856-42a8-9f73-b5b9b690f42b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Oct 30 09:31:43.145406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:43.141452 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-eff9fd-6b784dd94d-lrzsf"] Oct 30 09:31:43.169414 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:43.168563 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-eff9fd-847fb7448c-5n9rh"] Oct 30 09:31:44.946683 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:44.946650 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-d95d35-799494fd9b-rxn8v"] Oct 30 09:31:44.947256 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:44.946914 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d95d35-799494fd9b-rxn8v" podUID="d2a58d8b-b84a-403f-bd46-5dfe7d3cadec" containerName="maas-api" containerID="cri-o://6781ba2699c5a77587074d740f1f42847f81c217a42803e1c63b10b4e8890a7e" gracePeriod=30 Oct 30 09:31:45.332407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:45.326681 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d95d35-799494fd9b-rxn8v" Oct 30 09:31:45.425284 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:45.412863 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2a58d8b-b84a-403f-bd46-5dfe7d3cadec-kube-api-access-n78fl\" (UniqueName: \"kubernetes.io/projected/d2a58d8b-b84a-403f-bd46-5dfe7d3cadec-kube-api-access-n78fl\") pod \"d2a58d8b-b84a-403f-bd46-5dfe7d3cadec\" (UID: \"d2a58d8b-b84a-403f-bd46-5dfe7d3cadec\") " Oct 30 09:31:45.425284 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:45.413017 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2a58d8b-b84a-403f-bd46-5dfe7d3cadec-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2a58d8b-b84a-403f-bd46-5dfe7d3cadec-openshift-service-ca\") pod \"d2a58d8b-b84a-403f-bd46-5dfe7d3cadec\" (UID: \"d2a58d8b-b84a-403f-bd46-5dfe7d3cadec\") " Oct 30 09:31:45.425284 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:45.413125 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2a58d8b-b84a-403f-bd46-5dfe7d3cadec-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2a58d8b-b84a-403f-bd46-5dfe7d3cadec-metrics-certs\") pod \"d2a58d8b-b84a-403f-bd46-5dfe7d3cadec\" (UID: \"d2a58d8b-b84a-403f-bd46-5dfe7d3cadec\") " Oct 30 09:31:45.425284 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:45.413248 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2a58d8b-b84a-403f-bd46-5dfe7d3cadec-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2a58d8b-b84a-403f-bd46-5dfe7d3cadec-maas-api-tls\") pod \"d2a58d8b-b84a-403f-bd46-5dfe7d3cadec\" (UID: \"d2a58d8b-b84a-403f-bd46-5dfe7d3cadec\") " Oct 30 09:31:45.452789 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:45.450736 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2a58d8b-b84a-403f-bd46-5dfe7d3cadec-maas-api-tls" pod "d2a58d8b-b84a-403f-bd46-5dfe7d3cadec" (UID: "d2a58d8b-b84a-403f-bd46-5dfe7d3cadec"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:31:45.471794 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:45.470436 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2a58d8b-b84a-403f-bd46-5dfe7d3cadec-openshift-service-ca" pod "d2a58d8b-b84a-403f-bd46-5dfe7d3cadec" (UID: "d2a58d8b-b84a-403f-bd46-5dfe7d3cadec"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:31:45.495464 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:45.493831 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2a58d8b-b84a-403f-bd46-5dfe7d3cadec-metrics-certs" pod "d2a58d8b-b84a-403f-bd46-5dfe7d3cadec" (UID: "d2a58d8b-b84a-403f-bd46-5dfe7d3cadec"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:31:45.508433 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:45.504570 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2a58d8b-b84a-403f-bd46-5dfe7d3cadec-kube-api-access-n78fl" pod "d2a58d8b-b84a-403f-bd46-5dfe7d3cadec" (UID: "d2a58d8b-b84a-403f-bd46-5dfe7d3cadec"). InnerVolumeSpecName "kube-api-access-n78fl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:31:45.525117 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:45.525029 4562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2a58d8b-b84a-403f-bd46-5dfe7d3cadec-openshift-service-ca\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:31:45.525117 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:45.525064 4562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2a58d8b-b84a-403f-bd46-5dfe7d3cadec-metrics-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:31:45.525117 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:45.525078 4562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2a58d8b-b84a-403f-bd46-5dfe7d3cadec-maas-api-tls\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:31:45.525117 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:45.525092 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n78fl\" (UniqueName: \"kubernetes.io/projected/d2a58d8b-b84a-403f-bd46-5dfe7d3cadec-kube-api-access-n78fl\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:31:45.682200 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:45.682113 4562 generic.go:356] "Generic (PLEG): container finished" podID="d2a58d8b-b84a-403f-bd46-5dfe7d3cadec" containerID="6781ba2699c5a77587074d740f1f42847f81c217a42803e1c63b10b4e8890a7e" exitCode=0 Oct 30 09:31:45.682200 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:45.682170 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d95d35-799494fd9b-rxn8v" Oct 30 09:31:45.682200 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:45.682199 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d95d35-799494fd9b-rxn8v" event={"ID":"d2a58d8b-b84a-403f-bd46-5dfe7d3cadec","Type":"ContainerDied","Data":"6781ba2699c5a77587074d740f1f42847f81c217a42803e1c63b10b4e8890a7e"} Oct 30 09:31:45.682513 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:45.682284 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d95d35-799494fd9b-rxn8v" event={"ID":"d2a58d8b-b84a-403f-bd46-5dfe7d3cadec","Type":"ContainerDied","Data":"b64b313656c063f814b78b5f8c46a0407e81739a4fa79daccd82c994a0b5a171"} Oct 30 09:31:45.682513 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:45.682314 4562 scope.go:122] "RemoveContainer" containerID="6781ba2699c5a77587074d740f1f42847f81c217a42803e1c63b10b4e8890a7e" Oct 30 09:31:45.692470 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:45.692453 4562 scope.go:122] "RemoveContainer" containerID="6781ba2699c5a77587074d740f1f42847f81c217a42803e1c63b10b4e8890a7e" Oct 30 09:31:45.692712 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:31:45.692690 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6781ba2699c5a77587074d740f1f42847f81c217a42803e1c63b10b4e8890a7e\": container with ID starting with 6781ba2699c5a77587074d740f1f42847f81c217a42803e1c63b10b4e8890a7e not found: ID does not exist" containerID="6781ba2699c5a77587074d740f1f42847f81c217a42803e1c63b10b4e8890a7e" Oct 30 09:31:45.692819 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:45.692715 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6781ba2699c5a77587074d740f1f42847f81c217a42803e1c63b10b4e8890a7e"} err="failed to get container status \"6781ba2699c5a77587074d740f1f42847f81c217a42803e1c63b10b4e8890a7e\": rpc error: code = NotFound desc = could not find container \"6781ba2699c5a77587074d740f1f42847f81c217a42803e1c63b10b4e8890a7e\": container with ID starting with 6781ba2699c5a77587074d740f1f42847f81c217a42803e1c63b10b4e8890a7e not found: ID does not exist" Oct 30 09:31:45.744576 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:45.744545 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-d95d35-799494fd9b-rxn8v"] Oct 30 09:31:45.757728 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:45.757697 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-d95d35-799494fd9b-rxn8v"] Oct 30 09:31:45.919077 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:45.913653 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2a58d8b-b84a-403f-bd46-5dfe7d3cadec" path="/var/lib/kubelet/pods/d2a58d8b-b84a-403f-bd46-5dfe7d3cadec/volumes" Oct 30 09:31:48.142254 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:48.142176 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-eff9fd-6b784dd94d-lrzsf" podUID="2198e031-552b-46f4-adc2-bfbf3bb08dfd" containerName="payload-processing" containerID="cri-o://01f5816caf5272858c6ad8aa76fa122503121bd7d7f2584d011eccb03cff7f92" gracePeriod=25 Oct 30 09:31:48.169819 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:48.169776 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eff9fd-847fb7448c-5n9rh" podUID="c8b60df6-3c80-4c90-9960-0af22206d277" containerName="payload-pre-processing" containerID="cri-o://e13ac977f6e2fa1b1da1d44eb3973dc8615145942af3fbf2f0ff55b45d584c7e" gracePeriod=25 Oct 30 09:31:48.468623 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:48.468597 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-eff9fd-6b784dd94d-lrzsf" Oct 30 09:31:48.488136 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:48.488114 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eff9fd-847fb7448c-5n9rh" Oct 30 09:31:48.567067 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:48.567038 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2198e031-552b-46f4-adc2-bfbf3bb08dfd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2198e031-552b-46f4-adc2-bfbf3bb08dfd-plugins-config-volume\") pod \"2198e031-552b-46f4-adc2-bfbf3bb08dfd\" (UID: \"2198e031-552b-46f4-adc2-bfbf3bb08dfd\") " Oct 30 09:31:48.567233 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:48.567113 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8b60df6-3c80-4c90-9960-0af22206d277-kube-api-access-22x2s\" (UniqueName: \"kubernetes.io/projected/c8b60df6-3c80-4c90-9960-0af22206d277-kube-api-access-22x2s\") pod \"c8b60df6-3c80-4c90-9960-0af22206d277\" (UID: \"c8b60df6-3c80-4c90-9960-0af22206d277\") " Oct 30 09:31:48.567233 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:48.567139 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2198e031-552b-46f4-adc2-bfbf3bb08dfd-kube-api-access-rmz7s\" (UniqueName: \"kubernetes.io/projected/2198e031-552b-46f4-adc2-bfbf3bb08dfd-kube-api-access-rmz7s\") pod \"2198e031-552b-46f4-adc2-bfbf3bb08dfd\" (UID: \"2198e031-552b-46f4-adc2-bfbf3bb08dfd\") " Oct 30 09:31:48.567501 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:48.567477 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2198e031-552b-46f4-adc2-bfbf3bb08dfd-plugins-config-volume" pod "2198e031-552b-46f4-adc2-bfbf3bb08dfd" (UID: "2198e031-552b-46f4-adc2-bfbf3bb08dfd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:31:48.567572 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:48.567480 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8b60df6-3c80-4c90-9960-0af22206d277-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8b60df6-3c80-4c90-9960-0af22206d277-plugins-config-volume\") pod \"c8b60df6-3c80-4c90-9960-0af22206d277\" (UID: \"c8b60df6-3c80-4c90-9960-0af22206d277\") " Oct 30 09:31:48.567823 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:48.567797 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8b60df6-3c80-4c90-9960-0af22206d277-plugins-config-volume" pod "c8b60df6-3c80-4c90-9960-0af22206d277" (UID: "c8b60df6-3c80-4c90-9960-0af22206d277"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:31:48.568133 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:48.568118 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8b60df6-3c80-4c90-9960-0af22206d277-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:31:48.568204 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:48.568137 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2198e031-552b-46f4-adc2-bfbf3bb08dfd-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:31:48.569970 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:48.569939 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8b60df6-3c80-4c90-9960-0af22206d277-kube-api-access-22x2s" pod "c8b60df6-3c80-4c90-9960-0af22206d277" (UID: "c8b60df6-3c80-4c90-9960-0af22206d277"). InnerVolumeSpecName "kube-api-access-22x2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:31:48.570332 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:48.570310 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2198e031-552b-46f4-adc2-bfbf3bb08dfd-kube-api-access-rmz7s" pod "2198e031-552b-46f4-adc2-bfbf3bb08dfd" (UID: "2198e031-552b-46f4-adc2-bfbf3bb08dfd"). InnerVolumeSpecName "kube-api-access-rmz7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:31:48.669741 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:48.669663 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22x2s\" (UniqueName: \"kubernetes.io/projected/c8b60df6-3c80-4c90-9960-0af22206d277-kube-api-access-22x2s\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:31:48.669741 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:48.669690 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmz7s\" (UniqueName: \"kubernetes.io/projected/2198e031-552b-46f4-adc2-bfbf3bb08dfd-kube-api-access-rmz7s\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:31:48.705335 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:48.705295 4562 generic.go:356] "Generic (PLEG): container finished" podID="2198e031-552b-46f4-adc2-bfbf3bb08dfd" containerID="01f5816caf5272858c6ad8aa76fa122503121bd7d7f2584d011eccb03cff7f92" exitCode=0 Oct 30 09:31:48.705553 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:48.705342 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-eff9fd-6b784dd94d-lrzsf" Oct 30 09:31:48.705553 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:48.705390 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-eff9fd-6b784dd94d-lrzsf" event={"ID":"2198e031-552b-46f4-adc2-bfbf3bb08dfd","Type":"ContainerDied","Data":"01f5816caf5272858c6ad8aa76fa122503121bd7d7f2584d011eccb03cff7f92"} Oct 30 09:31:48.705553 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:48.705446 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-eff9fd-6b784dd94d-lrzsf" event={"ID":"2198e031-552b-46f4-adc2-bfbf3bb08dfd","Type":"ContainerDied","Data":"a1b7d4111220339dc9fea60ed5502ae9ed15b81f4cd86c4ec609195da8e646c5"} Oct 30 09:31:48.705553 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:48.705472 4562 scope.go:122] "RemoveContainer" containerID="01f5816caf5272858c6ad8aa76fa122503121bd7d7f2584d011eccb03cff7f92" Oct 30 09:31:48.706952 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:48.706922 4562 generic.go:356] "Generic (PLEG): container finished" podID="c8b60df6-3c80-4c90-9960-0af22206d277" containerID="e13ac977f6e2fa1b1da1d44eb3973dc8615145942af3fbf2f0ff55b45d584c7e" exitCode=0 Oct 30 09:31:48.707061 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:48.706972 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eff9fd-847fb7448c-5n9rh" Oct 30 09:31:48.707061 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:48.706971 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eff9fd-847fb7448c-5n9rh" event={"ID":"c8b60df6-3c80-4c90-9960-0af22206d277","Type":"ContainerDied","Data":"e13ac977f6e2fa1b1da1d44eb3973dc8615145942af3fbf2f0ff55b45d584c7e"} Oct 30 09:31:48.707172 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:48.707080 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eff9fd-847fb7448c-5n9rh" event={"ID":"c8b60df6-3c80-4c90-9960-0af22206d277","Type":"ContainerDied","Data":"2c82463ff68f0897664fdf3e14c456f195b5e59f3b35a6b8e9b3478cdf9a656f"} Oct 30 09:31:48.717557 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:48.717536 4562 scope.go:122] "RemoveContainer" containerID="01f5816caf5272858c6ad8aa76fa122503121bd7d7f2584d011eccb03cff7f92" Oct 30 09:31:48.718233 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:31:48.718211 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01f5816caf5272858c6ad8aa76fa122503121bd7d7f2584d011eccb03cff7f92\": container with ID starting with 01f5816caf5272858c6ad8aa76fa122503121bd7d7f2584d011eccb03cff7f92 not found: ID does not exist" containerID="01f5816caf5272858c6ad8aa76fa122503121bd7d7f2584d011eccb03cff7f92" Oct 30 09:31:48.718327 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:48.718239 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01f5816caf5272858c6ad8aa76fa122503121bd7d7f2584d011eccb03cff7f92"} err="failed to get container status \"01f5816caf5272858c6ad8aa76fa122503121bd7d7f2584d011eccb03cff7f92\": rpc error: code = NotFound desc = could not find container \"01f5816caf5272858c6ad8aa76fa122503121bd7d7f2584d011eccb03cff7f92\": container with ID starting with 01f5816caf5272858c6ad8aa76fa122503121bd7d7f2584d011eccb03cff7f92 not found: ID does not exist" Oct 30 09:31:48.718327 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:48.718257 4562 scope.go:122] "RemoveContainer" containerID="e13ac977f6e2fa1b1da1d44eb3973dc8615145942af3fbf2f0ff55b45d584c7e" Oct 30 09:31:48.727797 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:48.727771 4562 scope.go:122] "RemoveContainer" containerID="e13ac977f6e2fa1b1da1d44eb3973dc8615145942af3fbf2f0ff55b45d584c7e" Oct 30 09:31:48.728084 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:31:48.728064 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e13ac977f6e2fa1b1da1d44eb3973dc8615145942af3fbf2f0ff55b45d584c7e\": container with ID starting with e13ac977f6e2fa1b1da1d44eb3973dc8615145942af3fbf2f0ff55b45d584c7e not found: ID does not exist" containerID="e13ac977f6e2fa1b1da1d44eb3973dc8615145942af3fbf2f0ff55b45d584c7e" Oct 30 09:31:48.728146 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:48.728097 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e13ac977f6e2fa1b1da1d44eb3973dc8615145942af3fbf2f0ff55b45d584c7e"} err="failed to get container status \"e13ac977f6e2fa1b1da1d44eb3973dc8615145942af3fbf2f0ff55b45d584c7e\": rpc error: code = NotFound desc = could not find container \"e13ac977f6e2fa1b1da1d44eb3973dc8615145942af3fbf2f0ff55b45d584c7e\": container with ID starting with e13ac977f6e2fa1b1da1d44eb3973dc8615145942af3fbf2f0ff55b45d584c7e not found: ID does not exist" Oct 30 09:31:48.762070 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:48.762027 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-eff9fd-847fb7448c-5n9rh"] Oct 30 09:31:48.765950 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:48.765922 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-eff9fd-847fb7448c-5n9rh"] Oct 30 09:31:48.802640 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:48.802603 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-eff9fd-6b784dd94d-lrzsf"] Oct 30 09:31:48.807199 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:48.807170 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-eff9fd-6b784dd94d-lrzsf"] Oct 30 09:31:49.058932 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:49.058903 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-85rhg_2f5e6dbb-9b81-4734-ba2a-5c6a677e49e8/payload-processing/0.log" Oct 30 09:31:49.270001 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:49.269974 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-85rhg_2f5e6dbb-9b81-4734-ba2a-5c6a677e49e8/payload-processing/0.log" Oct 30 09:31:49.474421 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:49.471750 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-15e1cc-c988685bd-6z7kz_62afe007-634a-41ab-9a9b-b10b4578452e/payload-processing/0.log" Oct 30 09:31:49.664010 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:49.663982 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-15e1cc-c988685bd-6z7kz_62afe007-634a-41ab-9a9b-b10b4578452e/payload-processing/0.log" Oct 30 09:31:49.874728 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:49.874655 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2198e031-552b-46f4-adc2-bfbf3bb08dfd" path="/var/lib/kubelet/pods/2198e031-552b-46f4-adc2-bfbf3bb08dfd/volumes" Oct 30 09:31:49.875177 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:49.875152 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8b60df6-3c80-4c90-9960-0af22206d277" path="/var/lib/kubelet/pods/c8b60df6-3c80-4c90-9960-0af22206d277/volumes" Oct 30 09:31:55.156043 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.156002 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-eff9fd-b94d9594f-z9mbl"] Oct 30 09:31:55.156556 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.156269 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eff9fd-b94d9594f-z9mbl" podUID="3fb8f8cd-509a-49e4-abcf-82095b5a9731" containerName="maas-api" containerID="cri-o://010aee899fb1c91c00491e97836c2582c2b9d8d5db57a3221fc98e72cf17bc29" gracePeriod=30 Oct 30 09:31:55.211436 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.206644 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt"] Oct 30 09:31:55.211436 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.207291 4562 cpu_manager.go:414] "containerMap: removing container" podUID="d2a58d8b-b84a-403f-bd46-5dfe7d3cadec" containerName="maas-api" Oct 30 09:31:55.211436 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.207305 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2a58d8b-b84a-403f-bd46-5dfe7d3cadec" containerName="maas-api" Oct 30 09:31:55.211436 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.207323 4562 cpu_manager.go:414] "containerMap: removing container" podUID="49e0fd93-7200-46ea-bcf7-28d1df31d16f" containerName="payload-processing" Oct 30 09:31:55.211436 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.207328 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49e0fd93-7200-46ea-bcf7-28d1df31d16f" containerName="payload-processing" Oct 30 09:31:55.211436 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.207336 4562 cpu_manager.go:414] "containerMap: removing container" podUID="2198e031-552b-46f4-adc2-bfbf3bb08dfd" containerName="payload-processing" Oct 30 09:31:55.211436 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.207341 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2198e031-552b-46f4-adc2-bfbf3bb08dfd" containerName="payload-processing" Oct 30 09:31:55.211436 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.207353 4562 cpu_manager.go:414] "containerMap: removing container" podUID="14458dfc-296c-47f3-adb5-1b28e26b4314" containerName="payload-pre-processing" Oct 30 09:31:55.211436 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.207358 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14458dfc-296c-47f3-adb5-1b28e26b4314" containerName="payload-pre-processing" Oct 30 09:31:55.211436 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.207366 4562 cpu_manager.go:414] "containerMap: removing container" podUID="c8b60df6-3c80-4c90-9960-0af22206d277" containerName="payload-pre-processing" Oct 30 09:31:55.211436 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.207370 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8b60df6-3c80-4c90-9960-0af22206d277" containerName="payload-pre-processing" Oct 30 09:31:55.211436 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.207406 4562 cpu_manager.go:414] "containerMap: removing container" podUID="028fea98-1060-44f3-887a-bf0a119b428b" containerName="istio-proxy" Oct 30 09:31:55.211436 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.207414 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="028fea98-1060-44f3-887a-bf0a119b428b" containerName="istio-proxy" Oct 30 09:31:55.211436 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.207510 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2a58d8b-b84a-403f-bd46-5dfe7d3cadec" containerName="maas-api" Oct 30 09:31:55.211436 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.207518 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="49e0fd93-7200-46ea-bcf7-28d1df31d16f" containerName="payload-processing" Oct 30 09:31:55.211436 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.207524 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="2198e031-552b-46f4-adc2-bfbf3bb08dfd" containerName="payload-processing" Oct 30 09:31:55.211436 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.207531 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="14458dfc-296c-47f3-adb5-1b28e26b4314" containerName="payload-pre-processing" Oct 30 09:31:55.211436 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.207536 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8b60df6-3c80-4c90-9960-0af22206d277" containerName="payload-pre-processing" Oct 30 09:31:55.211436 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.207544 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="028fea98-1060-44f3-887a-bf0a119b428b" containerName="istio-proxy" Oct 30 09:31:55.211436 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.210994 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt" Oct 30 09:31:55.221758 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.218500 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt"] Oct 30 09:31:55.230838 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.229250 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-19c94a-openshift-default-dockercfg-md5pq\"" Oct 30 09:31:55.369319 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.369283 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcnlw\" (UniqueName: \"kubernetes.io/projected/5f848cb2-5cdc-4f31-b063-8745991865de-kube-api-access-rcnlw\") pod \"e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt\" (UID: \"5f848cb2-5cdc-4f31-b063-8745991865de\") " pod="openshift-ingress/e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt" Oct 30 09:31:55.369493 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.369339 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f848cb2-5cdc-4f31-b063-8745991865de-istio-podinfo\") pod \"e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt\" (UID: \"5f848cb2-5cdc-4f31-b063-8745991865de\") " pod="openshift-ingress/e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt" Oct 30 09:31:55.369493 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.369372 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f848cb2-5cdc-4f31-b063-8745991865de-istio-token\") pod \"e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt\" (UID: \"5f848cb2-5cdc-4f31-b063-8745991865de\") " pod="openshift-ingress/e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt" Oct 30 09:31:55.369493 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.369460 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f848cb2-5cdc-4f31-b063-8745991865de-credential-socket\") pod \"e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt\" (UID: \"5f848cb2-5cdc-4f31-b063-8745991865de\") " pod="openshift-ingress/e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt" Oct 30 09:31:55.369493 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.369488 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f848cb2-5cdc-4f31-b063-8745991865de-workload-socket\") pod \"e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt\" (UID: \"5f848cb2-5cdc-4f31-b063-8745991865de\") " pod="openshift-ingress/e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt" Oct 30 09:31:55.369732 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.369570 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f848cb2-5cdc-4f31-b063-8745991865de-workload-certs\") pod \"e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt\" (UID: \"5f848cb2-5cdc-4f31-b063-8745991865de\") " pod="openshift-ingress/e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt" Oct 30 09:31:55.369732 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.369643 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f848cb2-5cdc-4f31-b063-8745991865de-istio-envoy\") pod \"e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt\" (UID: \"5f848cb2-5cdc-4f31-b063-8745991865de\") " pod="openshift-ingress/e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt" Oct 30 09:31:55.369732 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.369702 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f848cb2-5cdc-4f31-b063-8745991865de-istiod-ca-cert\") pod \"e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt\" (UID: \"5f848cb2-5cdc-4f31-b063-8745991865de\") " pod="openshift-ingress/e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt" Oct 30 09:31:55.369879 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.369795 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f848cb2-5cdc-4f31-b063-8745991865de-istio-data\") pod \"e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt\" (UID: \"5f848cb2-5cdc-4f31-b063-8745991865de\") " pod="openshift-ingress/e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt" Oct 30 09:31:55.456279 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.456253 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eff9fd-b94d9594f-z9mbl" Oct 30 09:31:55.470857 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.470830 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f848cb2-5cdc-4f31-b063-8745991865de-istio-envoy\") pod \"e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt\" (UID: \"5f848cb2-5cdc-4f31-b063-8745991865de\") " pod="openshift-ingress/e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt" Oct 30 09:31:55.471014 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.470897 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f848cb2-5cdc-4f31-b063-8745991865de-istiod-ca-cert\") pod \"e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt\" (UID: \"5f848cb2-5cdc-4f31-b063-8745991865de\") " pod="openshift-ingress/e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt" Oct 30 09:31:55.471014 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.470993 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f848cb2-5cdc-4f31-b063-8745991865de-istio-data\") pod \"e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt\" (UID: \"5f848cb2-5cdc-4f31-b063-8745991865de\") " pod="openshift-ingress/e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt" Oct 30 09:31:55.471143 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.471062 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcnlw\" (UniqueName: \"kubernetes.io/projected/5f848cb2-5cdc-4f31-b063-8745991865de-kube-api-access-rcnlw\") pod \"e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt\" (UID: \"5f848cb2-5cdc-4f31-b063-8745991865de\") " pod="openshift-ingress/e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt" Oct 30 09:31:55.471143 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.471104 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f848cb2-5cdc-4f31-b063-8745991865de-istio-podinfo\") pod \"e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt\" (UID: \"5f848cb2-5cdc-4f31-b063-8745991865de\") " pod="openshift-ingress/e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt" Oct 30 09:31:55.471253 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.471173 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f848cb2-5cdc-4f31-b063-8745991865de-istio-token\") pod \"e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt\" (UID: \"5f848cb2-5cdc-4f31-b063-8745991865de\") " pod="openshift-ingress/e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt" Oct 30 09:31:55.471305 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.471260 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f848cb2-5cdc-4f31-b063-8745991865de-credential-socket\") pod \"e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt\" (UID: \"5f848cb2-5cdc-4f31-b063-8745991865de\") " pod="openshift-ingress/e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt" Oct 30 09:31:55.471305 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.471290 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f848cb2-5cdc-4f31-b063-8745991865de-workload-socket\") pod \"e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt\" (UID: \"5f848cb2-5cdc-4f31-b063-8745991865de\") " pod="openshift-ingress/e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt" Oct 30 09:31:55.471458 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.471311 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f848cb2-5cdc-4f31-b063-8745991865de-istio-data\") pod \"e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt\" (UID: \"5f848cb2-5cdc-4f31-b063-8745991865de\") " pod="openshift-ingress/e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt" Oct 30 09:31:55.471458 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.471415 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f848cb2-5cdc-4f31-b063-8745991865de-workload-certs\") pod \"e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt\" (UID: \"5f848cb2-5cdc-4f31-b063-8745991865de\") " pod="openshift-ingress/e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt" Oct 30 09:31:55.471626 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.471602 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f848cb2-5cdc-4f31-b063-8745991865de-workload-socket\") pod \"e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt\" (UID: \"5f848cb2-5cdc-4f31-b063-8745991865de\") " pod="openshift-ingress/e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt" Oct 30 09:31:55.471696 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.471631 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f848cb2-5cdc-4f31-b063-8745991865de-credential-socket\") pod \"e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt\" (UID: \"5f848cb2-5cdc-4f31-b063-8745991865de\") " pod="openshift-ingress/e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt" Oct 30 09:31:55.471696 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.471680 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f848cb2-5cdc-4f31-b063-8745991865de-istiod-ca-cert\") pod \"e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt\" (UID: \"5f848cb2-5cdc-4f31-b063-8745991865de\") " pod="openshift-ingress/e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt" Oct 30 09:31:55.471822 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.471799 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f848cb2-5cdc-4f31-b063-8745991865de-workload-certs\") pod \"e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt\" (UID: \"5f848cb2-5cdc-4f31-b063-8745991865de\") " pod="openshift-ingress/e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt" Oct 30 09:31:55.473136 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.473112 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f848cb2-5cdc-4f31-b063-8745991865de-istio-envoy\") pod \"e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt\" (UID: \"5f848cb2-5cdc-4f31-b063-8745991865de\") " pod="openshift-ingress/e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt" Oct 30 09:31:55.473590 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.473574 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f848cb2-5cdc-4f31-b063-8745991865de-istio-podinfo\") pod \"e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt\" (UID: \"5f848cb2-5cdc-4f31-b063-8745991865de\") " pod="openshift-ingress/e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt" Oct 30 09:31:55.491585 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.491558 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f848cb2-5cdc-4f31-b063-8745991865de-istio-token\") pod \"e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt\" (UID: \"5f848cb2-5cdc-4f31-b063-8745991865de\") " pod="openshift-ingress/e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt" Oct 30 09:31:55.491702 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.491628 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcnlw\" (UniqueName: \"kubernetes.io/projected/5f848cb2-5cdc-4f31-b063-8745991865de-kube-api-access-rcnlw\") pod \"e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt\" (UID: \"5f848cb2-5cdc-4f31-b063-8745991865de\") " pod="openshift-ingress/e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt" Oct 30 09:31:55.551680 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.551643 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt" Oct 30 09:31:55.572095 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.572064 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fb8f8cd-509a-49e4-abcf-82095b5a9731-kube-api-access-4psg5\" (UniqueName: \"kubernetes.io/projected/3fb8f8cd-509a-49e4-abcf-82095b5a9731-kube-api-access-4psg5\") pod \"3fb8f8cd-509a-49e4-abcf-82095b5a9731\" (UID: \"3fb8f8cd-509a-49e4-abcf-82095b5a9731\") " Oct 30 09:31:55.572209 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.572149 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fb8f8cd-509a-49e4-abcf-82095b5a9731-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fb8f8cd-509a-49e4-abcf-82095b5a9731-maas-api-tls\") pod \"3fb8f8cd-509a-49e4-abcf-82095b5a9731\" (UID: \"3fb8f8cd-509a-49e4-abcf-82095b5a9731\") " Oct 30 09:31:55.572260 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.572244 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fb8f8cd-509a-49e4-abcf-82095b5a9731-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fb8f8cd-509a-49e4-abcf-82095b5a9731-metrics-certs\") pod \"3fb8f8cd-509a-49e4-abcf-82095b5a9731\" (UID: \"3fb8f8cd-509a-49e4-abcf-82095b5a9731\") " Oct 30 09:31:55.572362 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.572347 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fb8f8cd-509a-49e4-abcf-82095b5a9731-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fb8f8cd-509a-49e4-abcf-82095b5a9731-openshift-service-ca\") pod \"3fb8f8cd-509a-49e4-abcf-82095b5a9731\" (UID: \"3fb8f8cd-509a-49e4-abcf-82095b5a9731\") " Oct 30 09:31:55.572653 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.572631 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fb8f8cd-509a-49e4-abcf-82095b5a9731-openshift-service-ca" pod "3fb8f8cd-509a-49e4-abcf-82095b5a9731" (UID: "3fb8f8cd-509a-49e4-abcf-82095b5a9731"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:31:55.573027 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.573010 4562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fb8f8cd-509a-49e4-abcf-82095b5a9731-openshift-service-ca\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:31:55.574328 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.574306 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fb8f8cd-509a-49e4-abcf-82095b5a9731-metrics-certs" pod "3fb8f8cd-509a-49e4-abcf-82095b5a9731" (UID: "3fb8f8cd-509a-49e4-abcf-82095b5a9731"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:31:55.574658 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.574640 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fb8f8cd-509a-49e4-abcf-82095b5a9731-maas-api-tls" pod "3fb8f8cd-509a-49e4-abcf-82095b5a9731" (UID: "3fb8f8cd-509a-49e4-abcf-82095b5a9731"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:31:55.574723 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.574664 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fb8f8cd-509a-49e4-abcf-82095b5a9731-kube-api-access-4psg5" pod "3fb8f8cd-509a-49e4-abcf-82095b5a9731" (UID: "3fb8f8cd-509a-49e4-abcf-82095b5a9731"). InnerVolumeSpecName "kube-api-access-4psg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:31:55.682098 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.681962 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4psg5\" (UniqueName: \"kubernetes.io/projected/3fb8f8cd-509a-49e4-abcf-82095b5a9731-kube-api-access-4psg5\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:31:55.682098 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.682002 4562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fb8f8cd-509a-49e4-abcf-82095b5a9731-maas-api-tls\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:31:55.682098 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.682018 4562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fb8f8cd-509a-49e4-abcf-82095b5a9731-metrics-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:31:55.849360 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.843981 4562 generic.go:356] "Generic (PLEG): container finished" podID="3fb8f8cd-509a-49e4-abcf-82095b5a9731" containerID="010aee899fb1c91c00491e97836c2582c2b9d8d5db57a3221fc98e72cf17bc29" exitCode=0 Oct 30 09:31:55.849360 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.844033 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eff9fd-b94d9594f-z9mbl" event={"ID":"3fb8f8cd-509a-49e4-abcf-82095b5a9731","Type":"ContainerDied","Data":"010aee899fb1c91c00491e97836c2582c2b9d8d5db57a3221fc98e72cf17bc29"} Oct 30 09:31:55.849360 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.844097 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eff9fd-b94d9594f-z9mbl" event={"ID":"3fb8f8cd-509a-49e4-abcf-82095b5a9731","Type":"ContainerDied","Data":"bba653ece5fe4154af56026a52f3b0a2c2a6c05fcf1d60a03c52a304962769fb"} Oct 30 09:31:55.849360 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.844126 4562 scope.go:122] "RemoveContainer" containerID="010aee899fb1c91c00491e97836c2582c2b9d8d5db57a3221fc98e72cf17bc29" Oct 30 09:31:55.849360 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.844295 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eff9fd-b94d9594f-z9mbl" Oct 30 09:31:55.865188 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.865167 4562 scope.go:122] "RemoveContainer" containerID="010aee899fb1c91c00491e97836c2582c2b9d8d5db57a3221fc98e72cf17bc29" Oct 30 09:31:55.865512 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:31:55.865490 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"010aee899fb1c91c00491e97836c2582c2b9d8d5db57a3221fc98e72cf17bc29\": container with ID starting with 010aee899fb1c91c00491e97836c2582c2b9d8d5db57a3221fc98e72cf17bc29 not found: ID does not exist" containerID="010aee899fb1c91c00491e97836c2582c2b9d8d5db57a3221fc98e72cf17bc29" Oct 30 09:31:55.865590 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.865521 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"010aee899fb1c91c00491e97836c2582c2b9d8d5db57a3221fc98e72cf17bc29"} err="failed to get container status \"010aee899fb1c91c00491e97836c2582c2b9d8d5db57a3221fc98e72cf17bc29\": rpc error: code = NotFound desc = could not find container \"010aee899fb1c91c00491e97836c2582c2b9d8d5db57a3221fc98e72cf17bc29\": container with ID starting with 010aee899fb1c91c00491e97836c2582c2b9d8d5db57a3221fc98e72cf17bc29 not found: ID does not exist" Oct 30 09:31:55.926170 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.926135 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-eff9fd-b94d9594f-z9mbl"] Oct 30 09:31:55.931843 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:55.931817 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-eff9fd-b94d9594f-z9mbl"] Oct 30 09:31:56.071986 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:31:56.071953 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f848cb2_5cdc_4f31_b063_8745991865de.slice/crio-db113b981633732aff3eab15ebe56fc3c92a95c982c7420ef87f49f6bf517af8 WatchSource:0}: Error finding container db113b981633732aff3eab15ebe56fc3c92a95c982c7420ef87f49f6bf517af8: Status 404 returned error can't find the container with id db113b981633732aff3eab15ebe56fc3c92a95c982c7420ef87f49f6bf517af8 Oct 30 09:31:56.074453 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:56.074426 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt"] Oct 30 09:31:56.075838 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:56.075804 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:31:56.075939 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:56.075883 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:31:56.075939 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:56.075924 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:31:56.853924 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:56.853879 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt" event={"ID":"5f848cb2-5cdc-4f31-b063-8745991865de","Type":"ContainerStarted","Data":"eadf17790c84ef362a51c2eaa13c23aa51452a48c29fc5918c8c6f02ba3eddf2"} Oct 30 09:31:56.854533 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:56.853988 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt" event={"ID":"5f848cb2-5cdc-4f31-b063-8745991865de","Type":"ContainerStarted","Data":"db113b981633732aff3eab15ebe56fc3c92a95c982c7420ef87f49f6bf517af8"} Oct 30 09:31:56.899621 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:56.899552 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt" podStartSLOduration=1.899533555 podStartE2EDuration="1.899533555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:31:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:31:56.896594246 +0000 UTC m=+2277.158155236" watchObservedRunningTime="2026-10-30 09:31:56.899533555 +0000 UTC m=+2277.161094542" Oct 30 09:31:57.552603 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:57.552572 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt" Oct 30 09:31:57.557959 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:57.557938 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt" Oct 30 09:31:57.860595 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:57.860515 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt" Oct 30 09:31:57.861496 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:57.861476 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt" Oct 30 09:31:57.874784 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:31:57.874751 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fb8f8cd-509a-49e4-abcf-82095b5a9731" path="/var/lib/kubelet/pods/3fb8f8cd-509a-49e4-abcf-82095b5a9731/volumes" Oct 30 09:31:58.596179 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:31:58.596136 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:32:01.271321 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:01.271283 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-19c94a-87d4f8f55-kt6zq"] Oct 30 09:32:01.271901 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:01.271886 4562 cpu_manager.go:414] "containerMap: removing container" podUID="3fb8f8cd-509a-49e4-abcf-82095b5a9731" containerName="maas-api" Oct 30 09:32:01.271953 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:01.271902 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fb8f8cd-509a-49e4-abcf-82095b5a9731" containerName="maas-api" Oct 30 09:32:01.272017 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:01.272007 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fb8f8cd-509a-49e4-abcf-82095b5a9731" containerName="maas-api" Oct 30 09:32:01.276062 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:01.276040 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-19c94a-87d4f8f55-kt6zq" Oct 30 09:32:01.295091 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:01.295064 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-19c94a\"" Oct 30 09:32:01.302400 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:01.300920 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-19c94a-87d4f8f55-kt6zq"] Oct 30 09:32:01.309067 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:01.307710 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-19c94a-dockercfg-sb2nd\"" Oct 30 09:32:01.355568 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:01.355544 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-19c94a-69db9c4498-s4qsp"] Oct 30 09:32:01.359768 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:01.359746 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-19c94a-69db9c4498-s4qsp" Oct 30 09:32:01.431405 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:01.428266 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8c76d69-147f-4a23-84e2-c5aedeaa359d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-19c94a-87d4f8f55-kt6zq\" (UID: \"a8c76d69-147f-4a23-84e2-c5aedeaa359d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-19c94a-87d4f8f55-kt6zq" Oct 30 09:32:01.431405 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:01.428444 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brmpd\" (UniqueName: \"kubernetes.io/projected/a8c76d69-147f-4a23-84e2-c5aedeaa359d-kube-api-access-brmpd\") pod \"payload-pre-processing-e2e-shared-a-w4-19c94a-87d4f8f55-kt6zq\" (UID: \"a8c76d69-147f-4a23-84e2-c5aedeaa359d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-19c94a-87d4f8f55-kt6zq" Oct 30 09:32:01.521749 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:01.521657 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-19c94a-69db9c4498-s4qsp"] Oct 30 09:32:01.535823 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:01.533212 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18022e3d-385e-44ce-8c05-e8bae4b030d9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-19c94a-69db9c4498-s4qsp\" (UID: \"18022e3d-385e-44ce-8c05-e8bae4b030d9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-19c94a-69db9c4498-s4qsp" Oct 30 09:32:01.535823 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:01.533293 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m9m8\" (UniqueName: \"kubernetes.io/projected/18022e3d-385e-44ce-8c05-e8bae4b030d9-kube-api-access-6m9m8\") pod \"payload-processing-e2e-shared-a-w4-19c94a-69db9c4498-s4qsp\" (UID: \"18022e3d-385e-44ce-8c05-e8bae4b030d9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-19c94a-69db9c4498-s4qsp" Oct 30 09:32:01.535823 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:01.533338 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8c76d69-147f-4a23-84e2-c5aedeaa359d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-19c94a-87d4f8f55-kt6zq\" (UID: \"a8c76d69-147f-4a23-84e2-c5aedeaa359d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-19c94a-87d4f8f55-kt6zq" Oct 30 09:32:01.536645 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:01.536616 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brmpd\" (UniqueName: \"kubernetes.io/projected/a8c76d69-147f-4a23-84e2-c5aedeaa359d-kube-api-access-brmpd\") pod \"payload-pre-processing-e2e-shared-a-w4-19c94a-87d4f8f55-kt6zq\" (UID: \"a8c76d69-147f-4a23-84e2-c5aedeaa359d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-19c94a-87d4f8f55-kt6zq" Oct 30 09:32:01.538670 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:01.538644 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8c76d69-147f-4a23-84e2-c5aedeaa359d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-19c94a-87d4f8f55-kt6zq\" (UID: \"a8c76d69-147f-4a23-84e2-c5aedeaa359d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-19c94a-87d4f8f55-kt6zq" Oct 30 09:32:01.609116 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:01.609083 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brmpd\" (UniqueName: \"kubernetes.io/projected/a8c76d69-147f-4a23-84e2-c5aedeaa359d-kube-api-access-brmpd\") pod \"payload-pre-processing-e2e-shared-a-w4-19c94a-87d4f8f55-kt6zq\" (UID: \"a8c76d69-147f-4a23-84e2-c5aedeaa359d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-19c94a-87d4f8f55-kt6zq" Oct 30 09:32:01.639358 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:01.639327 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18022e3d-385e-44ce-8c05-e8bae4b030d9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-19c94a-69db9c4498-s4qsp\" (UID: \"18022e3d-385e-44ce-8c05-e8bae4b030d9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-19c94a-69db9c4498-s4qsp" Oct 30 09:32:01.639570 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:01.639375 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m9m8\" (UniqueName: \"kubernetes.io/projected/18022e3d-385e-44ce-8c05-e8bae4b030d9-kube-api-access-6m9m8\") pod \"payload-processing-e2e-shared-a-w4-19c94a-69db9c4498-s4qsp\" (UID: \"18022e3d-385e-44ce-8c05-e8bae4b030d9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-19c94a-69db9c4498-s4qsp" Oct 30 09:32:01.640073 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:01.640052 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18022e3d-385e-44ce-8c05-e8bae4b030d9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-19c94a-69db9c4498-s4qsp\" (UID: \"18022e3d-385e-44ce-8c05-e8bae4b030d9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-19c94a-69db9c4498-s4qsp" Oct 30 09:32:01.665227 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:01.665201 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m9m8\" (UniqueName: \"kubernetes.io/projected/18022e3d-385e-44ce-8c05-e8bae4b030d9-kube-api-access-6m9m8\") pod \"payload-processing-e2e-shared-a-w4-19c94a-69db9c4498-s4qsp\" (UID: \"18022e3d-385e-44ce-8c05-e8bae4b030d9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-19c94a-69db9c4498-s4qsp" Oct 30 09:32:01.674494 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:01.674470 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-19c94a-69db9c4498-s4qsp" Oct 30 09:32:01.896816 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:01.896728 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-19c94a-87d4f8f55-kt6zq" Oct 30 09:32:02.000050 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:32:01.999228 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18022e3d_385e_44ce_8c05_e8bae4b030d9.slice/crio-bf2a482f7b499cac076136ae063fd8fd1522785ece943ba3c00e5ee250f84c34 WatchSource:0}: Error finding container bf2a482f7b499cac076136ae063fd8fd1522785ece943ba3c00e5ee250f84c34: Status 404 returned error can't find the container with id bf2a482f7b499cac076136ae063fd8fd1522785ece943ba3c00e5ee250f84c34 Oct 30 09:32:02.000846 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:02.000512 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-19c94a-69db9c4498-s4qsp"] Oct 30 09:32:02.206422 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:32:02.206360 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8c76d69_147f_4a23_84e2_c5aedeaa359d.slice/crio-1aeb6a14d3e0105bb4b1154c2814b72cdf2d2c6fe6db91920dd55ac901c54086 WatchSource:0}: Error finding container 1aeb6a14d3e0105bb4b1154c2814b72cdf2d2c6fe6db91920dd55ac901c54086: Status 404 returned error can't find the container with id 1aeb6a14d3e0105bb4b1154c2814b72cdf2d2c6fe6db91920dd55ac901c54086 Oct 30 09:32:02.207464 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:02.207434 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-19c94a-87d4f8f55-kt6zq"] Oct 30 09:32:02.898943 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:02.898846 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-19c94a-69db9c4498-s4qsp" event={"ID":"18022e3d-385e-44ce-8c05-e8bae4b030d9","Type":"ContainerStarted","Data":"e3756d3fb7b6f94cb8d13f43a8f527e842d89de77cdd6702de1bad035671b4b1"} Oct 30 09:32:02.898943 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:02.898905 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-19c94a-69db9c4498-s4qsp" event={"ID":"18022e3d-385e-44ce-8c05-e8bae4b030d9","Type":"ContainerStarted","Data":"bf2a482f7b499cac076136ae063fd8fd1522785ece943ba3c00e5ee250f84c34"} Oct 30 09:32:02.899473 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:02.899007 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-19c94a-69db9c4498-s4qsp" Oct 30 09:32:02.900556 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:02.900533 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-19c94a-87d4f8f55-kt6zq" event={"ID":"a8c76d69-147f-4a23-84e2-c5aedeaa359d","Type":"ContainerStarted","Data":"3bc4fb1e908cafea88773a7ec913ce01f8d92d8a35cf5b852e9d3e81ca2beb13"} Oct 30 09:32:02.900680 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:02.900570 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-19c94a-87d4f8f55-kt6zq" event={"ID":"a8c76d69-147f-4a23-84e2-c5aedeaa359d","Type":"ContainerStarted","Data":"1aeb6a14d3e0105bb4b1154c2814b72cdf2d2c6fe6db91920dd55ac901c54086"} Oct 30 09:32:02.900680 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:02.900588 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-19c94a-87d4f8f55-kt6zq" Oct 30 09:32:02.942099 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:02.942044 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-19c94a-69db9c4498-s4qsp" podStartSLOduration=1.9420276520000002 podStartE2EDuration="1.942027652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:32:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:32:02.939657915 +0000 UTC m=+2283.201218906" watchObservedRunningTime="2026-10-30 09:32:02.942027652 +0000 UTC m=+2283.203588635" Oct 30 09:32:02.984409 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:02.984350 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-19c94a-87d4f8f55-kt6zq" podStartSLOduration=1.984337383 podStartE2EDuration="1.984337383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:32:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:32:02.980688081 +0000 UTC m=+2283.242249069" watchObservedRunningTime="2026-10-30 09:32:02.984337383 +0000 UTC m=+2283.245898370" Oct 30 09:32:03.663363 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:03.663332 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj"] Oct 30 09:32:03.663616 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:03.663592 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj" podUID="f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64" containerName="istio-proxy" containerID="cri-o://5fef09b987b0725a65b240bee63a0d4a5825d8bb44ca482440308c8db0f8b0f9" gracePeriod=30 Oct 30 09:32:08.946362 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:08.946340 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj" Oct 30 09:32:08.999208 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:08.999178 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64-credential-socket\") pod \"f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64\" (UID: \"f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64\") " Oct 30 09:32:08.999339 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:08.999241 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64-istio-envoy\") pod \"f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64\" (UID: \"f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64\") " Oct 30 09:32:08.999339 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:08.999281 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64-istio-podinfo\") pod \"f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64\" (UID: \"f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64\") " Oct 30 09:32:08.999339 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:08.999316 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64-istiod-ca-cert\") pod \"f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64\" (UID: \"f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64\") " Oct 30 09:32:08.999547 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:08.999343 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64-kube-api-access-mj7wd\" (UniqueName: \"kubernetes.io/projected/f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64-kube-api-access-mj7wd\") pod \"f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64\" (UID: \"f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64\") " Oct 30 09:32:08.999547 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:08.999430 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64-istio-data\") pod \"f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64\" (UID: \"f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64\") " Oct 30 09:32:08.999547 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:08.999487 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64-credential-socket" pod "f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64" (UID: "f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:32:08.999717 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:08.999620 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64-workload-socket\") pod \"f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64\" (UID: \"f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64\") " Oct 30 09:32:08.999717 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:08.999682 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64-istio-token\" (UniqueName: \"kubernetes.io/projected/f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64-istio-token\") pod \"f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64\" (UID: \"f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64\") " Oct 30 09:32:08.999717 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:08.999697 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64-istiod-ca-cert" pod "f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64" (UID: "f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:32:08.999913 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:08.999717 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64-workload-certs\") pod \"f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64\" (UID: \"f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64\") " Oct 30 09:32:08.999992 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:08.999941 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64-workload-socket" pod "f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64" (UID: "f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:32:09.000175 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:09.000126 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64-workload-certs" pod "f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64" (UID: "f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:32:09.000318 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:09.000242 4562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64-workload-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:32:09.000318 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:09.000261 4562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64-credential-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:32:09.000318 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:08.999876 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64-istio-data" pod "f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64" (UID: "f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:32:09.000318 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:09.000275 4562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64-istiod-ca-cert\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:32:09.000530 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:09.000320 4562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64-workload-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:32:09.002027 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:09.001995 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64-istio-envoy" pod "f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64" (UID: "f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:32:09.002992 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:09.002916 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64-istio-podinfo" pod "f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64" (UID: "f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:32:09.002992 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:09.002960 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64-kube-api-access-mj7wd" pod "f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64" (UID: "f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64"). InnerVolumeSpecName "kube-api-access-mj7wd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:32:09.003313 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:09.003294 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64-istio-token" pod "f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64" (UID: "f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:32:09.101163 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:09.101133 4562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64-istio-token\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:32:09.101163 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:09.101158 4562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64-istio-envoy\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:32:09.101163 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:09.101167 4562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64-istio-podinfo\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:32:09.101374 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:09.101175 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mj7wd\" (UniqueName: \"kubernetes.io/projected/f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64-kube-api-access-mj7wd\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:32:09.101374 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:09.101196 4562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64-istio-data\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:32:09.112841 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:09.112814 4562 generic.go:356] "Generic (PLEG): container finished" podID="f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64" containerID="5fef09b987b0725a65b240bee63a0d4a5825d8bb44ca482440308c8db0f8b0f9" exitCode=0 Oct 30 09:32:09.112974 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:09.112850 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj" event={"ID":"f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64","Type":"ContainerDied","Data":"5fef09b987b0725a65b240bee63a0d4a5825d8bb44ca482440308c8db0f8b0f9"} Oct 30 09:32:09.112974 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:09.112865 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj" Oct 30 09:32:09.112974 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:09.112888 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj" event={"ID":"f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64","Type":"ContainerDied","Data":"dcf85573a1f48e421e41cd740b1d15d4b1bb9d384899f3ff41f7d4fe34889b09"} Oct 30 09:32:09.112974 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:09.112909 4562 scope.go:122] "RemoveContainer" containerID="5fef09b987b0725a65b240bee63a0d4a5825d8bb44ca482440308c8db0f8b0f9" Oct 30 09:32:09.123219 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:09.123199 4562 scope.go:122] "RemoveContainer" containerID="399e769bcd0cff3a38573e1a9c7d015ea6a1098c2ab43b44cb47e566bed33985" Oct 30 09:32:09.131797 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:09.131782 4562 scope.go:122] "RemoveContainer" containerID="5fef09b987b0725a65b240bee63a0d4a5825d8bb44ca482440308c8db0f8b0f9" Oct 30 09:32:09.132032 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:32:09.132017 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fef09b987b0725a65b240bee63a0d4a5825d8bb44ca482440308c8db0f8b0f9\": container with ID starting with 5fef09b987b0725a65b240bee63a0d4a5825d8bb44ca482440308c8db0f8b0f9 not found: ID does not exist" containerID="5fef09b987b0725a65b240bee63a0d4a5825d8bb44ca482440308c8db0f8b0f9" Oct 30 09:32:09.132080 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:09.132039 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fef09b987b0725a65b240bee63a0d4a5825d8bb44ca482440308c8db0f8b0f9"} err="failed to get container status \"5fef09b987b0725a65b240bee63a0d4a5825d8bb44ca482440308c8db0f8b0f9\": rpc error: code = NotFound desc = could not find container \"5fef09b987b0725a65b240bee63a0d4a5825d8bb44ca482440308c8db0f8b0f9\": container with ID starting with 5fef09b987b0725a65b240bee63a0d4a5825d8bb44ca482440308c8db0f8b0f9 not found: ID does not exist" Oct 30 09:32:09.132080 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:09.132055 4562 scope.go:122] "RemoveContainer" containerID="399e769bcd0cff3a38573e1a9c7d015ea6a1098c2ab43b44cb47e566bed33985" Oct 30 09:32:09.132270 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:32:09.132248 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"399e769bcd0cff3a38573e1a9c7d015ea6a1098c2ab43b44cb47e566bed33985\": container with ID starting with 399e769bcd0cff3a38573e1a9c7d015ea6a1098c2ab43b44cb47e566bed33985 not found: ID does not exist" containerID="399e769bcd0cff3a38573e1a9c7d015ea6a1098c2ab43b44cb47e566bed33985" Oct 30 09:32:09.132316 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:09.132276 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"399e769bcd0cff3a38573e1a9c7d015ea6a1098c2ab43b44cb47e566bed33985"} err="failed to get container status \"399e769bcd0cff3a38573e1a9c7d015ea6a1098c2ab43b44cb47e566bed33985\": rpc error: code = NotFound desc = could not find container \"399e769bcd0cff3a38573e1a9c7d015ea6a1098c2ab43b44cb47e566bed33985\": container with ID starting with 399e769bcd0cff3a38573e1a9c7d015ea6a1098c2ab43b44cb47e566bed33985 not found: ID does not exist" Oct 30 09:32:09.175951 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:09.175914 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj"] Oct 30 09:32:09.183170 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:09.183139 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-eff9fd-openshift-default-57db6b7d5-287bj"] Oct 30 09:32:09.874847 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:09.874815 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64" path="/var/lib/kubelet/pods/f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64/volumes" Oct 30 09:32:13.972603 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:13.972529 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-19c94a-87d4f8f55-kt6zq" Oct 30 09:32:13.972603 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:13.972600 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-19c94a-69db9c4498-s4qsp" Oct 30 09:32:14.984617 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:14.975863 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-19c94a-bc94dcbd4-6k9zl"] Oct 30 09:32:14.985840 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:14.985801 4562 cpu_manager.go:414] "containerMap: removing container" podUID="f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64" containerName="istio-proxy" Oct 30 09:32:14.985840 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:14.985838 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64" containerName="istio-proxy" Oct 30 09:32:14.986071 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:14.985900 4562 cpu_manager.go:414] "containerMap: removing container" podUID="f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64" containerName="istio-proxy" Oct 30 09:32:14.986071 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:14.985908 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64" containerName="istio-proxy" Oct 30 09:32:14.986185 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:14.986073 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64" containerName="istio-proxy" Oct 30 09:32:14.991611 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:14.991586 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-19c94a-bc94dcbd4-6k9zl" Oct 30 09:32:14.995187 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:14.995154 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-19c94a-bc94dcbd4-6k9zl"] Oct 30 09:32:15.007658 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:15.007630 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-19c94a-serving-cert\"" Oct 30 09:32:15.088466 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:15.088430 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5a67e64-3b82-4bb2-b28e-67903deedd63-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-19c94a-bc94dcbd4-6k9zl\" (UID: \"b5a67e64-3b82-4bb2-b28e-67903deedd63\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-19c94a-bc94dcbd4-6k9zl" Oct 30 09:32:15.088700 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:15.088678 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx6bk\" (UniqueName: \"kubernetes.io/projected/b5a67e64-3b82-4bb2-b28e-67903deedd63-kube-api-access-mx6bk\") pod \"maas-api-e2e-shared-a-w4-19c94a-bc94dcbd4-6k9zl\" (UID: \"b5a67e64-3b82-4bb2-b28e-67903deedd63\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-19c94a-bc94dcbd4-6k9zl" Oct 30 09:32:15.088836 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:15.088816 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5a67e64-3b82-4bb2-b28e-67903deedd63-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-19c94a-bc94dcbd4-6k9zl\" (UID: \"b5a67e64-3b82-4bb2-b28e-67903deedd63\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-19c94a-bc94dcbd4-6k9zl" Oct 30 09:32:15.088932 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:15.088914 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5a67e64-3b82-4bb2-b28e-67903deedd63-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-19c94a-bc94dcbd4-6k9zl\" (UID: \"b5a67e64-3b82-4bb2-b28e-67903deedd63\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-19c94a-bc94dcbd4-6k9zl" Oct 30 09:32:15.190236 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:15.190204 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx6bk\" (UniqueName: \"kubernetes.io/projected/b5a67e64-3b82-4bb2-b28e-67903deedd63-kube-api-access-mx6bk\") pod \"maas-api-e2e-shared-a-w4-19c94a-bc94dcbd4-6k9zl\" (UID: \"b5a67e64-3b82-4bb2-b28e-67903deedd63\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-19c94a-bc94dcbd4-6k9zl" Oct 30 09:32:15.190471 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:15.190266 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5a67e64-3b82-4bb2-b28e-67903deedd63-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-19c94a-bc94dcbd4-6k9zl\" (UID: \"b5a67e64-3b82-4bb2-b28e-67903deedd63\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-19c94a-bc94dcbd4-6k9zl" Oct 30 09:32:15.190471 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:15.190303 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5a67e64-3b82-4bb2-b28e-67903deedd63-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-19c94a-bc94dcbd4-6k9zl\" (UID: \"b5a67e64-3b82-4bb2-b28e-67903deedd63\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-19c94a-bc94dcbd4-6k9zl" Oct 30 09:32:15.190471 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:15.190452 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5a67e64-3b82-4bb2-b28e-67903deedd63-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-19c94a-bc94dcbd4-6k9zl\" (UID: \"b5a67e64-3b82-4bb2-b28e-67903deedd63\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-19c94a-bc94dcbd4-6k9zl" Oct 30 09:32:15.191025 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:15.191002 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5a67e64-3b82-4bb2-b28e-67903deedd63-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-19c94a-bc94dcbd4-6k9zl\" (UID: \"b5a67e64-3b82-4bb2-b28e-67903deedd63\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-19c94a-bc94dcbd4-6k9zl" Oct 30 09:32:15.193036 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:15.193013 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5a67e64-3b82-4bb2-b28e-67903deedd63-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-19c94a-bc94dcbd4-6k9zl\" (UID: \"b5a67e64-3b82-4bb2-b28e-67903deedd63\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-19c94a-bc94dcbd4-6k9zl" Oct 30 09:32:15.193127 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:15.193064 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5a67e64-3b82-4bb2-b28e-67903deedd63-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-19c94a-bc94dcbd4-6k9zl\" (UID: \"b5a67e64-3b82-4bb2-b28e-67903deedd63\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-19c94a-bc94dcbd4-6k9zl" Oct 30 09:32:15.212042 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:15.212012 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx6bk\" (UniqueName: \"kubernetes.io/projected/b5a67e64-3b82-4bb2-b28e-67903deedd63-kube-api-access-mx6bk\") pod \"maas-api-e2e-shared-a-w4-19c94a-bc94dcbd4-6k9zl\" (UID: \"b5a67e64-3b82-4bb2-b28e-67903deedd63\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-19c94a-bc94dcbd4-6k9zl" Oct 30 09:32:15.311288 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:15.311209 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-19c94a-bc94dcbd4-6k9zl" Oct 30 09:32:15.653667 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:32:15.653630 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5a67e64_3b82_4bb2_b28e_67903deedd63.slice/crio-4055948bf9bbca6f28a0a486847799624931789a33097c6cea6895e05583f119 WatchSource:0}: Error finding container 4055948bf9bbca6f28a0a486847799624931789a33097c6cea6895e05583f119: Status 404 returned error can't find the container with id 4055948bf9bbca6f28a0a486847799624931789a33097c6cea6895e05583f119 Oct 30 09:32:15.655137 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:15.655103 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-19c94a-bc94dcbd4-6k9zl"] Oct 30 09:32:15.655631 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:15.655614 4562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 09:32:16.233908 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:16.233866 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-19c94a-bc94dcbd4-6k9zl" event={"ID":"b5a67e64-3b82-4bb2-b28e-67903deedd63","Type":"ContainerStarted","Data":"4055948bf9bbca6f28a0a486847799624931789a33097c6cea6895e05583f119"} Oct 30 09:32:17.241357 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:17.241324 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-19c94a-bc94dcbd4-6k9zl" event={"ID":"b5a67e64-3b82-4bb2-b28e-67903deedd63","Type":"ContainerStarted","Data":"5cff1b5033977b6ec0a654891935c6cf9d1569c76df0c48c548750cb95f646c6"} Oct 30 09:32:17.241788 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:17.241536 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-19c94a-bc94dcbd4-6k9zl" Oct 30 09:32:17.288854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:17.288805 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-19c94a-bc94dcbd4-6k9zl" podStartSLOduration=2.860860539 podStartE2EDuration="3.288792497s" totalImagesPullingTime="427.931958ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:32:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:32:17.287750284 +0000 UTC m=+2297.549311276" watchObservedRunningTime="2026-10-30 09:32:17.288792497 +0000 UTC m=+2297.550353485" Oct 30 09:32:19.031030 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:19.030996 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fc7ee7-5dd775bb8c-7fcnp"] Oct 30 09:32:19.031724 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:19.031709 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9ffcc00-69fb-4d92-ae62-20c4e3a3fa64" containerName="istio-proxy" Oct 30 09:32:19.035517 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:19.035495 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fc7ee7-5dd775bb8c-7fcnp" Oct 30 09:32:19.046860 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:19.046831 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-fc7ee7-6b565b88bd-c4tht"] Oct 30 09:32:19.051237 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:19.051209 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fc7ee7-6b565b88bd-c4tht" Oct 30 09:32:19.054502 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:19.054478 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fc7ee7-5dd775bb8c-7fcnp"] Oct 30 09:32:19.057965 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:19.057945 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-fc7ee7\"" Oct 30 09:32:19.079663 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:19.079635 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-fc7ee7-6b565b88bd-c4tht"] Oct 30 09:32:19.093607 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:19.093584 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-fc7ee7-dockercfg-hd9jc\"" Oct 30 09:32:19.100761 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:19.100729 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6qvb\" (UniqueName: \"kubernetes.io/projected/4a4b5a98-b1b2-41ff-9130-1f2367ccff29-kube-api-access-n6qvb\") pod \"payload-pre-processing-e2e-shared-a-w1-fc7ee7-5dd775bb8c-7fcnp\" (UID: \"4a4b5a98-b1b2-41ff-9130-1f2367ccff29\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fc7ee7-5dd775bb8c-7fcnp" Oct 30 09:32:19.100884 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:19.100864 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9d75\" (UniqueName: \"kubernetes.io/projected/e574e901-6a15-4cb8-9a91-0a96e5347b07-kube-api-access-z9d75\") pod \"payload-processing-e2e-shared-a-w1-fc7ee7-6b565b88bd-c4tht\" (UID: \"e574e901-6a15-4cb8-9a91-0a96e5347b07\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fc7ee7-6b565b88bd-c4tht" Oct 30 09:32:19.101032 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:19.101009 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a4b5a98-b1b2-41ff-9130-1f2367ccff29-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-fc7ee7-5dd775bb8c-7fcnp\" (UID: \"4a4b5a98-b1b2-41ff-9130-1f2367ccff29\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fc7ee7-5dd775bb8c-7fcnp" Oct 30 09:32:19.101196 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:19.101179 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e574e901-6a15-4cb8-9a91-0a96e5347b07-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-fc7ee7-6b565b88bd-c4tht\" (UID: \"e574e901-6a15-4cb8-9a91-0a96e5347b07\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fc7ee7-6b565b88bd-c4tht" Oct 30 09:32:19.221402 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:19.202657 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a4b5a98-b1b2-41ff-9130-1f2367ccff29-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-fc7ee7-5dd775bb8c-7fcnp\" (UID: \"4a4b5a98-b1b2-41ff-9130-1f2367ccff29\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fc7ee7-5dd775bb8c-7fcnp" Oct 30 09:32:19.221402 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:19.202834 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e574e901-6a15-4cb8-9a91-0a96e5347b07-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-fc7ee7-6b565b88bd-c4tht\" (UID: \"e574e901-6a15-4cb8-9a91-0a96e5347b07\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fc7ee7-6b565b88bd-c4tht" Oct 30 09:32:19.221402 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:19.202905 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6qvb\" (UniqueName: \"kubernetes.io/projected/4a4b5a98-b1b2-41ff-9130-1f2367ccff29-kube-api-access-n6qvb\") pod \"payload-pre-processing-e2e-shared-a-w1-fc7ee7-5dd775bb8c-7fcnp\" (UID: \"4a4b5a98-b1b2-41ff-9130-1f2367ccff29\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fc7ee7-5dd775bb8c-7fcnp" Oct 30 09:32:19.221402 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:19.202991 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9d75\" (UniqueName: \"kubernetes.io/projected/e574e901-6a15-4cb8-9a91-0a96e5347b07-kube-api-access-z9d75\") pod \"payload-processing-e2e-shared-a-w1-fc7ee7-6b565b88bd-c4tht\" (UID: \"e574e901-6a15-4cb8-9a91-0a96e5347b07\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fc7ee7-6b565b88bd-c4tht" Oct 30 09:32:19.230410 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:19.227110 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e574e901-6a15-4cb8-9a91-0a96e5347b07-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-fc7ee7-6b565b88bd-c4tht\" (UID: \"e574e901-6a15-4cb8-9a91-0a96e5347b07\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fc7ee7-6b565b88bd-c4tht" Oct 30 09:32:19.230410 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:19.228061 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a4b5a98-b1b2-41ff-9130-1f2367ccff29-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-fc7ee7-5dd775bb8c-7fcnp\" (UID: \"4a4b5a98-b1b2-41ff-9130-1f2367ccff29\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fc7ee7-5dd775bb8c-7fcnp" Oct 30 09:32:19.259172 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:19.259091 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9d75\" (UniqueName: \"kubernetes.io/projected/e574e901-6a15-4cb8-9a91-0a96e5347b07-kube-api-access-z9d75\") pod \"payload-processing-e2e-shared-a-w1-fc7ee7-6b565b88bd-c4tht\" (UID: \"e574e901-6a15-4cb8-9a91-0a96e5347b07\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fc7ee7-6b565b88bd-c4tht" Oct 30 09:32:19.291929 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:19.291848 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6qvb\" (UniqueName: \"kubernetes.io/projected/4a4b5a98-b1b2-41ff-9130-1f2367ccff29-kube-api-access-n6qvb\") pod \"payload-pre-processing-e2e-shared-a-w1-fc7ee7-5dd775bb8c-7fcnp\" (UID: \"4a4b5a98-b1b2-41ff-9130-1f2367ccff29\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fc7ee7-5dd775bb8c-7fcnp" Oct 30 09:32:19.353165 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:19.353130 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fc7ee7-5dd775bb8c-7fcnp" Oct 30 09:32:19.375623 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:19.375594 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fc7ee7-6b565b88bd-c4tht" Oct 30 09:32:19.662821 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:32:19.662791 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a4b5a98_b1b2_41ff_9130_1f2367ccff29.slice/crio-0da2b03de5e455f63613327f13467ef9f77db26a63837b557cc1a2b6d3a3ff22 WatchSource:0}: Error finding container 0da2b03de5e455f63613327f13467ef9f77db26a63837b557cc1a2b6d3a3ff22: Status 404 returned error can't find the container with id 0da2b03de5e455f63613327f13467ef9f77db26a63837b557cc1a2b6d3a3ff22 Oct 30 09:32:19.664697 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:19.664676 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fc7ee7-5dd775bb8c-7fcnp"] Oct 30 09:32:19.711270 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:32:19.711246 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode574e901_6a15_4cb8_9a91_0a96e5347b07.slice/crio-651dd78e7ab41f9cb4609d89c7d50b067146e05fbac2340199639319f3f893d0 WatchSource:0}: Error finding container 651dd78e7ab41f9cb4609d89c7d50b067146e05fbac2340199639319f3f893d0: Status 404 returned error can't find the container with id 651dd78e7ab41f9cb4609d89c7d50b067146e05fbac2340199639319f3f893d0 Oct 30 09:32:19.713279 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:19.713258 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-fc7ee7-6b565b88bd-c4tht"] Oct 30 09:32:20.277654 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:20.277610 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fc7ee7-6b565b88bd-c4tht" event={"ID":"e574e901-6a15-4cb8-9a91-0a96e5347b07","Type":"ContainerStarted","Data":"46c6d2c076349127a537c7ed7c9953d4b8cc85d63ef2bfc05ce36161a72bf1ba"} Oct 30 09:32:20.278121 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:20.277693 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fc7ee7-6b565b88bd-c4tht" Oct 30 09:32:20.278121 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:20.277721 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fc7ee7-6b565b88bd-c4tht" event={"ID":"e574e901-6a15-4cb8-9a91-0a96e5347b07","Type":"ContainerStarted","Data":"651dd78e7ab41f9cb4609d89c7d50b067146e05fbac2340199639319f3f893d0"} Oct 30 09:32:20.279577 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:20.279530 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fc7ee7-5dd775bb8c-7fcnp" event={"ID":"4a4b5a98-b1b2-41ff-9130-1f2367ccff29","Type":"ContainerStarted","Data":"7d49268029e774a7174387f49e4bb2fc72eaf6b3d44cc2182aaaa68d5c5c5a0c"} Oct 30 09:32:20.279829 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:20.279583 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fc7ee7-5dd775bb8c-7fcnp" event={"ID":"4a4b5a98-b1b2-41ff-9130-1f2367ccff29","Type":"ContainerStarted","Data":"0da2b03de5e455f63613327f13467ef9f77db26a63837b557cc1a2b6d3a3ff22"} Oct 30 09:32:20.279829 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:20.279632 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fc7ee7-5dd775bb8c-7fcnp" Oct 30 09:32:20.319833 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:20.319786 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fc7ee7-6b565b88bd-c4tht" podStartSLOduration=1.319773737 podStartE2EDuration="1.319773737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:32:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:32:20.318575394 +0000 UTC m=+2300.580136398" watchObservedRunningTime="2026-10-30 09:32:20.319773737 +0000 UTC m=+2300.581334725" Oct 30 09:32:20.358276 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:20.358220 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fc7ee7-5dd775bb8c-7fcnp" podStartSLOduration=1.35820618 podStartE2EDuration="1.35820618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:32:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:32:20.357973802 +0000 UTC m=+2300.619534811" watchObservedRunningTime="2026-10-30 09:32:20.35820618 +0000 UTC m=+2300.619767167" Oct 30 09:32:21.732358 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:21.732322 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb"] Oct 30 09:32:21.736823 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:21.736794 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb" Oct 30 09:32:21.761866 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:21.761835 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb"] Oct 30 09:32:21.854300 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:21.854268 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94202efb-a6b4-4260-bd94-16473a214a5c-istiod-ca-cert\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb\" (UID: \"94202efb-a6b4-4260-bd94-16473a214a5c\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb" Oct 30 09:32:21.854500 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:21.854310 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94202efb-a6b4-4260-bd94-16473a214a5c-istio-podinfo\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb\" (UID: \"94202efb-a6b4-4260-bd94-16473a214a5c\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb" Oct 30 09:32:21.854500 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:21.854465 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9rv7\" (UniqueName: \"kubernetes.io/projected/94202efb-a6b4-4260-bd94-16473a214a5c-kube-api-access-x9rv7\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb\" (UID: \"94202efb-a6b4-4260-bd94-16473a214a5c\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb" Oct 30 09:32:21.854602 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:21.854552 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94202efb-a6b4-4260-bd94-16473a214a5c-istio-token\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb\" (UID: \"94202efb-a6b4-4260-bd94-16473a214a5c\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb" Oct 30 09:32:21.854640 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:21.854620 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94202efb-a6b4-4260-bd94-16473a214a5c-istio-data\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb\" (UID: \"94202efb-a6b4-4260-bd94-16473a214a5c\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb" Oct 30 09:32:21.854755 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:21.854742 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94202efb-a6b4-4260-bd94-16473a214a5c-workload-certs\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb\" (UID: \"94202efb-a6b4-4260-bd94-16473a214a5c\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb" Oct 30 09:32:21.854812 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:21.854779 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94202efb-a6b4-4260-bd94-16473a214a5c-credential-socket\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb\" (UID: \"94202efb-a6b4-4260-bd94-16473a214a5c\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb" Oct 30 09:32:21.854812 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:21.854800 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94202efb-a6b4-4260-bd94-16473a214a5c-istio-envoy\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb\" (UID: \"94202efb-a6b4-4260-bd94-16473a214a5c\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb" Oct 30 09:32:21.855024 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:21.855006 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94202efb-a6b4-4260-bd94-16473a214a5c-workload-socket\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb\" (UID: \"94202efb-a6b4-4260-bd94-16473a214a5c\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb" Oct 30 09:32:21.956412 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:21.956349 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94202efb-a6b4-4260-bd94-16473a214a5c-workload-certs\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb\" (UID: \"94202efb-a6b4-4260-bd94-16473a214a5c\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb" Oct 30 09:32:21.956607 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:21.956429 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94202efb-a6b4-4260-bd94-16473a214a5c-credential-socket\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb\" (UID: \"94202efb-a6b4-4260-bd94-16473a214a5c\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb" Oct 30 09:32:21.956607 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:21.956448 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94202efb-a6b4-4260-bd94-16473a214a5c-istio-envoy\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb\" (UID: \"94202efb-a6b4-4260-bd94-16473a214a5c\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb" Oct 30 09:32:21.956607 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:21.956535 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94202efb-a6b4-4260-bd94-16473a214a5c-workload-socket\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb\" (UID: \"94202efb-a6b4-4260-bd94-16473a214a5c\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb" Oct 30 09:32:21.956607 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:21.956570 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94202efb-a6b4-4260-bd94-16473a214a5c-istiod-ca-cert\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb\" (UID: \"94202efb-a6b4-4260-bd94-16473a214a5c\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb" Oct 30 09:32:21.956607 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:21.956601 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94202efb-a6b4-4260-bd94-16473a214a5c-istio-podinfo\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb\" (UID: \"94202efb-a6b4-4260-bd94-16473a214a5c\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb" Oct 30 09:32:21.956897 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:21.956662 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9rv7\" (UniqueName: \"kubernetes.io/projected/94202efb-a6b4-4260-bd94-16473a214a5c-kube-api-access-x9rv7\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb\" (UID: \"94202efb-a6b4-4260-bd94-16473a214a5c\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb" Oct 30 09:32:21.956897 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:21.956701 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94202efb-a6b4-4260-bd94-16473a214a5c-istio-token\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb\" (UID: \"94202efb-a6b4-4260-bd94-16473a214a5c\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb" Oct 30 09:32:21.956897 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:21.956742 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94202efb-a6b4-4260-bd94-16473a214a5c-istio-data\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb\" (UID: \"94202efb-a6b4-4260-bd94-16473a214a5c\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb" Oct 30 09:32:21.956897 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:21.956816 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94202efb-a6b4-4260-bd94-16473a214a5c-workload-certs\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb\" (UID: \"94202efb-a6b4-4260-bd94-16473a214a5c\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb" Oct 30 09:32:21.956897 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:21.956866 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94202efb-a6b4-4260-bd94-16473a214a5c-credential-socket\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb\" (UID: \"94202efb-a6b4-4260-bd94-16473a214a5c\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb" Oct 30 09:32:21.957140 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:21.956901 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94202efb-a6b4-4260-bd94-16473a214a5c-workload-socket\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb\" (UID: \"94202efb-a6b4-4260-bd94-16473a214a5c\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb" Oct 30 09:32:21.957198 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:21.957162 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94202efb-a6b4-4260-bd94-16473a214a5c-istio-data\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb\" (UID: \"94202efb-a6b4-4260-bd94-16473a214a5c\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb" Oct 30 09:32:21.957423 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:21.957395 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94202efb-a6b4-4260-bd94-16473a214a5c-istiod-ca-cert\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb\" (UID: \"94202efb-a6b4-4260-bd94-16473a214a5c\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb" Oct 30 09:32:21.958929 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:21.958904 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94202efb-a6b4-4260-bd94-16473a214a5c-istio-envoy\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb\" (UID: \"94202efb-a6b4-4260-bd94-16473a214a5c\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb" Oct 30 09:32:21.959277 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:21.959256 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94202efb-a6b4-4260-bd94-16473a214a5c-istio-podinfo\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb\" (UID: \"94202efb-a6b4-4260-bd94-16473a214a5c\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb" Oct 30 09:32:21.976845 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:21.976821 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94202efb-a6b4-4260-bd94-16473a214a5c-istio-token\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb\" (UID: \"94202efb-a6b4-4260-bd94-16473a214a5c\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb" Oct 30 09:32:21.977235 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:21.977217 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9rv7\" (UniqueName: \"kubernetes.io/projected/94202efb-a6b4-4260-bd94-16473a214a5c-kube-api-access-x9rv7\") pod \"e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb\" (UID: \"94202efb-a6b4-4260-bd94-16473a214a5c\") " pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb" Oct 30 09:32:22.055489 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:22.055373 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb" Oct 30 09:32:22.396898 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:32:22.396857 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94202efb_a6b4_4260_bd94_16473a214a5c.slice/crio-27dfba87591a74416c61b8a66b8da0f83046881f0936fcc31aee8ff6072f7499 WatchSource:0}: Error finding container 27dfba87591a74416c61b8a66b8da0f83046881f0936fcc31aee8ff6072f7499: Status 404 returned error can't find the container with id 27dfba87591a74416c61b8a66b8da0f83046881f0936fcc31aee8ff6072f7499 Oct 30 09:32:22.398881 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:22.398854 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb"] Oct 30 09:32:22.399478 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:22.399450 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:32:22.399533 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:22.399512 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:32:22.399570 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:22.399542 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:32:23.303407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:23.299568 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-19c94a-bc94dcbd4-6k9zl" Oct 30 09:32:23.334810 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:23.334770 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb" event={"ID":"94202efb-a6b4-4260-bd94-16473a214a5c","Type":"ContainerStarted","Data":"101f0b8ae7a931c275b34237e93dca0566e73cdd34f4889ea503aeac9911e3a0"} Oct 30 09:32:23.335008 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:23.334881 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb" event={"ID":"94202efb-a6b4-4260-bd94-16473a214a5c","Type":"ContainerStarted","Data":"27dfba87591a74416c61b8a66b8da0f83046881f0936fcc31aee8ff6072f7499"} Oct 30 09:32:23.421497 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:23.421422 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb" podStartSLOduration=2.421401951 podStartE2EDuration="2.421401951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:32:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:32:23.414841715 +0000 UTC m=+2303.676402706" watchObservedRunningTime="2026-10-30 09:32:23.421401951 +0000 UTC m=+2303.682962940" Oct 30 09:32:24.058405 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:24.057664 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb" Oct 30 09:32:24.062403 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:24.062266 4562 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Oct 30 09:32:24.062403 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:24.062348 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb" podUID="94202efb-a6b4-4260-bd94-16473a214a5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Oct 30 09:32:25.058333 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:25.057606 4562 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Oct 30 09:32:25.058333 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:25.057666 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb" podUID="94202efb-a6b4-4260-bd94-16473a214a5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Oct 30 09:32:26.064406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:26.058458 4562 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Oct 30 09:32:26.064406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:26.058534 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb" podUID="94202efb-a6b4-4260-bd94-16473a214a5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Oct 30 09:32:27.058415 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:27.057414 4562 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Oct 30 09:32:27.058415 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:27.057473 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb" podUID="94202efb-a6b4-4260-bd94-16473a214a5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Oct 30 09:32:27.869088 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:27.868881 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2"] Oct 30 09:32:27.869712 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:27.869675 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2" podUID="162a9d13-c5b1-4752-91e7-3b19049a4582" containerName="istio-proxy" containerID="cri-o://e3750ac93490e3484e232029cea39ffa2e3a02ac137b43d723cdbdbae320ca0d" gracePeriod=30 Oct 30 09:32:28.070025 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:28.069226 4562 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Oct 30 09:32:28.070025 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:28.069298 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb" podUID="94202efb-a6b4-4260-bd94-16473a214a5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Oct 30 09:32:28.601036 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:32:28.600986 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:32:29.056151 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:29.056112 4562 patch_prober.go:29] interesting pod/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Oct 30 09:32:29.056568 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:29.056189 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb" podUID="94202efb-a6b4-4260-bd94-16473a214a5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Oct 30 09:32:30.142776 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:30.142732 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb" Oct 30 09:32:30.143313 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:30.143131 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb" Oct 30 09:32:30.144142 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:30.144120 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb" Oct 30 09:32:31.288976 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:31.288946 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fc7ee7-5dd775bb8c-7fcnp" Oct 30 09:32:31.304043 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:31.303977 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fc7ee7-6b565b88bd-c4tht" Oct 30 09:32:33.718405 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:33.718300 4562 generic.go:356] "Generic (PLEG): container finished" podID="162a9d13-c5b1-4752-91e7-3b19049a4582" containerID="e3750ac93490e3484e232029cea39ffa2e3a02ac137b43d723cdbdbae320ca0d" exitCode=0 Oct 30 09:32:33.779962 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:33.749429 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2" event={"ID":"162a9d13-c5b1-4752-91e7-3b19049a4582","Type":"ContainerDied","Data":"e3750ac93490e3484e232029cea39ffa2e3a02ac137b43d723cdbdbae320ca0d"} Oct 30 09:32:33.779962 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:33.749527 4562 scope.go:122] "RemoveContainer" containerID="c32877b05aef6dfc4183c5146c9c45c43715d14a9be1b5ade2b2746c3980f2b2" Oct 30 09:32:33.839636 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:33.835299 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-176b6c/distinct-w3-176b6c-kserve-f57b7579c-k4pc5" podUID="28fd4072-399c-4be5-a0d8-1be6966da4af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:32:33.884406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:33.856692 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55" podUID="88be34c3-a9e9-4489-a449-206a99ef1663" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:32:34.425873 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:34.425809 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2" Oct 30 09:32:34.517902 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:34.517866 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/162a9d13-c5b1-4752-91e7-3b19049a4582-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/162a9d13-c5b1-4752-91e7-3b19049a4582-istio-data\") pod \"162a9d13-c5b1-4752-91e7-3b19049a4582\" (UID: \"162a9d13-c5b1-4752-91e7-3b19049a4582\") " Oct 30 09:32:34.520036 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:34.518175 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/162a9d13-c5b1-4752-91e7-3b19049a4582-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/162a9d13-c5b1-4752-91e7-3b19049a4582-istiod-ca-cert\") pod \"162a9d13-c5b1-4752-91e7-3b19049a4582\" (UID: \"162a9d13-c5b1-4752-91e7-3b19049a4582\") " Oct 30 09:32:34.520283 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:34.518294 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/162a9d13-c5b1-4752-91e7-3b19049a4582-istio-data" pod "162a9d13-c5b1-4752-91e7-3b19049a4582" (UID: "162a9d13-c5b1-4752-91e7-3b19049a4582"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:32:34.520411 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:34.518822 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/162a9d13-c5b1-4752-91e7-3b19049a4582-istiod-ca-cert" pod "162a9d13-c5b1-4752-91e7-3b19049a4582" (UID: "162a9d13-c5b1-4752-91e7-3b19049a4582"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:32:34.520619 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:34.520599 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/162a9d13-c5b1-4752-91e7-3b19049a4582-istio-token\" (UniqueName: \"kubernetes.io/projected/162a9d13-c5b1-4752-91e7-3b19049a4582-istio-token\") pod \"162a9d13-c5b1-4752-91e7-3b19049a4582\" (UID: \"162a9d13-c5b1-4752-91e7-3b19049a4582\") " Oct 30 09:32:34.521099 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:34.521078 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/162a9d13-c5b1-4752-91e7-3b19049a4582-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/162a9d13-c5b1-4752-91e7-3b19049a4582-istio-podinfo\") pod \"162a9d13-c5b1-4752-91e7-3b19049a4582\" (UID: \"162a9d13-c5b1-4752-91e7-3b19049a4582\") " Oct 30 09:32:34.521325 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:34.521305 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/162a9d13-c5b1-4752-91e7-3b19049a4582-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/162a9d13-c5b1-4752-91e7-3b19049a4582-workload-certs\") pod \"162a9d13-c5b1-4752-91e7-3b19049a4582\" (UID: \"162a9d13-c5b1-4752-91e7-3b19049a4582\") " Oct 30 09:32:34.523137 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:34.523018 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/162a9d13-c5b1-4752-91e7-3b19049a4582-workload-certs" pod "162a9d13-c5b1-4752-91e7-3b19049a4582" (UID: "162a9d13-c5b1-4752-91e7-3b19049a4582"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:32:34.523340 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:34.523322 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/162a9d13-c5b1-4752-91e7-3b19049a4582-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/162a9d13-c5b1-4752-91e7-3b19049a4582-istio-envoy\") pod \"162a9d13-c5b1-4752-91e7-3b19049a4582\" (UID: \"162a9d13-c5b1-4752-91e7-3b19049a4582\") " Oct 30 09:32:34.523915 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:34.523895 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/162a9d13-c5b1-4752-91e7-3b19049a4582-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/162a9d13-c5b1-4752-91e7-3b19049a4582-workload-socket\") pod \"162a9d13-c5b1-4752-91e7-3b19049a4582\" (UID: \"162a9d13-c5b1-4752-91e7-3b19049a4582\") " Oct 30 09:32:34.524324 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:34.524224 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/162a9d13-c5b1-4752-91e7-3b19049a4582-workload-socket" pod "162a9d13-c5b1-4752-91e7-3b19049a4582" (UID: "162a9d13-c5b1-4752-91e7-3b19049a4582"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:32:34.524475 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:34.524301 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/162a9d13-c5b1-4752-91e7-3b19049a4582-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/162a9d13-c5b1-4752-91e7-3b19049a4582-credential-socket\") pod \"162a9d13-c5b1-4752-91e7-3b19049a4582\" (UID: \"162a9d13-c5b1-4752-91e7-3b19049a4582\") " Oct 30 09:32:34.524668 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:34.524650 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/162a9d13-c5b1-4752-91e7-3b19049a4582-kube-api-access-tv9zw\" (UniqueName: \"kubernetes.io/projected/162a9d13-c5b1-4752-91e7-3b19049a4582-kube-api-access-tv9zw\") pod \"162a9d13-c5b1-4752-91e7-3b19049a4582\" (UID: \"162a9d13-c5b1-4752-91e7-3b19049a4582\") " Oct 30 09:32:34.525746 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:34.525712 4562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/162a9d13-c5b1-4752-91e7-3b19049a4582-istio-data\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:32:34.525886 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:34.525871 4562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/162a9d13-c5b1-4752-91e7-3b19049a4582-istiod-ca-cert\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:32:34.525991 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:34.525979 4562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/162a9d13-c5b1-4752-91e7-3b19049a4582-workload-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:32:34.526090 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:34.526076 4562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/162a9d13-c5b1-4752-91e7-3b19049a4582-workload-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:32:34.527662 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:34.527641 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/162a9d13-c5b1-4752-91e7-3b19049a4582-credential-socket" pod "162a9d13-c5b1-4752-91e7-3b19049a4582" (UID: "162a9d13-c5b1-4752-91e7-3b19049a4582"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:32:34.529149 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:34.529120 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/162a9d13-c5b1-4752-91e7-3b19049a4582-istio-envoy" pod "162a9d13-c5b1-4752-91e7-3b19049a4582" (UID: "162a9d13-c5b1-4752-91e7-3b19049a4582"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:32:34.529284 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:34.529261 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/162a9d13-c5b1-4752-91e7-3b19049a4582-kube-api-access-tv9zw" pod "162a9d13-c5b1-4752-91e7-3b19049a4582" (UID: "162a9d13-c5b1-4752-91e7-3b19049a4582"). InnerVolumeSpecName "kube-api-access-tv9zw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:32:34.530666 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:34.529673 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/162a9d13-c5b1-4752-91e7-3b19049a4582-istio-podinfo" pod "162a9d13-c5b1-4752-91e7-3b19049a4582" (UID: "162a9d13-c5b1-4752-91e7-3b19049a4582"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:32:34.533010 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:34.532978 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/162a9d13-c5b1-4752-91e7-3b19049a4582-istio-token" pod "162a9d13-c5b1-4752-91e7-3b19049a4582" (UID: "162a9d13-c5b1-4752-91e7-3b19049a4582"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:32:34.637374 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:34.637257 4562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/162a9d13-c5b1-4752-91e7-3b19049a4582-istio-token\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:32:34.637374 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:34.637301 4562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/162a9d13-c5b1-4752-91e7-3b19049a4582-istio-podinfo\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:32:34.637374 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:34.637314 4562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/162a9d13-c5b1-4752-91e7-3b19049a4582-istio-envoy\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:32:34.637374 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:34.637328 4562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/162a9d13-c5b1-4752-91e7-3b19049a4582-credential-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:32:34.637374 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:34.637342 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tv9zw\" (UniqueName: \"kubernetes.io/projected/162a9d13-c5b1-4752-91e7-3b19049a4582-kube-api-access-tv9zw\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:32:34.655498 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:34.655457 4562 patch_prober.go:29] interesting pod/e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 09:32:34.656114 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:34.655826 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2" podUID="162a9d13-c5b1-4752-91e7-3b19049a4582" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 09:32:34.811756 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:34.811724 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2" event={"ID":"162a9d13-c5b1-4752-91e7-3b19049a4582","Type":"ContainerDied","Data":"2fd1179b892bbee0b5cabc9ecab5c7447cd5bcc624eba39c7405bdfc9514e32e"} Oct 30 09:32:34.812242 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:34.812225 4562 scope.go:122] "RemoveContainer" containerID="e3750ac93490e3484e232029cea39ffa2e3a02ac137b43d723cdbdbae320ca0d" Oct 30 09:32:34.812395 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:34.811772 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2" Oct 30 09:32:34.931540 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:34.931511 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-d87c85/ipp-route-d87c85-kserve-7cc5cb6f7c-qk4q7"] Oct 30 09:32:34.974861 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:34.974830 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2"] Oct 30 09:32:34.980688 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:34.980666 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-066f95-openshift-default-7fd777976f-zdlz2"] Oct 30 09:32:35.917403 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:35.915363 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="162a9d13-c5b1-4752-91e7-3b19049a4582" path="/var/lib/kubelet/pods/162a9d13-c5b1-4752-91e7-3b19049a4582/volumes" Oct 30 09:32:41.046347 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:41.046307 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb1a2a-5d7444785b-h7zjw"] Oct 30 09:32:41.047031 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:41.047012 4562 cpu_manager.go:414] "containerMap: removing container" podUID="162a9d13-c5b1-4752-91e7-3b19049a4582" containerName="istio-proxy" Oct 30 09:32:41.047125 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:41.047033 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="162a9d13-c5b1-4752-91e7-3b19049a4582" containerName="istio-proxy" Oct 30 09:32:41.047125 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:41.047057 4562 cpu_manager.go:414] "containerMap: removing container" podUID="162a9d13-c5b1-4752-91e7-3b19049a4582" containerName="istio-proxy" Oct 30 09:32:41.047125 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:41.047062 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="162a9d13-c5b1-4752-91e7-3b19049a4582" containerName="istio-proxy" Oct 30 09:32:41.047290 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:41.047175 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="162a9d13-c5b1-4752-91e7-3b19049a4582" containerName="istio-proxy" Oct 30 09:32:41.047290 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:41.047189 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="162a9d13-c5b1-4752-91e7-3b19049a4582" containerName="istio-proxy" Oct 30 09:32:41.052490 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:41.052461 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb1a2a-5d7444785b-h7zjw" Oct 30 09:32:41.088891 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:41.088532 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-fb1a2a-serving-cert\"" Oct 30 09:32:41.092431 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:41.092404 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb1a2a-5d7444785b-h7zjw"] Oct 30 09:32:41.098317 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:41.098280 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a163089-8bc6-4fde-afbb-7970c8ae3a08-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-fb1a2a-5d7444785b-h7zjw\" (UID: \"7a163089-8bc6-4fde-afbb-7970c8ae3a08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb1a2a-5d7444785b-h7zjw" Oct 30 09:32:41.098516 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:41.098426 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cvg6\" (UniqueName: \"kubernetes.io/projected/7a163089-8bc6-4fde-afbb-7970c8ae3a08-kube-api-access-9cvg6\") pod \"maas-api-e2e-shared-a-w2-fb1a2a-5d7444785b-h7zjw\" (UID: \"7a163089-8bc6-4fde-afbb-7970c8ae3a08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb1a2a-5d7444785b-h7zjw" Oct 30 09:32:41.098516 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:41.098476 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a163089-8bc6-4fde-afbb-7970c8ae3a08-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-fb1a2a-5d7444785b-h7zjw\" (UID: \"7a163089-8bc6-4fde-afbb-7970c8ae3a08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb1a2a-5d7444785b-h7zjw" Oct 30 09:32:41.098657 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:41.098589 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a163089-8bc6-4fde-afbb-7970c8ae3a08-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-fb1a2a-5d7444785b-h7zjw\" (UID: \"7a163089-8bc6-4fde-afbb-7970c8ae3a08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb1a2a-5d7444785b-h7zjw" Oct 30 09:32:41.200622 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:41.200575 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a163089-8bc6-4fde-afbb-7970c8ae3a08-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-fb1a2a-5d7444785b-h7zjw\" (UID: \"7a163089-8bc6-4fde-afbb-7970c8ae3a08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb1a2a-5d7444785b-h7zjw" Oct 30 09:32:41.200818 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:41.200661 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cvg6\" (UniqueName: \"kubernetes.io/projected/7a163089-8bc6-4fde-afbb-7970c8ae3a08-kube-api-access-9cvg6\") pod \"maas-api-e2e-shared-a-w2-fb1a2a-5d7444785b-h7zjw\" (UID: \"7a163089-8bc6-4fde-afbb-7970c8ae3a08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb1a2a-5d7444785b-h7zjw" Oct 30 09:32:41.200818 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:41.200710 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a163089-8bc6-4fde-afbb-7970c8ae3a08-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-fb1a2a-5d7444785b-h7zjw\" (UID: \"7a163089-8bc6-4fde-afbb-7970c8ae3a08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb1a2a-5d7444785b-h7zjw" Oct 30 09:32:41.200818 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:41.200803 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a163089-8bc6-4fde-afbb-7970c8ae3a08-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-fb1a2a-5d7444785b-h7zjw\" (UID: \"7a163089-8bc6-4fde-afbb-7970c8ae3a08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb1a2a-5d7444785b-h7zjw" Oct 30 09:32:41.201556 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:41.201513 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a163089-8bc6-4fde-afbb-7970c8ae3a08-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-fb1a2a-5d7444785b-h7zjw\" (UID: \"7a163089-8bc6-4fde-afbb-7970c8ae3a08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb1a2a-5d7444785b-h7zjw" Oct 30 09:32:41.203511 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:41.203486 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a163089-8bc6-4fde-afbb-7970c8ae3a08-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-fb1a2a-5d7444785b-h7zjw\" (UID: \"7a163089-8bc6-4fde-afbb-7970c8ae3a08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb1a2a-5d7444785b-h7zjw" Oct 30 09:32:41.203682 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:41.203660 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a163089-8bc6-4fde-afbb-7970c8ae3a08-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-fb1a2a-5d7444785b-h7zjw\" (UID: \"7a163089-8bc6-4fde-afbb-7970c8ae3a08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb1a2a-5d7444785b-h7zjw" Oct 30 09:32:41.228630 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:41.228599 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cvg6\" (UniqueName: \"kubernetes.io/projected/7a163089-8bc6-4fde-afbb-7970c8ae3a08-kube-api-access-9cvg6\") pod \"maas-api-e2e-shared-a-w2-fb1a2a-5d7444785b-h7zjw\" (UID: \"7a163089-8bc6-4fde-afbb-7970c8ae3a08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb1a2a-5d7444785b-h7zjw" Oct 30 09:32:41.381062 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:41.380979 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb1a2a-5d7444785b-h7zjw" Oct 30 09:32:41.706241 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:41.704588 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb1a2a-5d7444785b-h7zjw"] Oct 30 09:32:41.912021 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:41.911989 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb1a2a-5d7444785b-h7zjw" event={"ID":"7a163089-8bc6-4fde-afbb-7970c8ae3a08","Type":"ContainerStarted","Data":"8c8b8ce21f8bacbd78fe4ba1aa67f024900d87123cdd238867f8036bc0f148fe"} Oct 30 09:32:42.862347 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:42.862319 4562 patch_prober.go:29] interesting pod/e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Oct 30 09:32:42.862741 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:42.862376 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt" podUID="5f848cb2-5cdc-4f31-b063-8745991865de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Oct 30 09:32:42.919536 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:42.919497 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb1a2a-5d7444785b-h7zjw" event={"ID":"7a163089-8bc6-4fde-afbb-7970c8ae3a08","Type":"ContainerStarted","Data":"116268acfca5119aac2a934f021d97f6f5ad58b1259c3406dd473cfffd34e180"} Oct 30 09:32:42.919675 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:42.919571 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb1a2a-5d7444785b-h7zjw" Oct 30 09:32:42.921278 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:42.921255 4562 generic.go:356] "Generic (PLEG): container finished" podID="5f848cb2-5cdc-4f31-b063-8745991865de" containerID="eadf17790c84ef362a51c2eaa13c23aa51452a48c29fc5918c8c6f02ba3eddf2" exitCode=0 Oct 30 09:32:42.921409 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:42.921323 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt" event={"ID":"5f848cb2-5cdc-4f31-b063-8745991865de","Type":"ContainerDied","Data":"eadf17790c84ef362a51c2eaa13c23aa51452a48c29fc5918c8c6f02ba3eddf2"} Oct 30 09:32:42.921756 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:42.921739 4562 scope.go:122] "RemoveContainer" containerID="eadf17790c84ef362a51c2eaa13c23aa51452a48c29fc5918c8c6f02ba3eddf2" Oct 30 09:32:42.923142 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:42.923111 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:32:42.923244 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:42.923197 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:32:42.923304 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:42.923247 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:32:42.964912 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:42.964828 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb1a2a-5d7444785b-h7zjw" podStartSLOduration=1.558174736 podStartE2EDuration="1.964808259s" totalImagesPullingTime="406.633523ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:32:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:32:42.963536223 +0000 UTC m=+2323.225097258" watchObservedRunningTime="2026-10-30 09:32:42.964808259 +0000 UTC m=+2323.226369248" Oct 30 09:32:43.929784 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:43.929742 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt" event={"ID":"5f848cb2-5cdc-4f31-b063-8745991865de","Type":"ContainerStarted","Data":"a5fa224cbc089dbbe3f234b9afb2a1efa705b8512aba57382d3769ee79135d20"} Oct 30 09:32:44.562404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:44.561836 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt" Oct 30 09:32:44.562404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:44.561968 4562 patch_prober.go:29] interesting pod/e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Oct 30 09:32:44.562404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:44.562011 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt" podUID="5f848cb2-5cdc-4f31-b063-8745991865de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Oct 30 09:32:45.559409 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:45.552754 4562 patch_prober.go:29] interesting pod/e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Oct 30 09:32:45.559409 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:45.552826 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt" podUID="5f848cb2-5cdc-4f31-b063-8745991865de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Oct 30 09:32:46.560407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:46.552825 4562 patch_prober.go:29] interesting pod/e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Oct 30 09:32:46.560407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:46.552895 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt" podUID="5f848cb2-5cdc-4f31-b063-8745991865de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Oct 30 09:32:47.563409 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:47.559821 4562 patch_prober.go:29] interesting pod/e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Oct 30 09:32:47.563409 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:47.559905 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt" podUID="5f848cb2-5cdc-4f31-b063-8745991865de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Oct 30 09:32:48.555406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:48.553743 4562 patch_prober.go:29] interesting pod/e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Oct 30 09:32:48.555406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:48.553808 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt" podUID="5f848cb2-5cdc-4f31-b063-8745991865de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Oct 30 09:32:48.935959 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:48.935882 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb1a2a-5d7444785b-h7zjw" Oct 30 09:32:49.557302 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:49.557270 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt" Oct 30 09:32:49.557676 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:49.557654 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt" Oct 30 09:32:49.558268 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:49.558250 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt" Oct 30 09:32:49.987536 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:49.987498 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-d87c85/ipp-route-d87c85-kserve-7cc5cb6f7c-qk4q7" podUID="9bfc138b-8938-473c-adfe-052d9a722233" containerName="main" containerID="cri-o://b6b5cb03d0e4f89c74177e9a690093f3b032a244561b16401f52900e43de085f" gracePeriod=45 Oct 30 09:32:50.265166 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:50.265137 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-d87c85/ipp-route-d87c85-kserve-7cc5cb6f7c-qk4q7" Oct 30 09:32:50.330883 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:50.330851 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bfc138b-8938-473c-adfe-052d9a722233-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bfc138b-8938-473c-adfe-052d9a722233-dshm\") pod \"9bfc138b-8938-473c-adfe-052d9a722233\" (UID: \"9bfc138b-8938-473c-adfe-052d9a722233\") " Oct 30 09:32:50.331068 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:50.330892 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bfc138b-8938-473c-adfe-052d9a722233-kube-api-access-2mdp8\" (UniqueName: \"kubernetes.io/projected/9bfc138b-8938-473c-adfe-052d9a722233-kube-api-access-2mdp8\") pod \"9bfc138b-8938-473c-adfe-052d9a722233\" (UID: \"9bfc138b-8938-473c-adfe-052d9a722233\") " Oct 30 09:32:50.331068 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:50.330960 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bfc138b-8938-473c-adfe-052d9a722233-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bfc138b-8938-473c-adfe-052d9a722233-tmp-dir\") pod \"9bfc138b-8938-473c-adfe-052d9a722233\" (UID: \"9bfc138b-8938-473c-adfe-052d9a722233\") " Oct 30 09:32:50.331215 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:50.331068 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bfc138b-8938-473c-adfe-052d9a722233-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bfc138b-8938-473c-adfe-052d9a722233-model-cache\") pod \"9bfc138b-8938-473c-adfe-052d9a722233\" (UID: \"9bfc138b-8938-473c-adfe-052d9a722233\") " Oct 30 09:32:50.331215 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:50.331143 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bfc138b-8938-473c-adfe-052d9a722233-home\" (UniqueName: \"kubernetes.io/empty-dir/9bfc138b-8938-473c-adfe-052d9a722233-home\") pod \"9bfc138b-8938-473c-adfe-052d9a722233\" (UID: \"9bfc138b-8938-473c-adfe-052d9a722233\") " Oct 30 09:32:50.331330 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:50.331253 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bfc138b-8938-473c-adfe-052d9a722233-tls-certs\" (UniqueName: \"kubernetes.io/secret/9bfc138b-8938-473c-adfe-052d9a722233-tls-certs\") pod \"9bfc138b-8938-473c-adfe-052d9a722233\" (UID: \"9bfc138b-8938-473c-adfe-052d9a722233\") " Oct 30 09:32:50.331587 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:50.331427 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bfc138b-8938-473c-adfe-052d9a722233-model-cache" pod "9bfc138b-8938-473c-adfe-052d9a722233" (UID: "9bfc138b-8938-473c-adfe-052d9a722233"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:32:50.331804 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:50.331741 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bfc138b-8938-473c-adfe-052d9a722233-home" pod "9bfc138b-8938-473c-adfe-052d9a722233" (UID: "9bfc138b-8938-473c-adfe-052d9a722233"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:32:50.331978 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:50.331806 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bfc138b-8938-473c-adfe-052d9a722233-tmp-dir" pod "9bfc138b-8938-473c-adfe-052d9a722233" (UID: "9bfc138b-8938-473c-adfe-052d9a722233"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:32:50.331978 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:50.331964 4562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bfc138b-8938-473c-adfe-052d9a722233-model-cache\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:32:50.333517 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:50.333492 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bfc138b-8938-473c-adfe-052d9a722233-dshm" pod "9bfc138b-8938-473c-adfe-052d9a722233" (UID: "9bfc138b-8938-473c-adfe-052d9a722233"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:32:50.333796 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:50.333776 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bfc138b-8938-473c-adfe-052d9a722233-tls-certs" pod "9bfc138b-8938-473c-adfe-052d9a722233" (UID: "9bfc138b-8938-473c-adfe-052d9a722233"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:32:50.334002 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:50.333977 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bfc138b-8938-473c-adfe-052d9a722233-kube-api-access-2mdp8" pod "9bfc138b-8938-473c-adfe-052d9a722233" (UID: "9bfc138b-8938-473c-adfe-052d9a722233"). InnerVolumeSpecName "kube-api-access-2mdp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:32:50.433093 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:50.433061 4562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bfc138b-8938-473c-adfe-052d9a722233-dshm\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:32:50.433093 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:50.433092 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mdp8\" (UniqueName: \"kubernetes.io/projected/9bfc138b-8938-473c-adfe-052d9a722233-kube-api-access-2mdp8\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:32:50.433292 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:50.433104 4562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bfc138b-8938-473c-adfe-052d9a722233-tmp-dir\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:32:50.433292 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:50.433114 4562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bfc138b-8938-473c-adfe-052d9a722233-home\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:32:50.433292 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:50.433122 4562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bfc138b-8938-473c-adfe-052d9a722233-tls-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:32:51.040768 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:51.040736 4562 generic.go:356] "Generic (PLEG): container finished" podID="9bfc138b-8938-473c-adfe-052d9a722233" containerID="b6b5cb03d0e4f89c74177e9a690093f3b032a244561b16401f52900e43de085f" exitCode=0 Oct 30 09:32:51.041261 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:51.040799 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-d87c85/ipp-route-d87c85-kserve-7cc5cb6f7c-qk4q7" Oct 30 09:32:51.041261 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:51.040901 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-d87c85/ipp-route-d87c85-kserve-7cc5cb6f7c-qk4q7" event={"ID":"9bfc138b-8938-473c-adfe-052d9a722233","Type":"ContainerDied","Data":"b6b5cb03d0e4f89c74177e9a690093f3b032a244561b16401f52900e43de085f"} Oct 30 09:32:51.041261 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:51.040927 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-d87c85/ipp-route-d87c85-kserve-7cc5cb6f7c-qk4q7" event={"ID":"9bfc138b-8938-473c-adfe-052d9a722233","Type":"ContainerDied","Data":"e8065c2a4f305b2f784a60ae2e7cc0c55cb3bc4f5c99ceebe967e6fe986f123e"} Oct 30 09:32:51.041261 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:51.040946 4562 scope.go:122] "RemoveContainer" containerID="b6b5cb03d0e4f89c74177e9a690093f3b032a244561b16401f52900e43de085f" Oct 30 09:32:51.052865 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:51.052844 4562 scope.go:122] "RemoveContainer" containerID="b6b5cb03d0e4f89c74177e9a690093f3b032a244561b16401f52900e43de085f" Oct 30 09:32:51.053157 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:32:51.053136 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6b5cb03d0e4f89c74177e9a690093f3b032a244561b16401f52900e43de085f\": container with ID starting with b6b5cb03d0e4f89c74177e9a690093f3b032a244561b16401f52900e43de085f not found: ID does not exist" containerID="b6b5cb03d0e4f89c74177e9a690093f3b032a244561b16401f52900e43de085f" Oct 30 09:32:51.053230 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:51.053165 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6b5cb03d0e4f89c74177e9a690093f3b032a244561b16401f52900e43de085f"} err="failed to get container status \"b6b5cb03d0e4f89c74177e9a690093f3b032a244561b16401f52900e43de085f\": rpc error: code = NotFound desc = could not find container \"b6b5cb03d0e4f89c74177e9a690093f3b032a244561b16401f52900e43de085f\": container with ID starting with b6b5cb03d0e4f89c74177e9a690093f3b032a244561b16401f52900e43de085f not found: ID does not exist" Oct 30 09:32:51.097191 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:51.097168 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-d87c85/ipp-route-d87c85-kserve-7cc5cb6f7c-qk4q7"] Oct 30 09:32:51.110141 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:51.110116 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-d87c85/ipp-route-d87c85-kserve-7cc5cb6f7c-qk4q7"] Oct 30 09:32:51.874774 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:51.874728 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bfc138b-8938-473c-adfe-052d9a722233" path="/var/lib/kubelet/pods/9bfc138b-8938-473c-adfe-052d9a722233/volumes" Oct 30 09:32:52.486506 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:52.486466 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-304459-69f5bf967f-l4kvg"] Oct 30 09:32:52.487324 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:52.487301 4562 cpu_manager.go:414] "containerMap: removing container" podUID="9bfc138b-8938-473c-adfe-052d9a722233" containerName="main" Oct 30 09:32:52.487462 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:52.487325 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bfc138b-8938-473c-adfe-052d9a722233" containerName="main" Oct 30 09:32:52.487551 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:52.487536 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bfc138b-8938-473c-adfe-052d9a722233" containerName="main" Oct 30 09:32:52.490942 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:52.490922 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-304459-69f5bf967f-l4kvg" Oct 30 09:32:52.510130 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:52.510102 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-304459-69f5bf967f-l4kvg"] Oct 30 09:32:52.519807 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:52.519785 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-304459-dockercfg-srp55\"" Oct 30 09:32:52.519960 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:52.519879 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-304459\"" Oct 30 09:32:52.611404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:52.581814 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a75406b8-14aa-4fc1-975e-620b911cb9b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-304459-69f5bf967f-l4kvg\" (UID: \"a75406b8-14aa-4fc1-975e-620b911cb9b7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-304459-69f5bf967f-l4kvg" Oct 30 09:32:52.638553 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:52.638514 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9942\" (UniqueName: \"kubernetes.io/projected/a75406b8-14aa-4fc1-975e-620b911cb9b7-kube-api-access-v9942\") pod \"payload-pre-processing-e2e-shared-a-w3-304459-69f5bf967f-l4kvg\" (UID: \"a75406b8-14aa-4fc1-975e-620b911cb9b7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-304459-69f5bf967f-l4kvg" Oct 30 09:32:52.766502 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:52.762578 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9942\" (UniqueName: \"kubernetes.io/projected/a75406b8-14aa-4fc1-975e-620b911cb9b7-kube-api-access-v9942\") pod \"payload-pre-processing-e2e-shared-a-w3-304459-69f5bf967f-l4kvg\" (UID: \"a75406b8-14aa-4fc1-975e-620b911cb9b7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-304459-69f5bf967f-l4kvg" Oct 30 09:32:52.766502 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:52.764063 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a75406b8-14aa-4fc1-975e-620b911cb9b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-304459-69f5bf967f-l4kvg\" (UID: \"a75406b8-14aa-4fc1-975e-620b911cb9b7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-304459-69f5bf967f-l4kvg" Oct 30 09:32:52.766502 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:52.765008 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a75406b8-14aa-4fc1-975e-620b911cb9b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-304459-69f5bf967f-l4kvg\" (UID: \"a75406b8-14aa-4fc1-975e-620b911cb9b7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-304459-69f5bf967f-l4kvg" Oct 30 09:32:52.791546 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:52.791504 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9942\" (UniqueName: \"kubernetes.io/projected/a75406b8-14aa-4fc1-975e-620b911cb9b7-kube-api-access-v9942\") pod \"payload-pre-processing-e2e-shared-a-w3-304459-69f5bf967f-l4kvg\" (UID: \"a75406b8-14aa-4fc1-975e-620b911cb9b7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-304459-69f5bf967f-l4kvg" Oct 30 09:32:52.805114 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:52.805081 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-304459-69f5bf967f-l4kvg" Oct 30 09:32:53.109649 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:32:53.109621 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda75406b8_14aa_4fc1_975e_620b911cb9b7.slice/crio-75c2ed182902b5a92aa05f04afc974f4d6757be9a41ecd689543dd85f7e438a2 WatchSource:0}: Error finding container 75c2ed182902b5a92aa05f04afc974f4d6757be9a41ecd689543dd85f7e438a2: Status 404 returned error can't find the container with id 75c2ed182902b5a92aa05f04afc974f4d6757be9a41ecd689543dd85f7e438a2 Oct 30 09:32:53.109968 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:53.109940 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-304459-69f5bf967f-l4kvg"] Oct 30 09:32:54.073682 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:54.073624 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-304459-69f5bf967f-l4kvg" event={"ID":"a75406b8-14aa-4fc1-975e-620b911cb9b7","Type":"ContainerStarted","Data":"f85f9e1f90c5f66cac6e0e729237d2a15487fcb905a1919cef7b40bc14a45c09"} Oct 30 09:32:54.074336 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:54.074312 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-304459-69f5bf967f-l4kvg" event={"ID":"a75406b8-14aa-4fc1-975e-620b911cb9b7","Type":"ContainerStarted","Data":"75c2ed182902b5a92aa05f04afc974f4d6757be9a41ecd689543dd85f7e438a2"} Oct 30 09:32:54.074675 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:54.074654 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-304459-69f5bf967f-l4kvg" Oct 30 09:32:54.114315 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:54.114266 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-304459-69f5bf967f-l4kvg" podStartSLOduration=2.114254067 podStartE2EDuration="2.114254067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:32:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:32:54.113445924 +0000 UTC m=+2334.375006934" watchObservedRunningTime="2026-10-30 09:32:54.114254067 +0000 UTC m=+2334.375815054" Oct 30 09:32:58.261515 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:58.261481 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67"] Oct 30 09:32:58.265787 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:58.265758 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67" Oct 30 09:32:58.292346 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:58.292316 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67"] Oct 30 09:32:58.302006 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:58.299975 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-aa2fdf-openshift-default-dockercfg-krds6\"" Oct 30 09:32:58.350039 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:58.350002 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/821c88fc-b72b-4104-97f3-e3528428498e-istiod-ca-cert\") pod \"e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67\" (UID: \"821c88fc-b72b-4104-97f3-e3528428498e\") " pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67" Oct 30 09:32:58.350182 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:58.350079 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/821c88fc-b72b-4104-97f3-e3528428498e-istio-envoy\") pod \"e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67\" (UID: \"821c88fc-b72b-4104-97f3-e3528428498e\") " pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67" Oct 30 09:32:58.350228 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:58.350171 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rr2x\" (UniqueName: \"kubernetes.io/projected/821c88fc-b72b-4104-97f3-e3528428498e-kube-api-access-9rr2x\") pod \"e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67\" (UID: \"821c88fc-b72b-4104-97f3-e3528428498e\") " pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67" Oct 30 09:32:58.350272 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:58.350228 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/821c88fc-b72b-4104-97f3-e3528428498e-istio-token\") pod \"e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67\" (UID: \"821c88fc-b72b-4104-97f3-e3528428498e\") " pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67" Oct 30 09:32:58.350319 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:58.350295 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/821c88fc-b72b-4104-97f3-e3528428498e-workload-socket\") pod \"e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67\" (UID: \"821c88fc-b72b-4104-97f3-e3528428498e\") " pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67" Oct 30 09:32:58.350359 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:58.350330 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/821c88fc-b72b-4104-97f3-e3528428498e-istio-podinfo\") pod \"e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67\" (UID: \"821c88fc-b72b-4104-97f3-e3528428498e\") " pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67" Oct 30 09:32:58.350442 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:58.350405 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/821c88fc-b72b-4104-97f3-e3528428498e-credential-socket\") pod \"e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67\" (UID: \"821c88fc-b72b-4104-97f3-e3528428498e\") " pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67" Oct 30 09:32:58.350532 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:58.350510 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/821c88fc-b72b-4104-97f3-e3528428498e-istio-data\") pod \"e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67\" (UID: \"821c88fc-b72b-4104-97f3-e3528428498e\") " pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67" Oct 30 09:32:58.350672 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:58.350656 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/821c88fc-b72b-4104-97f3-e3528428498e-workload-certs\") pod \"e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67\" (UID: \"821c88fc-b72b-4104-97f3-e3528428498e\") " pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67" Oct 30 09:32:58.451759 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:58.451724 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/821c88fc-b72b-4104-97f3-e3528428498e-istio-envoy\") pod \"e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67\" (UID: \"821c88fc-b72b-4104-97f3-e3528428498e\") " pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67" Oct 30 09:32:58.451941 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:58.451780 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rr2x\" (UniqueName: \"kubernetes.io/projected/821c88fc-b72b-4104-97f3-e3528428498e-kube-api-access-9rr2x\") pod \"e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67\" (UID: \"821c88fc-b72b-4104-97f3-e3528428498e\") " pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67" Oct 30 09:32:58.451941 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:58.451829 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/821c88fc-b72b-4104-97f3-e3528428498e-istio-token\") pod \"e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67\" (UID: \"821c88fc-b72b-4104-97f3-e3528428498e\") " pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67" Oct 30 09:32:58.451941 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:58.451850 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/821c88fc-b72b-4104-97f3-e3528428498e-workload-socket\") pod \"e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67\" (UID: \"821c88fc-b72b-4104-97f3-e3528428498e\") " pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67" Oct 30 09:32:58.451941 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:58.451873 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/821c88fc-b72b-4104-97f3-e3528428498e-istio-podinfo\") pod \"e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67\" (UID: \"821c88fc-b72b-4104-97f3-e3528428498e\") " pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67" Oct 30 09:32:58.451941 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:58.451921 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/821c88fc-b72b-4104-97f3-e3528428498e-credential-socket\") pod \"e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67\" (UID: \"821c88fc-b72b-4104-97f3-e3528428498e\") " pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67" Oct 30 09:32:58.451941 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:58.451945 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/821c88fc-b72b-4104-97f3-e3528428498e-istio-data\") pod \"e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67\" (UID: \"821c88fc-b72b-4104-97f3-e3528428498e\") " pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67" Oct 30 09:32:58.452250 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:58.452081 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/821c88fc-b72b-4104-97f3-e3528428498e-workload-certs\") pod \"e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67\" (UID: \"821c88fc-b72b-4104-97f3-e3528428498e\") " pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67" Oct 30 09:32:58.452250 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:58.452142 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/821c88fc-b72b-4104-97f3-e3528428498e-istiod-ca-cert\") pod \"e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67\" (UID: \"821c88fc-b72b-4104-97f3-e3528428498e\") " pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67" Oct 30 09:32:58.452366 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:58.452257 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/821c88fc-b72b-4104-97f3-e3528428498e-workload-socket\") pod \"e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67\" (UID: \"821c88fc-b72b-4104-97f3-e3528428498e\") " pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67" Oct 30 09:32:58.452366 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:58.452325 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/821c88fc-b72b-4104-97f3-e3528428498e-credential-socket\") pod \"e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67\" (UID: \"821c88fc-b72b-4104-97f3-e3528428498e\") " pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67" Oct 30 09:32:58.452519 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:58.452494 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/821c88fc-b72b-4104-97f3-e3528428498e-istio-data\") pod \"e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67\" (UID: \"821c88fc-b72b-4104-97f3-e3528428498e\") " pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67" Oct 30 09:32:58.452579 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:58.452502 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/821c88fc-b72b-4104-97f3-e3528428498e-workload-certs\") pod \"e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67\" (UID: \"821c88fc-b72b-4104-97f3-e3528428498e\") " pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67" Oct 30 09:32:58.452792 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:58.452774 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/821c88fc-b72b-4104-97f3-e3528428498e-istiod-ca-cert\") pod \"e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67\" (UID: \"821c88fc-b72b-4104-97f3-e3528428498e\") " pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67" Oct 30 09:32:58.454834 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:58.454795 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/821c88fc-b72b-4104-97f3-e3528428498e-istio-envoy\") pod \"e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67\" (UID: \"821c88fc-b72b-4104-97f3-e3528428498e\") " pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67" Oct 30 09:32:58.455184 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:58.455008 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/821c88fc-b72b-4104-97f3-e3528428498e-istio-podinfo\") pod \"e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67\" (UID: \"821c88fc-b72b-4104-97f3-e3528428498e\") " pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67" Oct 30 09:32:58.498404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:58.491266 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rr2x\" (UniqueName: \"kubernetes.io/projected/821c88fc-b72b-4104-97f3-e3528428498e-kube-api-access-9rr2x\") pod \"e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67\" (UID: \"821c88fc-b72b-4104-97f3-e3528428498e\") " pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67" Oct 30 09:32:58.498404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:58.491827 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/821c88fc-b72b-4104-97f3-e3528428498e-istio-token\") pod \"e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67\" (UID: \"821c88fc-b72b-4104-97f3-e3528428498e\") " pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67" Oct 30 09:32:58.592990 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:58.592836 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67" Oct 30 09:32:58.689055 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:32:58.689009 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:32:58.990446 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:32:58.990415 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod821c88fc_b72b_4104_97f3_e3528428498e.slice/crio-aeca570386888e4772d4c5095fa92650e30fd03967abe132e4cf436ea9e67e50 WatchSource:0}: Error finding container aeca570386888e4772d4c5095fa92650e30fd03967abe132e4cf436ea9e67e50: Status 404 returned error can't find the container with id aeca570386888e4772d4c5095fa92650e30fd03967abe132e4cf436ea9e67e50 Oct 30 09:32:58.991540 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:58.991516 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67"] Oct 30 09:32:58.993028 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:58.992993 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:32:58.993122 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:58.993058 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:32:58.993122 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:58.993088 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:32:59.136613 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:59.136566 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67" event={"ID":"821c88fc-b72b-4104-97f3-e3528428498e","Type":"ContainerStarted","Data":"adb2c7f68c8ef0c8e496656d9db26ad4174a5e4dc87e308101da44dc9998967b"} Oct 30 09:32:59.136779 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:59.136636 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67" event={"ID":"821c88fc-b72b-4104-97f3-e3528428498e","Type":"ContainerStarted","Data":"aeca570386888e4772d4c5095fa92650e30fd03967abe132e4cf436ea9e67e50"} Oct 30 09:32:59.183298 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:59.183247 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67" podStartSLOduration=1.183234275 podStartE2EDuration="1.183234275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:32:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:32:59.181148249 +0000 UTC m=+2339.442709234" watchObservedRunningTime="2026-10-30 09:32:59.183234275 +0000 UTC m=+2339.444795286" Oct 30 09:32:59.596569 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:32:59.596536 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67" Oct 30 09:33:00.604542 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:00.604507 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67" Oct 30 09:33:01.150822 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:01.150782 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67" Oct 30 09:33:01.151655 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:01.151634 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67" Oct 30 09:33:06.114095 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:06.114049 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-304459-69f5bf967f-l4kvg" Oct 30 09:33:09.604464 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:09.604432 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0dc148-7b74d7b55-zzv9d"] Oct 30 09:33:09.610431 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:09.610407 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0dc148-7b74d7b55-zzv9d" Oct 30 09:33:09.624858 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:09.624831 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0dc148-7b74d7b55-zzv9d"] Oct 30 09:33:09.634113 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:09.634082 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-0dc148\"" Oct 30 09:33:09.634244 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:09.634132 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-0dc148-dockercfg-8z2vc\"" Oct 30 09:33:09.681807 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:09.681763 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rskd2\" (UniqueName: \"kubernetes.io/projected/fa734120-4c82-426f-947c-1a36330750ba-kube-api-access-rskd2\") pod \"payload-pre-processing-e2e-shared-b-w2-0dc148-7b74d7b55-zzv9d\" (UID: \"fa734120-4c82-426f-947c-1a36330750ba\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0dc148-7b74d7b55-zzv9d" Oct 30 09:33:09.681807 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:09.681839 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa734120-4c82-426f-947c-1a36330750ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-0dc148-7b74d7b55-zzv9d\" (UID: \"fa734120-4c82-426f-947c-1a36330750ba\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0dc148-7b74d7b55-zzv9d" Oct 30 09:33:09.800467 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:09.796577 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rskd2\" (UniqueName: \"kubernetes.io/projected/fa734120-4c82-426f-947c-1a36330750ba-kube-api-access-rskd2\") pod \"payload-pre-processing-e2e-shared-b-w2-0dc148-7b74d7b55-zzv9d\" (UID: \"fa734120-4c82-426f-947c-1a36330750ba\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0dc148-7b74d7b55-zzv9d" Oct 30 09:33:09.800467 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:09.796655 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa734120-4c82-426f-947c-1a36330750ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-0dc148-7b74d7b55-zzv9d\" (UID: \"fa734120-4c82-426f-947c-1a36330750ba\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0dc148-7b74d7b55-zzv9d" Oct 30 09:33:09.800467 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:09.797716 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa734120-4c82-426f-947c-1a36330750ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-0dc148-7b74d7b55-zzv9d\" (UID: \"fa734120-4c82-426f-947c-1a36330750ba\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0dc148-7b74d7b55-zzv9d" Oct 30 09:33:09.918487 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:09.903769 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rskd2\" (UniqueName: \"kubernetes.io/projected/fa734120-4c82-426f-947c-1a36330750ba-kube-api-access-rskd2\") pod \"payload-pre-processing-e2e-shared-b-w2-0dc148-7b74d7b55-zzv9d\" (UID: \"fa734120-4c82-426f-947c-1a36330750ba\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0dc148-7b74d7b55-zzv9d" Oct 30 09:33:09.924671 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:09.924432 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0dc148-7b74d7b55-zzv9d" Oct 30 09:33:10.236969 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:33:10.236936 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa734120_4c82_426f_947c_1a36330750ba.slice/crio-b73cd36deecb1149a8656f1068d944d662ae1082c3168a75ccdd5a7538413c65 WatchSource:0}: Error finding container b73cd36deecb1149a8656f1068d944d662ae1082c3168a75ccdd5a7538413c65: Status 404 returned error can't find the container with id b73cd36deecb1149a8656f1068d944d662ae1082c3168a75ccdd5a7538413c65 Oct 30 09:33:10.238325 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:10.238300 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0dc148-7b74d7b55-zzv9d"] Oct 30 09:33:11.220916 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:11.220880 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0dc148-7b74d7b55-zzv9d" event={"ID":"fa734120-4c82-426f-947c-1a36330750ba","Type":"ContainerStarted","Data":"88e862975b14935fa9dc419ea24f2e4f04494605d9049bbc0ddb76838445f0c5"} Oct 30 09:33:11.221333 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:11.220964 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0dc148-7b74d7b55-zzv9d" event={"ID":"fa734120-4c82-426f-947c-1a36330750ba","Type":"ContainerStarted","Data":"b73cd36deecb1149a8656f1068d944d662ae1082c3168a75ccdd5a7538413c65"} Oct 30 09:33:11.221333 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:11.221023 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0dc148-7b74d7b55-zzv9d" Oct 30 09:33:11.261303 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:11.261251 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0dc148-7b74d7b55-zzv9d" podStartSLOduration=2.261236985 podStartE2EDuration="2.261236985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:33:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:33:11.260166838 +0000 UTC m=+2351.521727826" watchObservedRunningTime="2026-10-30 09:33:11.261236985 +0000 UTC m=+2351.522797972" Oct 30 09:33:21.984162 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:21.984125 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-aa2fdf-799786b6bf-vtjc9"] Oct 30 09:33:21.989905 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:21.989887 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-aa2fdf-799786b6bf-vtjc9" Oct 30 09:33:22.009808 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:22.009767 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-aa2fdf-799786b6bf-vtjc9"] Oct 30 09:33:22.020369 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:22.020322 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-aa2fdf-serving-cert\"" Oct 30 09:33:22.030439 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:22.027100 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37fad2d6-d1ba-4440-bf0f-0f46c12987fb-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-aa2fdf-799786b6bf-vtjc9\" (UID: \"37fad2d6-d1ba-4440-bf0f-0f46c12987fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-aa2fdf-799786b6bf-vtjc9" Oct 30 09:33:22.030439 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:22.027432 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37fad2d6-d1ba-4440-bf0f-0f46c12987fb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-aa2fdf-799786b6bf-vtjc9\" (UID: \"37fad2d6-d1ba-4440-bf0f-0f46c12987fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-aa2fdf-799786b6bf-vtjc9" Oct 30 09:33:22.030439 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:22.027532 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37fad2d6-d1ba-4440-bf0f-0f46c12987fb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-aa2fdf-799786b6bf-vtjc9\" (UID: \"37fad2d6-d1ba-4440-bf0f-0f46c12987fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-aa2fdf-799786b6bf-vtjc9" Oct 30 09:33:22.030439 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:22.027591 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkhhk\" (UniqueName: \"kubernetes.io/projected/37fad2d6-d1ba-4440-bf0f-0f46c12987fb-kube-api-access-xkhhk\") pod \"maas-api-e2e-shared-b-w1-aa2fdf-799786b6bf-vtjc9\" (UID: \"37fad2d6-d1ba-4440-bf0f-0f46c12987fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-aa2fdf-799786b6bf-vtjc9" Oct 30 09:33:22.129347 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:22.129310 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37fad2d6-d1ba-4440-bf0f-0f46c12987fb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-aa2fdf-799786b6bf-vtjc9\" (UID: \"37fad2d6-d1ba-4440-bf0f-0f46c12987fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-aa2fdf-799786b6bf-vtjc9" Oct 30 09:33:22.129571 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:22.129377 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37fad2d6-d1ba-4440-bf0f-0f46c12987fb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-aa2fdf-799786b6bf-vtjc9\" (UID: \"37fad2d6-d1ba-4440-bf0f-0f46c12987fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-aa2fdf-799786b6bf-vtjc9" Oct 30 09:33:22.129571 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:22.129430 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkhhk\" (UniqueName: \"kubernetes.io/projected/37fad2d6-d1ba-4440-bf0f-0f46c12987fb-kube-api-access-xkhhk\") pod \"maas-api-e2e-shared-b-w1-aa2fdf-799786b6bf-vtjc9\" (UID: \"37fad2d6-d1ba-4440-bf0f-0f46c12987fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-aa2fdf-799786b6bf-vtjc9" Oct 30 09:33:22.129721 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:22.129575 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37fad2d6-d1ba-4440-bf0f-0f46c12987fb-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-aa2fdf-799786b6bf-vtjc9\" (UID: \"37fad2d6-d1ba-4440-bf0f-0f46c12987fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-aa2fdf-799786b6bf-vtjc9" Oct 30 09:33:22.130170 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:22.130146 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37fad2d6-d1ba-4440-bf0f-0f46c12987fb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-aa2fdf-799786b6bf-vtjc9\" (UID: \"37fad2d6-d1ba-4440-bf0f-0f46c12987fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-aa2fdf-799786b6bf-vtjc9" Oct 30 09:33:22.131927 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:22.131902 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37fad2d6-d1ba-4440-bf0f-0f46c12987fb-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-aa2fdf-799786b6bf-vtjc9\" (UID: \"37fad2d6-d1ba-4440-bf0f-0f46c12987fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-aa2fdf-799786b6bf-vtjc9" Oct 30 09:33:22.131927 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:22.131922 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37fad2d6-d1ba-4440-bf0f-0f46c12987fb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-aa2fdf-799786b6bf-vtjc9\" (UID: \"37fad2d6-d1ba-4440-bf0f-0f46c12987fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-aa2fdf-799786b6bf-vtjc9" Oct 30 09:33:22.152632 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:22.152602 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkhhk\" (UniqueName: \"kubernetes.io/projected/37fad2d6-d1ba-4440-bf0f-0f46c12987fb-kube-api-access-xkhhk\") pod \"maas-api-e2e-shared-b-w1-aa2fdf-799786b6bf-vtjc9\" (UID: \"37fad2d6-d1ba-4440-bf0f-0f46c12987fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-aa2fdf-799786b6bf-vtjc9" Oct 30 09:33:22.291921 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:22.291827 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0dc148-7b74d7b55-zzv9d" Oct 30 09:33:22.304220 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:22.304177 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-aa2fdf-799786b6bf-vtjc9" Oct 30 09:33:22.632198 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:33:22.632169 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37fad2d6_d1ba_4440_bf0f_0f46c12987fb.slice/crio-4597e9a969cc8229029c87e5dba6999d2ac77208c3112462139c8187519be431 WatchSource:0}: Error finding container 4597e9a969cc8229029c87e5dba6999d2ac77208c3112462139c8187519be431: Status 404 returned error can't find the container with id 4597e9a969cc8229029c87e5dba6999d2ac77208c3112462139c8187519be431 Oct 30 09:33:22.633273 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:22.633250 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-aa2fdf-799786b6bf-vtjc9"] Oct 30 09:33:23.363064 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:23.362979 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-aa2fdf-799786b6bf-vtjc9" event={"ID":"37fad2d6-d1ba-4440-bf0f-0f46c12987fb","Type":"ContainerStarted","Data":"19600e41907c2749e4a4e3d197702cc014c643bd306b2c7de963218a81156f9f"} Oct 30 09:33:23.363064 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:23.363032 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-aa2fdf-799786b6bf-vtjc9" event={"ID":"37fad2d6-d1ba-4440-bf0f-0f46c12987fb","Type":"ContainerStarted","Data":"4597e9a969cc8229029c87e5dba6999d2ac77208c3112462139c8187519be431"} Oct 30 09:33:23.363525 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:23.363157 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-aa2fdf-799786b6bf-vtjc9" Oct 30 09:33:23.405844 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:23.405787 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-aa2fdf-799786b6bf-vtjc9" podStartSLOduration=2.069880825 podStartE2EDuration="2.405773039s" totalImagesPullingTime="335.892214ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:33:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:33:23.403889328 +0000 UTC m=+2363.665450339" watchObservedRunningTime="2026-10-30 09:33:23.405773039 +0000 UTC m=+2363.667334027" Oct 30 09:33:28.597126 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:33:28.597087 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:33:29.374006 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:29.373979 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-aa2fdf-799786b6bf-vtjc9" Oct 30 09:33:43.337861 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:43.337781 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7a213d-65fd87589f-rr7jz"] Oct 30 09:33:43.344807 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:43.344776 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7a213d-65fd87589f-rr7jz" Oct 30 09:33:43.361525 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:43.361493 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7a213d-65fd87589f-rr7jz"] Oct 30 09:33:43.368981 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:43.368956 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-7a213d-dockercfg-j8xcr\"" Oct 30 09:33:43.369289 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:43.369267 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-7a213d\"" Oct 30 09:33:43.443932 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:43.443888 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7a213d-689cb5bf76-9b72m"] Oct 30 09:33:43.452886 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:43.452860 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7a213d-689cb5bf76-9b72m" Oct 30 09:33:43.488461 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:43.479006 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7a213d-689cb5bf76-9b72m"] Oct 30 09:33:43.568177 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:43.560218 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c33c6c07-06d5-4069-9eb8-2c176a66ab2a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-7a213d-65fd87589f-rr7jz\" (UID: \"c33c6c07-06d5-4069-9eb8-2c176a66ab2a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7a213d-65fd87589f-rr7jz" Oct 30 09:33:43.568177 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:43.560325 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gfrr\" (UniqueName: \"kubernetes.io/projected/c33c6c07-06d5-4069-9eb8-2c176a66ab2a-kube-api-access-8gfrr\") pod \"payload-pre-processing-e2e-shared-b-w3-7a213d-65fd87589f-rr7jz\" (UID: \"c33c6c07-06d5-4069-9eb8-2c176a66ab2a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7a213d-65fd87589f-rr7jz" Oct 30 09:33:43.568177 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:43.560354 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvzvr\" (UniqueName: \"kubernetes.io/projected/8db504e0-4d30-41b1-91cb-12daa3e2ac43-kube-api-access-gvzvr\") pod \"payload-processing-e2e-shared-b-w3-7a213d-689cb5bf76-9b72m\" (UID: \"8db504e0-4d30-41b1-91cb-12daa3e2ac43\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7a213d-689cb5bf76-9b72m" Oct 30 09:33:43.568177 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:43.560439 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8db504e0-4d30-41b1-91cb-12daa3e2ac43-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-7a213d-689cb5bf76-9b72m\" (UID: \"8db504e0-4d30-41b1-91cb-12daa3e2ac43\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7a213d-689cb5bf76-9b72m" Oct 30 09:33:43.661961 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:43.661873 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gfrr\" (UniqueName: \"kubernetes.io/projected/c33c6c07-06d5-4069-9eb8-2c176a66ab2a-kube-api-access-8gfrr\") pod \"payload-pre-processing-e2e-shared-b-w3-7a213d-65fd87589f-rr7jz\" (UID: \"c33c6c07-06d5-4069-9eb8-2c176a66ab2a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7a213d-65fd87589f-rr7jz" Oct 30 09:33:43.662257 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:43.662227 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvzvr\" (UniqueName: \"kubernetes.io/projected/8db504e0-4d30-41b1-91cb-12daa3e2ac43-kube-api-access-gvzvr\") pod \"payload-processing-e2e-shared-b-w3-7a213d-689cb5bf76-9b72m\" (UID: \"8db504e0-4d30-41b1-91cb-12daa3e2ac43\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7a213d-689cb5bf76-9b72m" Oct 30 09:33:43.662511 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:43.662488 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8db504e0-4d30-41b1-91cb-12daa3e2ac43-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-7a213d-689cb5bf76-9b72m\" (UID: \"8db504e0-4d30-41b1-91cb-12daa3e2ac43\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7a213d-689cb5bf76-9b72m" Oct 30 09:33:43.663020 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:43.662996 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c33c6c07-06d5-4069-9eb8-2c176a66ab2a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-7a213d-65fd87589f-rr7jz\" (UID: \"c33c6c07-06d5-4069-9eb8-2c176a66ab2a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7a213d-65fd87589f-rr7jz" Oct 30 09:33:43.663706 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:43.663664 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c33c6c07-06d5-4069-9eb8-2c176a66ab2a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-7a213d-65fd87589f-rr7jz\" (UID: \"c33c6c07-06d5-4069-9eb8-2c176a66ab2a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7a213d-65fd87589f-rr7jz" Oct 30 09:33:43.663795 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:43.663667 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8db504e0-4d30-41b1-91cb-12daa3e2ac43-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-7a213d-689cb5bf76-9b72m\" (UID: \"8db504e0-4d30-41b1-91cb-12daa3e2ac43\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7a213d-689cb5bf76-9b72m" Oct 30 09:33:43.687428 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:43.682323 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gfrr\" (UniqueName: \"kubernetes.io/projected/c33c6c07-06d5-4069-9eb8-2c176a66ab2a-kube-api-access-8gfrr\") pod \"payload-pre-processing-e2e-shared-b-w3-7a213d-65fd87589f-rr7jz\" (UID: \"c33c6c07-06d5-4069-9eb8-2c176a66ab2a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7a213d-65fd87589f-rr7jz" Oct 30 09:33:43.687428 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:43.683764 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvzvr\" (UniqueName: \"kubernetes.io/projected/8db504e0-4d30-41b1-91cb-12daa3e2ac43-kube-api-access-gvzvr\") pod \"payload-processing-e2e-shared-b-w3-7a213d-689cb5bf76-9b72m\" (UID: \"8db504e0-4d30-41b1-91cb-12daa3e2ac43\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7a213d-689cb5bf76-9b72m" Oct 30 09:33:43.797803 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:43.797762 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7a213d-689cb5bf76-9b72m" Oct 30 09:33:43.967063 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:43.967021 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7a213d-65fd87589f-rr7jz" Oct 30 09:33:44.118731 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:44.118695 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7a213d-689cb5bf76-9b72m"] Oct 30 09:33:44.125991 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:33:44.125950 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8db504e0_4d30_41b1_91cb_12daa3e2ac43.slice/crio-0d20b53da21cd8c69e65cd01b5fcf87a7f8ee7684419845f096ccdfcae856aff WatchSource:0}: Error finding container 0d20b53da21cd8c69e65cd01b5fcf87a7f8ee7684419845f096ccdfcae856aff: Status 404 returned error can't find the container with id 0d20b53da21cd8c69e65cd01b5fcf87a7f8ee7684419845f096ccdfcae856aff Oct 30 09:33:44.272173 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:33:44.272137 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc33c6c07_06d5_4069_9eb8_2c176a66ab2a.slice/crio-e8992c5d7548b1d7a42664749d7f2a863206fa2fc4906e27e2a0e419a428d882 WatchSource:0}: Error finding container e8992c5d7548b1d7a42664749d7f2a863206fa2fc4906e27e2a0e419a428d882: Status 404 returned error can't find the container with id e8992c5d7548b1d7a42664749d7f2a863206fa2fc4906e27e2a0e419a428d882 Oct 30 09:33:44.275096 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:44.275068 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7a213d-65fd87589f-rr7jz"] Oct 30 09:33:44.657115 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:44.657020 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7a213d-65fd87589f-rr7jz" event={"ID":"c33c6c07-06d5-4069-9eb8-2c176a66ab2a","Type":"ContainerStarted","Data":"a62b796d94f3c63ac074a903113ce6f9ab0a573ae27eea5c1888390cdc2dabf4"} Oct 30 09:33:44.657115 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:44.657081 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7a213d-65fd87589f-rr7jz" event={"ID":"c33c6c07-06d5-4069-9eb8-2c176a66ab2a","Type":"ContainerStarted","Data":"e8992c5d7548b1d7a42664749d7f2a863206fa2fc4906e27e2a0e419a428d882"} Oct 30 09:33:44.657115 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:44.657108 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7a213d-65fd87589f-rr7jz" Oct 30 09:33:44.658866 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:44.658840 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7a213d-689cb5bf76-9b72m" event={"ID":"8db504e0-4d30-41b1-91cb-12daa3e2ac43","Type":"ContainerStarted","Data":"edcdbff1f4f7329212ff0cd68aed0bac37e8b0a25ea633b113a37429a4fa77b3"} Oct 30 09:33:44.658966 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:44.658887 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7a213d-689cb5bf76-9b72m" event={"ID":"8db504e0-4d30-41b1-91cb-12daa3e2ac43","Type":"ContainerStarted","Data":"0d20b53da21cd8c69e65cd01b5fcf87a7f8ee7684419845f096ccdfcae856aff"} Oct 30 09:33:44.658966 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:44.658942 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7a213d-689cb5bf76-9b72m" Oct 30 09:33:44.699057 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:44.699012 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7a213d-65fd87589f-rr7jz" podStartSLOduration=1.698998668 podStartE2EDuration="1.698998668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:33:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:33:44.696925224 +0000 UTC m=+2384.958486212" watchObservedRunningTime="2026-10-30 09:33:44.698998668 +0000 UTC m=+2384.960559655" Oct 30 09:33:44.740835 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:44.740783 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7a213d-689cb5bf76-9b72m" podStartSLOduration=1.740768832 podStartE2EDuration="1.740768832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:33:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:33:44.736217764 +0000 UTC m=+2384.997778753" watchObservedRunningTime="2026-10-30 09:33:44.740768832 +0000 UTC m=+2385.002329819" Oct 30 09:33:46.219752 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:46.219134 4562 patch_prober.go:29] interesting pod/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": read tcp 10.133.0.2:49340->10.133.0.180:15021: read: connection reset by peer" start-of-body= Oct 30 09:33:46.219752 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:46.219186 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67" podUID="821c88fc-b72b-4104-97f3-e3528428498e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": read tcp 10.133.0.2:49340->10.133.0.180:15021: read: connection reset by peer" Oct 30 09:33:46.673049 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:46.672961 4562 generic.go:356] "Generic (PLEG): container finished" podID="821c88fc-b72b-4104-97f3-e3528428498e" containerID="adb2c7f68c8ef0c8e496656d9db26ad4174a5e4dc87e308101da44dc9998967b" exitCode=0 Oct 30 09:33:46.673049 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:46.673035 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67" event={"ID":"821c88fc-b72b-4104-97f3-e3528428498e","Type":"ContainerDied","Data":"adb2c7f68c8ef0c8e496656d9db26ad4174a5e4dc87e308101da44dc9998967b"} Oct 30 09:33:46.673519 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:46.673503 4562 scope.go:122] "RemoveContainer" containerID="adb2c7f68c8ef0c8e496656d9db26ad4174a5e4dc87e308101da44dc9998967b" Oct 30 09:33:46.675120 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:46.675092 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:33:46.675205 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:46.675158 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:33:46.675205 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:46.675194 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:33:46.822668 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:46.822633 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh"] Oct 30 09:33:46.828177 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:46.828149 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh" Oct 30 09:33:46.875007 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:46.874976 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh"] Oct 30 09:33:46.925238 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:46.925141 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7f74510-23f8-4f0b-af8e-6c9861215203-istio-data\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh\" (UID: \"d7f74510-23f8-4f0b-af8e-6c9861215203\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh" Oct 30 09:33:46.925238 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:46.925204 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f74510-23f8-4f0b-af8e-6c9861215203-workload-socket\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh\" (UID: \"d7f74510-23f8-4f0b-af8e-6c9861215203\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh" Oct 30 09:33:46.925484 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:46.925341 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7f74510-23f8-4f0b-af8e-6c9861215203-istio-token\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh\" (UID: \"d7f74510-23f8-4f0b-af8e-6c9861215203\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh" Oct 30 09:33:46.925484 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:46.925462 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wl4g\" (UniqueName: \"kubernetes.io/projected/d7f74510-23f8-4f0b-af8e-6c9861215203-kube-api-access-9wl4g\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh\" (UID: \"d7f74510-23f8-4f0b-af8e-6c9861215203\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh" Oct 30 09:33:46.925578 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:46.925530 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7f74510-23f8-4f0b-af8e-6c9861215203-istio-envoy\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh\" (UID: \"d7f74510-23f8-4f0b-af8e-6c9861215203\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh" Oct 30 09:33:46.925630 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:46.925611 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7f74510-23f8-4f0b-af8e-6c9861215203-istio-podinfo\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh\" (UID: \"d7f74510-23f8-4f0b-af8e-6c9861215203\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh" Oct 30 09:33:46.925675 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:46.925647 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7f74510-23f8-4f0b-af8e-6c9861215203-istiod-ca-cert\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh\" (UID: \"d7f74510-23f8-4f0b-af8e-6c9861215203\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh" Oct 30 09:33:46.925812 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:46.925793 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f74510-23f8-4f0b-af8e-6c9861215203-credential-socket\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh\" (UID: \"d7f74510-23f8-4f0b-af8e-6c9861215203\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh" Oct 30 09:33:46.925877 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:46.925834 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7f74510-23f8-4f0b-af8e-6c9861215203-workload-certs\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh\" (UID: \"d7f74510-23f8-4f0b-af8e-6c9861215203\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh" Oct 30 09:33:46.932118 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:46.932077 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-304459-openshift-default-dockercfg-8w4bn\"" Oct 30 09:33:47.026854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:47.026818 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7f74510-23f8-4f0b-af8e-6c9861215203-istio-token\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh\" (UID: \"d7f74510-23f8-4f0b-af8e-6c9861215203\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh" Oct 30 09:33:47.027046 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:47.026881 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wl4g\" (UniqueName: \"kubernetes.io/projected/d7f74510-23f8-4f0b-af8e-6c9861215203-kube-api-access-9wl4g\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh\" (UID: \"d7f74510-23f8-4f0b-af8e-6c9861215203\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh" Oct 30 09:33:47.027046 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:47.026915 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7f74510-23f8-4f0b-af8e-6c9861215203-istio-envoy\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh\" (UID: \"d7f74510-23f8-4f0b-af8e-6c9861215203\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh" Oct 30 09:33:47.027046 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:47.026982 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7f74510-23f8-4f0b-af8e-6c9861215203-istio-podinfo\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh\" (UID: \"d7f74510-23f8-4f0b-af8e-6c9861215203\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh" Oct 30 09:33:47.027046 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:47.027018 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7f74510-23f8-4f0b-af8e-6c9861215203-istiod-ca-cert\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh\" (UID: \"d7f74510-23f8-4f0b-af8e-6c9861215203\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh" Oct 30 09:33:47.027211 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:47.027138 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f74510-23f8-4f0b-af8e-6c9861215203-credential-socket\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh\" (UID: \"d7f74510-23f8-4f0b-af8e-6c9861215203\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh" Oct 30 09:33:47.027211 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:47.027173 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7f74510-23f8-4f0b-af8e-6c9861215203-workload-certs\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh\" (UID: \"d7f74510-23f8-4f0b-af8e-6c9861215203\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh" Oct 30 09:33:47.027290 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:47.027220 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7f74510-23f8-4f0b-af8e-6c9861215203-istio-data\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh\" (UID: \"d7f74510-23f8-4f0b-af8e-6c9861215203\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh" Oct 30 09:33:47.027290 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:47.027276 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f74510-23f8-4f0b-af8e-6c9861215203-workload-socket\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh\" (UID: \"d7f74510-23f8-4f0b-af8e-6c9861215203\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh" Oct 30 09:33:47.027644 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:47.027618 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f74510-23f8-4f0b-af8e-6c9861215203-credential-socket\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh\" (UID: \"d7f74510-23f8-4f0b-af8e-6c9861215203\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh" Oct 30 09:33:47.027775 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:47.027723 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7f74510-23f8-4f0b-af8e-6c9861215203-workload-certs\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh\" (UID: \"d7f74510-23f8-4f0b-af8e-6c9861215203\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh" Oct 30 09:33:47.027914 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:47.027895 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7f74510-23f8-4f0b-af8e-6c9861215203-istio-data\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh\" (UID: \"d7f74510-23f8-4f0b-af8e-6c9861215203\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh" Oct 30 09:33:47.027969 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:47.027932 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f74510-23f8-4f0b-af8e-6c9861215203-workload-socket\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh\" (UID: \"d7f74510-23f8-4f0b-af8e-6c9861215203\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh" Oct 30 09:33:47.028028 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:47.028004 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7f74510-23f8-4f0b-af8e-6c9861215203-istiod-ca-cert\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh\" (UID: \"d7f74510-23f8-4f0b-af8e-6c9861215203\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh" Oct 30 09:33:47.029538 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:47.029513 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7f74510-23f8-4f0b-af8e-6c9861215203-istio-envoy\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh\" (UID: \"d7f74510-23f8-4f0b-af8e-6c9861215203\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh" Oct 30 09:33:47.029747 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:47.029729 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7f74510-23f8-4f0b-af8e-6c9861215203-istio-podinfo\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh\" (UID: \"d7f74510-23f8-4f0b-af8e-6c9861215203\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh" Oct 30 09:33:47.047841 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:47.047817 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7f74510-23f8-4f0b-af8e-6c9861215203-istio-token\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh\" (UID: \"d7f74510-23f8-4f0b-af8e-6c9861215203\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh" Oct 30 09:33:47.048060 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:47.048037 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wl4g\" (UniqueName: \"kubernetes.io/projected/d7f74510-23f8-4f0b-af8e-6c9861215203-kube-api-access-9wl4g\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh\" (UID: \"d7f74510-23f8-4f0b-af8e-6c9861215203\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh" Oct 30 09:33:47.174736 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:47.174702 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh" Oct 30 09:33:47.629484 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:33:47.629443 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7f74510_23f8_4f0b_af8e_6c9861215203.slice/crio-14d4851afc2e113871d21b4ac0c00de9a6d70016014dedf28437f4bb5cc179bf WatchSource:0}: Error finding container 14d4851afc2e113871d21b4ac0c00de9a6d70016014dedf28437f4bb5cc179bf: Status 404 returned error can't find the container with id 14d4851afc2e113871d21b4ac0c00de9a6d70016014dedf28437f4bb5cc179bf Oct 30 09:33:47.632287 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:47.632261 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh"] Oct 30 09:33:47.632394 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:47.632327 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:33:47.632445 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:47.632423 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:33:47.632499 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:47.632472 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:33:47.690318 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:47.688701 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67" event={"ID":"821c88fc-b72b-4104-97f3-e3528428498e","Type":"ContainerStarted","Data":"c21f8960ca996a0e9b070c9a4bcd15ff37dfd7532869991b5332e577f3cc37e4"} Oct 30 09:33:47.691084 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:47.691059 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh" event={"ID":"d7f74510-23f8-4f0b-af8e-6c9861215203","Type":"ContainerStarted","Data":"14d4851afc2e113871d21b4ac0c00de9a6d70016014dedf28437f4bb5cc179bf"} Oct 30 09:33:48.596850 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:48.596807 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67" Oct 30 09:33:48.597042 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:48.597007 4562 patch_prober.go:29] interesting pod/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Oct 30 09:33:48.597093 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:48.597051 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67" podUID="821c88fc-b72b-4104-97f3-e3528428498e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Oct 30 09:33:48.701834 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:48.701788 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh" event={"ID":"d7f74510-23f8-4f0b-af8e-6c9861215203","Type":"ContainerStarted","Data":"d19181cb5caa03a0eca73cae1ddd5f62d68d096a3eaed7827dfec612c8f21545"} Oct 30 09:33:48.787305 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:48.787206 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh" podStartSLOduration=2.7871813579999998 podStartE2EDuration="2.787181358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:33:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:33:48.760944915 +0000 UTC m=+2389.022505928" watchObservedRunningTime="2026-10-30 09:33:48.787181358 +0000 UTC m=+2389.048742347" Oct 30 09:33:49.182766 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:49.182730 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh" Oct 30 09:33:49.183173 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:49.183148 4562 patch_prober.go:29] interesting pod/e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body= Oct 30 09:33:49.183350 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:49.183326 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh" podUID="d7f74510-23f8-4f0b-af8e-6c9861215203" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" Oct 30 09:33:49.596895 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:49.596857 4562 patch_prober.go:29] interesting pod/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Oct 30 09:33:49.597105 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:49.596925 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67" podUID="821c88fc-b72b-4104-97f3-e3528428498e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Oct 30 09:33:50.190279 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:50.190073 4562 patch_prober.go:29] interesting pod/e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body= Oct 30 09:33:50.190279 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:50.190131 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh" podUID="d7f74510-23f8-4f0b-af8e-6c9861215203" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" Oct 30 09:33:50.597446 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:50.597337 4562 patch_prober.go:29] interesting pod/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Oct 30 09:33:50.597586 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:50.597435 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67" podUID="821c88fc-b72b-4104-97f3-e3528428498e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Oct 30 09:33:51.203409 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:51.203044 4562 patch_prober.go:29] interesting pod/e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body= Oct 30 09:33:51.203409 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:51.203121 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh" podUID="d7f74510-23f8-4f0b-af8e-6c9861215203" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" Oct 30 09:33:51.299026 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:51.298991 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-15e1cc-c988685bd-6z7kz"] Oct 30 09:33:51.299252 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:51.299224 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-15e1cc-c988685bd-6z7kz" podUID="62afe007-634a-41ab-9a9b-b10b4578452e" containerName="payload-processing" containerID="cri-o://b26cfa46daaf6d09b79191fa68295550341d62e66ac7de354de0e208cf9869e7" gracePeriod=30 Oct 30 09:33:51.334573 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:51.334518 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-15e1cc-79d76cd75-hkf2r"] Oct 30 09:33:51.335080 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:51.334888 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-15e1cc-79d76cd75-hkf2r" podUID="10e23f86-d880-4196-b329-175b8629d542" containerName="payload-pre-processing" containerID="cri-o://da3911b4a92a556c23130fbfcfc5cb525dfedd5abf19c62d6546765b80236e5a" gracePeriod=30 Oct 30 09:33:51.606543 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:51.605924 4562 patch_prober.go:29] interesting pod/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Oct 30 09:33:51.606543 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:51.605988 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67" podUID="821c88fc-b72b-4104-97f3-e3528428498e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Oct 30 09:33:52.019334 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:52.018982 4562 generic.go:356] "Generic (PLEG): container finished" podID="62afe007-634a-41ab-9a9b-b10b4578452e" containerID="b26cfa46daaf6d09b79191fa68295550341d62e66ac7de354de0e208cf9869e7" exitCode=0 Oct 30 09:33:52.026870 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:52.025130 4562 generic.go:356] "Generic (PLEG): container finished" podID="10e23f86-d880-4196-b329-175b8629d542" containerID="da3911b4a92a556c23130fbfcfc5cb525dfedd5abf19c62d6546765b80236e5a" exitCode=0 Oct 30 09:33:52.028260 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:52.028013 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-15e1cc-c988685bd-6z7kz" event={"ID":"62afe007-634a-41ab-9a9b-b10b4578452e","Type":"ContainerDied","Data":"b26cfa46daaf6d09b79191fa68295550341d62e66ac7de354de0e208cf9869e7"} Oct 30 09:33:52.028260 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:52.028075 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-15e1cc-79d76cd75-hkf2r" event={"ID":"10e23f86-d880-4196-b329-175b8629d542","Type":"ContainerDied","Data":"da3911b4a92a556c23130fbfcfc5cb525dfedd5abf19c62d6546765b80236e5a"} Oct 30 09:33:52.296455 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:52.295737 4562 patch_prober.go:29] interesting pod/e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body= Oct 30 09:33:52.296455 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:52.295897 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh" podUID="d7f74510-23f8-4f0b-af8e-6c9861215203" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" Oct 30 09:33:52.297669 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:52.297648 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-15e1cc-c988685bd-6z7kz" Oct 30 09:33:52.522534 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:52.522325 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62afe007-634a-41ab-9a9b-b10b4578452e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62afe007-634a-41ab-9a9b-b10b4578452e-plugins-config-volume\") pod \"62afe007-634a-41ab-9a9b-b10b4578452e\" (UID: \"62afe007-634a-41ab-9a9b-b10b4578452e\") " Oct 30 09:33:52.522534 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:52.522413 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62afe007-634a-41ab-9a9b-b10b4578452e-kube-api-access-lq5br\" (UniqueName: \"kubernetes.io/projected/62afe007-634a-41ab-9a9b-b10b4578452e-kube-api-access-lq5br\") pod \"62afe007-634a-41ab-9a9b-b10b4578452e\" (UID: \"62afe007-634a-41ab-9a9b-b10b4578452e\") " Oct 30 09:33:52.527699 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:52.527061 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62afe007-634a-41ab-9a9b-b10b4578452e-kube-api-access-lq5br" pod "62afe007-634a-41ab-9a9b-b10b4578452e" (UID: "62afe007-634a-41ab-9a9b-b10b4578452e"). InnerVolumeSpecName "kube-api-access-lq5br". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:33:52.527699 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:52.527232 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62afe007-634a-41ab-9a9b-b10b4578452e-plugins-config-volume" pod "62afe007-634a-41ab-9a9b-b10b4578452e" (UID: "62afe007-634a-41ab-9a9b-b10b4578452e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:33:52.609518 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:52.601041 4562 patch_prober.go:29] interesting pod/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Oct 30 09:33:52.609518 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:52.601133 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67" podUID="821c88fc-b72b-4104-97f3-e3528428498e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Oct 30 09:33:52.627949 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:52.627910 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lq5br\" (UniqueName: \"kubernetes.io/projected/62afe007-634a-41ab-9a9b-b10b4578452e-kube-api-access-lq5br\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:33:52.628194 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:52.628174 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62afe007-634a-41ab-9a9b-b10b4578452e-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:33:52.670415 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:52.663374 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-15e1cc-79d76cd75-hkf2r" Oct 30 09:33:52.830543 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:52.830045 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10e23f86-d880-4196-b329-175b8629d542-kube-api-access-szxjr\" (UniqueName: \"kubernetes.io/projected/10e23f86-d880-4196-b329-175b8629d542-kube-api-access-szxjr\") pod \"10e23f86-d880-4196-b329-175b8629d542\" (UID: \"10e23f86-d880-4196-b329-175b8629d542\") " Oct 30 09:33:52.830543 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:52.830091 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10e23f86-d880-4196-b329-175b8629d542-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10e23f86-d880-4196-b329-175b8629d542-plugins-config-volume\") pod \"10e23f86-d880-4196-b329-175b8629d542\" (UID: \"10e23f86-d880-4196-b329-175b8629d542\") " Oct 30 09:33:52.830752 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:52.830722 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10e23f86-d880-4196-b329-175b8629d542-plugins-config-volume" pod "10e23f86-d880-4196-b329-175b8629d542" (UID: "10e23f86-d880-4196-b329-175b8629d542"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:33:52.832443 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:52.832414 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10e23f86-d880-4196-b329-175b8629d542-kube-api-access-szxjr" pod "10e23f86-d880-4196-b329-175b8629d542" (UID: "10e23f86-d880-4196-b329-175b8629d542"). InnerVolumeSpecName "kube-api-access-szxjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:33:52.965407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:52.964159 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szxjr\" (UniqueName: \"kubernetes.io/projected/10e23f86-d880-4196-b329-175b8629d542-kube-api-access-szxjr\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:33:52.965407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:52.964210 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10e23f86-d880-4196-b329-175b8629d542-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:33:53.033390 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:53.033343 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-15e1cc-79d76cd75-hkf2r" event={"ID":"10e23f86-d880-4196-b329-175b8629d542","Type":"ContainerDied","Data":"45c04102f7b0920ce77652acff2780b18191052dfc1e3c1f64a9225c837a5fd2"} Oct 30 09:33:53.033579 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:53.033441 4562 scope.go:122] "RemoveContainer" containerID="da3911b4a92a556c23130fbfcfc5cb525dfedd5abf19c62d6546765b80236e5a" Oct 30 09:33:53.033579 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:53.033552 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-15e1cc-79d76cd75-hkf2r" Oct 30 09:33:53.036485 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:53.036442 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-15e1cc-c988685bd-6z7kz" event={"ID":"62afe007-634a-41ab-9a9b-b10b4578452e","Type":"ContainerDied","Data":"a3d6a0c1d8c5974cc98f8501c0f31bf50b701392baaf1574fce58080be43c4cc"} Oct 30 09:33:53.036606 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:53.036530 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-15e1cc-c988685bd-6z7kz" Oct 30 09:33:53.091284 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:53.085768 4562 scope.go:122] "RemoveContainer" containerID="b26cfa46daaf6d09b79191fa68295550341d62e66ac7de354de0e208cf9869e7" Oct 30 09:33:53.175097 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:53.175061 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-15e1cc-79d76cd75-hkf2r"] Oct 30 09:33:53.175486 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:53.175457 4562 patch_prober.go:29] interesting pod/e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body= Oct 30 09:33:53.175590 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:53.175523 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh" podUID="d7f74510-23f8-4f0b-af8e-6c9861215203" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" Oct 30 09:33:53.178975 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:53.178949 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-15e1cc-79d76cd75-hkf2r"] Oct 30 09:33:53.222345 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:53.222245 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-15e1cc-c988685bd-6z7kz"] Oct 30 09:33:53.225938 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:53.225905 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-15e1cc-c988685bd-6z7kz"] Oct 30 09:33:53.617535 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:53.617446 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67" Oct 30 09:33:53.618424 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:53.618365 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67" Oct 30 09:33:53.619104 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:53.619083 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67" Oct 30 09:33:53.882510 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:53.882428 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10e23f86-d880-4196-b329-175b8629d542" path="/var/lib/kubelet/pods/10e23f86-d880-4196-b329-175b8629d542/volumes" Oct 30 09:33:53.883037 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:53.883019 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62afe007-634a-41ab-9a9b-b10b4578452e" path="/var/lib/kubelet/pods/62afe007-634a-41ab-9a9b-b10b4578452e/volumes" Oct 30 09:33:54.176882 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:54.176774 4562 patch_prober.go:29] interesting pod/e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body= Oct 30 09:33:54.176882 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:54.176854 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh" podUID="d7f74510-23f8-4f0b-af8e-6c9861215203" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" Oct 30 09:33:55.180111 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:55.180084 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh" Oct 30 09:33:55.180542 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:55.180367 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh" Oct 30 09:33:55.181074 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:55.181053 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh" Oct 30 09:33:55.668856 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:55.668824 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7a213d-65fd87589f-rr7jz" Oct 30 09:33:55.669049 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:33:55.668893 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7a213d-689cb5bf76-9b72m" Oct 30 09:33:58.636428 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:33:58.635520 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:34:00.211404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:00.210686 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6q7n_1f51db5a-5720-4dea-ad02-f336b35d43b1/ovn-acl-logging/0.log" Oct 30 09:34:00.220260 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:00.220236 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6q7n_1f51db5a-5720-4dea-ad02-f336b35d43b1/ovn-acl-logging/0.log" Oct 30 09:34:02.716863 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:02.716826 4562 scope.go:122] "RemoveContainer" containerID="799122ba9c236f204c0702b715ddc327b30d2516497c4a2ad8d12666e8a66d82" Oct 30 09:34:05.993246 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:05.993207 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-1413c8-76f699c45f-rv8gt"] Oct 30 09:34:05.994245 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:05.994218 4562 cpu_manager.go:414] "containerMap: removing container" podUID="10e23f86-d880-4196-b329-175b8629d542" containerName="payload-pre-processing" Oct 30 09:34:05.994245 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:05.994244 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10e23f86-d880-4196-b329-175b8629d542" containerName="payload-pre-processing" Oct 30 09:34:05.994451 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:05.994323 4562 cpu_manager.go:414] "containerMap: removing container" podUID="62afe007-634a-41ab-9a9b-b10b4578452e" containerName="payload-processing" Oct 30 09:34:05.994451 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:05.994332 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62afe007-634a-41ab-9a9b-b10b4578452e" containerName="payload-processing" Oct 30 09:34:05.994520 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:05.994510 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="62afe007-634a-41ab-9a9b-b10b4578452e" containerName="payload-processing" Oct 30 09:34:05.994562 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:05.994529 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="10e23f86-d880-4196-b329-175b8629d542" containerName="payload-pre-processing" Oct 30 09:34:06.000279 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:06.000255 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-1413c8-76f699c45f-rv8gt" Oct 30 09:34:06.036423 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:06.030231 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-1413c8\"" Oct 30 09:34:06.036423 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:06.030473 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-1413c8-dockercfg-td8rm\"" Oct 30 09:34:06.036423 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:06.034185 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-1413c8-76f699c45f-rv8gt"] Oct 30 09:34:06.039691 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:06.039650 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23788da7-9c3c-45aa-b327-5f3f05c35cb6-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-1413c8-76f699c45f-rv8gt\" (UID: \"23788da7-9c3c-45aa-b327-5f3f05c35cb6\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-1413c8-76f699c45f-rv8gt" Oct 30 09:34:06.040428 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:06.039722 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p8dx\" (UniqueName: \"kubernetes.io/projected/23788da7-9c3c-45aa-b327-5f3f05c35cb6-kube-api-access-4p8dx\") pod \"payload-pre-processing-e2e-explicit-1413c8-76f699c45f-rv8gt\" (UID: \"23788da7-9c3c-45aa-b327-5f3f05c35cb6\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-1413c8-76f699c45f-rv8gt" Oct 30 09:34:06.066595 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:06.066566 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-1413c8-6d66d75c58-ncqbq"] Oct 30 09:34:06.093323 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:06.093297 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-1413c8-6d66d75c58-ncqbq"] Oct 30 09:34:06.093536 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:06.093418 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-1413c8-6d66d75c58-ncqbq" Oct 30 09:34:06.195467 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:06.167654 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4084b690-89cf-4f47-911e-a49aa5368653-plugins-config-volume\") pod \"payload-processing-e2e-explicit-1413c8-6d66d75c58-ncqbq\" (UID: \"4084b690-89cf-4f47-911e-a49aa5368653\") " pod="openshift-ingress/payload-processing-e2e-explicit-1413c8-6d66d75c58-ncqbq" Oct 30 09:34:06.195467 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:06.167941 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23788da7-9c3c-45aa-b327-5f3f05c35cb6-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-1413c8-76f699c45f-rv8gt\" (UID: \"23788da7-9c3c-45aa-b327-5f3f05c35cb6\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-1413c8-76f699c45f-rv8gt" Oct 30 09:34:06.195467 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:06.168004 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p8dx\" (UniqueName: \"kubernetes.io/projected/23788da7-9c3c-45aa-b327-5f3f05c35cb6-kube-api-access-4p8dx\") pod \"payload-pre-processing-e2e-explicit-1413c8-76f699c45f-rv8gt\" (UID: \"23788da7-9c3c-45aa-b327-5f3f05c35cb6\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-1413c8-76f699c45f-rv8gt" Oct 30 09:34:06.195467 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:06.168193 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r427g\" (UniqueName: \"kubernetes.io/projected/4084b690-89cf-4f47-911e-a49aa5368653-kube-api-access-r427g\") pod \"payload-processing-e2e-explicit-1413c8-6d66d75c58-ncqbq\" (UID: \"4084b690-89cf-4f47-911e-a49aa5368653\") " pod="openshift-ingress/payload-processing-e2e-explicit-1413c8-6d66d75c58-ncqbq" Oct 30 09:34:06.195467 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:06.169254 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23788da7-9c3c-45aa-b327-5f3f05c35cb6-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-1413c8-76f699c45f-rv8gt\" (UID: \"23788da7-9c3c-45aa-b327-5f3f05c35cb6\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-1413c8-76f699c45f-rv8gt" Oct 30 09:34:06.301062 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:06.300964 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r427g\" (UniqueName: \"kubernetes.io/projected/4084b690-89cf-4f47-911e-a49aa5368653-kube-api-access-r427g\") pod \"payload-processing-e2e-explicit-1413c8-6d66d75c58-ncqbq\" (UID: \"4084b690-89cf-4f47-911e-a49aa5368653\") " pod="openshift-ingress/payload-processing-e2e-explicit-1413c8-6d66d75c58-ncqbq" Oct 30 09:34:06.301342 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:06.301320 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4084b690-89cf-4f47-911e-a49aa5368653-plugins-config-volume\") pod \"payload-processing-e2e-explicit-1413c8-6d66d75c58-ncqbq\" (UID: \"4084b690-89cf-4f47-911e-a49aa5368653\") " pod="openshift-ingress/payload-processing-e2e-explicit-1413c8-6d66d75c58-ncqbq" Oct 30 09:34:06.302911 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:06.302892 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4084b690-89cf-4f47-911e-a49aa5368653-plugins-config-volume\") pod \"payload-processing-e2e-explicit-1413c8-6d66d75c58-ncqbq\" (UID: \"4084b690-89cf-4f47-911e-a49aa5368653\") " pod="openshift-ingress/payload-processing-e2e-explicit-1413c8-6d66d75c58-ncqbq" Oct 30 09:34:06.332415 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:06.326916 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p8dx\" (UniqueName: \"kubernetes.io/projected/23788da7-9c3c-45aa-b327-5f3f05c35cb6-kube-api-access-4p8dx\") pod \"payload-pre-processing-e2e-explicit-1413c8-76f699c45f-rv8gt\" (UID: \"23788da7-9c3c-45aa-b327-5f3f05c35cb6\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-1413c8-76f699c45f-rv8gt" Oct 30 09:34:06.351435 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:06.351341 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r427g\" (UniqueName: \"kubernetes.io/projected/4084b690-89cf-4f47-911e-a49aa5368653-kube-api-access-r427g\") pod \"payload-processing-e2e-explicit-1413c8-6d66d75c58-ncqbq\" (UID: \"4084b690-89cf-4f47-911e-a49aa5368653\") " pod="openshift-ingress/payload-processing-e2e-explicit-1413c8-6d66d75c58-ncqbq" Oct 30 09:34:06.552006 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:06.551907 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-1413c8-6d66d75c58-ncqbq" Oct 30 09:34:06.617561 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:06.615090 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-1413c8-76f699c45f-rv8gt" Oct 30 09:34:06.861571 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:34:06.861538 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4084b690_89cf_4f47_911e_a49aa5368653.slice/crio-a14dda18406b2d62ccdc0894efc1328fe050a65d3fab5165bc66364f088f6af0 WatchSource:0}: Error finding container a14dda18406b2d62ccdc0894efc1328fe050a65d3fab5165bc66364f088f6af0: Status 404 returned error can't find the container with id a14dda18406b2d62ccdc0894efc1328fe050a65d3fab5165bc66364f088f6af0 Oct 30 09:34:06.863265 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:06.863241 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-1413c8-6d66d75c58-ncqbq"] Oct 30 09:34:06.929986 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:34:06.929948 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23788da7_9c3c_45aa_b327_5f3f05c35cb6.slice/crio-af5e2ff2cce2578848e89db3a929653a9245a2770a839c4828b86200f4c3a424 WatchSource:0}: Error finding container af5e2ff2cce2578848e89db3a929653a9245a2770a839c4828b86200f4c3a424: Status 404 returned error can't find the container with id af5e2ff2cce2578848e89db3a929653a9245a2770a839c4828b86200f4c3a424 Oct 30 09:34:06.932709 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:06.932686 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-1413c8-76f699c45f-rv8gt"] Oct 30 09:34:07.332482 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:07.332434 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-1413c8-76f699c45f-rv8gt" event={"ID":"23788da7-9c3c-45aa-b327-5f3f05c35cb6","Type":"ContainerStarted","Data":"5fe063f4ebaa9701fa8eef5294797dece6f568b69bcf8d70e80c1075b02c6655"} Oct 30 09:34:07.332940 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:07.332502 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-1413c8-76f699c45f-rv8gt" event={"ID":"23788da7-9c3c-45aa-b327-5f3f05c35cb6","Type":"ContainerStarted","Data":"af5e2ff2cce2578848e89db3a929653a9245a2770a839c4828b86200f4c3a424"} Oct 30 09:34:07.332940 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:07.332531 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-1413c8-76f699c45f-rv8gt" Oct 30 09:34:07.334096 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:07.334065 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-1413c8-6d66d75c58-ncqbq" event={"ID":"4084b690-89cf-4f47-911e-a49aa5368653","Type":"ContainerStarted","Data":"1c4753e6d584c6f971a048d687cab572d054a22aa0b189ea2f9962bf4c706193"} Oct 30 09:34:07.334218 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:07.334110 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-1413c8-6d66d75c58-ncqbq" event={"ID":"4084b690-89cf-4f47-911e-a49aa5368653","Type":"ContainerStarted","Data":"a14dda18406b2d62ccdc0894efc1328fe050a65d3fab5165bc66364f088f6af0"} Oct 30 09:34:07.334218 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:07.334145 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-1413c8-6d66d75c58-ncqbq" Oct 30 09:34:07.375135 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:07.375071 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-1413c8-76f699c45f-rv8gt" podStartSLOduration=2.375053833 podStartE2EDuration="2.375053833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:34:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:34:07.372433297 +0000 UTC m=+2407.633994285" watchObservedRunningTime="2026-10-30 09:34:07.375053833 +0000 UTC m=+2407.636614821" Oct 30 09:34:07.414762 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:07.414712 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-1413c8-6d66d75c58-ncqbq" podStartSLOduration=1.414697802 podStartE2EDuration="1.414697802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:34:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:34:07.41212998 +0000 UTC m=+2407.673690967" watchObservedRunningTime="2026-10-30 09:34:07.414697802 +0000 UTC m=+2407.676258790" Oct 30 09:34:11.704305 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:11.704268 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g"] Oct 30 09:34:11.704806 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:11.704472 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g" podUID="0d2f5f60-9f83-49d0-aac6-94d56afc2d56" containerName="istio-proxy" containerID="cri-o://07d2eae4fd38949486594f677cab355a35c411e53d804c78081372e1d0527f9b" gracePeriod=30 Oct 30 09:34:17.115179 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:17.115144 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g" Oct 30 09:34:17.222899 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:17.222851 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d2f5f60-9f83-49d0-aac6-94d56afc2d56-istio-token\" (UniqueName: \"kubernetes.io/projected/0d2f5f60-9f83-49d0-aac6-94d56afc2d56-istio-token\") pod \"0d2f5f60-9f83-49d0-aac6-94d56afc2d56\" (UID: \"0d2f5f60-9f83-49d0-aac6-94d56afc2d56\") " Oct 30 09:34:17.223110 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:17.222922 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d2f5f60-9f83-49d0-aac6-94d56afc2d56-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d2f5f60-9f83-49d0-aac6-94d56afc2d56-istio-podinfo\") pod \"0d2f5f60-9f83-49d0-aac6-94d56afc2d56\" (UID: \"0d2f5f60-9f83-49d0-aac6-94d56afc2d56\") " Oct 30 09:34:17.223110 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:17.222989 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d2f5f60-9f83-49d0-aac6-94d56afc2d56-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d2f5f60-9f83-49d0-aac6-94d56afc2d56-workload-certs\") pod \"0d2f5f60-9f83-49d0-aac6-94d56afc2d56\" (UID: \"0d2f5f60-9f83-49d0-aac6-94d56afc2d56\") " Oct 30 09:34:17.223110 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:17.223019 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d2f5f60-9f83-49d0-aac6-94d56afc2d56-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d2f5f60-9f83-49d0-aac6-94d56afc2d56-workload-socket\") pod \"0d2f5f60-9f83-49d0-aac6-94d56afc2d56\" (UID: \"0d2f5f60-9f83-49d0-aac6-94d56afc2d56\") " Oct 30 09:34:17.223291 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:17.223146 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d2f5f60-9f83-49d0-aac6-94d56afc2d56-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d2f5f60-9f83-49d0-aac6-94d56afc2d56-istio-envoy\") pod \"0d2f5f60-9f83-49d0-aac6-94d56afc2d56\" (UID: \"0d2f5f60-9f83-49d0-aac6-94d56afc2d56\") " Oct 30 09:34:17.223291 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:17.223284 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d2f5f60-9f83-49d0-aac6-94d56afc2d56-kube-api-access-pjcr6\" (UniqueName: \"kubernetes.io/projected/0d2f5f60-9f83-49d0-aac6-94d56afc2d56-kube-api-access-pjcr6\") pod \"0d2f5f60-9f83-49d0-aac6-94d56afc2d56\" (UID: \"0d2f5f60-9f83-49d0-aac6-94d56afc2d56\") " Oct 30 09:34:17.223427 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:17.223317 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d2f5f60-9f83-49d0-aac6-94d56afc2d56-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d2f5f60-9f83-49d0-aac6-94d56afc2d56-istio-data\") pod \"0d2f5f60-9f83-49d0-aac6-94d56afc2d56\" (UID: \"0d2f5f60-9f83-49d0-aac6-94d56afc2d56\") " Oct 30 09:34:17.223427 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:17.223371 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d2f5f60-9f83-49d0-aac6-94d56afc2d56-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d2f5f60-9f83-49d0-aac6-94d56afc2d56-credential-socket\") pod \"0d2f5f60-9f83-49d0-aac6-94d56afc2d56\" (UID: \"0d2f5f60-9f83-49d0-aac6-94d56afc2d56\") " Oct 30 09:34:17.223553 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:17.223431 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d2f5f60-9f83-49d0-aac6-94d56afc2d56-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d2f5f60-9f83-49d0-aac6-94d56afc2d56-istiod-ca-cert\") pod \"0d2f5f60-9f83-49d0-aac6-94d56afc2d56\" (UID: \"0d2f5f60-9f83-49d0-aac6-94d56afc2d56\") " Oct 30 09:34:17.223614 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:17.223593 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d2f5f60-9f83-49d0-aac6-94d56afc2d56-workload-socket" pod "0d2f5f60-9f83-49d0-aac6-94d56afc2d56" (UID: "0d2f5f60-9f83-49d0-aac6-94d56afc2d56"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:34:17.223825 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:17.223800 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d2f5f60-9f83-49d0-aac6-94d56afc2d56-workload-certs" pod "0d2f5f60-9f83-49d0-aac6-94d56afc2d56" (UID: "0d2f5f60-9f83-49d0-aac6-94d56afc2d56"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:34:17.224174 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:17.224147 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d2f5f60-9f83-49d0-aac6-94d56afc2d56-istiod-ca-cert" pod "0d2f5f60-9f83-49d0-aac6-94d56afc2d56" (UID: "0d2f5f60-9f83-49d0-aac6-94d56afc2d56"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:34:17.224492 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:17.224458 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d2f5f60-9f83-49d0-aac6-94d56afc2d56-istio-data" pod "0d2f5f60-9f83-49d0-aac6-94d56afc2d56" (UID: "0d2f5f60-9f83-49d0-aac6-94d56afc2d56"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:34:17.224588 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:17.224555 4562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d2f5f60-9f83-49d0-aac6-94d56afc2d56-workload-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:34:17.224588 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:17.224581 4562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d2f5f60-9f83-49d0-aac6-94d56afc2d56-workload-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:34:17.224714 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:17.224597 4562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d2f5f60-9f83-49d0-aac6-94d56afc2d56-istiod-ca-cert\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:34:17.224714 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:17.224645 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d2f5f60-9f83-49d0-aac6-94d56afc2d56-credential-socket" pod "0d2f5f60-9f83-49d0-aac6-94d56afc2d56" (UID: "0d2f5f60-9f83-49d0-aac6-94d56afc2d56"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:34:17.225689 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:17.225667 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d2f5f60-9f83-49d0-aac6-94d56afc2d56-istio-token" pod "0d2f5f60-9f83-49d0-aac6-94d56afc2d56" (UID: "0d2f5f60-9f83-49d0-aac6-94d56afc2d56"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:34:17.227044 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:17.226983 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d2f5f60-9f83-49d0-aac6-94d56afc2d56-istio-envoy" pod "0d2f5f60-9f83-49d0-aac6-94d56afc2d56" (UID: "0d2f5f60-9f83-49d0-aac6-94d56afc2d56"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:34:17.227323 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:17.227292 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d2f5f60-9f83-49d0-aac6-94d56afc2d56-istio-podinfo" pod "0d2f5f60-9f83-49d0-aac6-94d56afc2d56" (UID: "0d2f5f60-9f83-49d0-aac6-94d56afc2d56"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:34:17.227492 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:17.227474 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d2f5f60-9f83-49d0-aac6-94d56afc2d56-kube-api-access-pjcr6" pod "0d2f5f60-9f83-49d0-aac6-94d56afc2d56" (UID: "0d2f5f60-9f83-49d0-aac6-94d56afc2d56"). InnerVolumeSpecName "kube-api-access-pjcr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:34:17.326588 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:17.326552 4562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d2f5f60-9f83-49d0-aac6-94d56afc2d56-istio-envoy\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:34:17.326588 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:17.326591 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjcr6\" (UniqueName: \"kubernetes.io/projected/0d2f5f60-9f83-49d0-aac6-94d56afc2d56-kube-api-access-pjcr6\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:34:17.326803 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:17.326606 4562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d2f5f60-9f83-49d0-aac6-94d56afc2d56-istio-data\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:34:17.326803 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:17.326619 4562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d2f5f60-9f83-49d0-aac6-94d56afc2d56-credential-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:34:17.326803 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:17.326634 4562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d2f5f60-9f83-49d0-aac6-94d56afc2d56-istio-token\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:34:17.326803 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:17.326650 4562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d2f5f60-9f83-49d0-aac6-94d56afc2d56-istio-podinfo\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:34:17.400756 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:17.400722 4562 generic.go:356] "Generic (PLEG): container finished" podID="0d2f5f60-9f83-49d0-aac6-94d56afc2d56" containerID="07d2eae4fd38949486594f677cab355a35c411e53d804c78081372e1d0527f9b" exitCode=0 Oct 30 09:34:17.400935 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:17.400768 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g" Oct 30 09:34:17.400935 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:17.400802 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g" event={"ID":"0d2f5f60-9f83-49d0-aac6-94d56afc2d56","Type":"ContainerDied","Data":"07d2eae4fd38949486594f677cab355a35c411e53d804c78081372e1d0527f9b"} Oct 30 09:34:17.400935 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:17.400854 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g" event={"ID":"0d2f5f60-9f83-49d0-aac6-94d56afc2d56","Type":"ContainerDied","Data":"2a408d1ac72a520551712466bf7b81a474b928ce7aed30a7f27c44b3a4bafed1"} Oct 30 09:34:17.400935 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:17.400867 4562 scope.go:122] "RemoveContainer" containerID="07d2eae4fd38949486594f677cab355a35c411e53d804c78081372e1d0527f9b" Oct 30 09:34:17.411563 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:17.411545 4562 scope.go:122] "RemoveContainer" containerID="07d2eae4fd38949486594f677cab355a35c411e53d804c78081372e1d0527f9b" Oct 30 09:34:17.411836 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:34:17.411816 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07d2eae4fd38949486594f677cab355a35c411e53d804c78081372e1d0527f9b\": container with ID starting with 07d2eae4fd38949486594f677cab355a35c411e53d804c78081372e1d0527f9b not found: ID does not exist" containerID="07d2eae4fd38949486594f677cab355a35c411e53d804c78081372e1d0527f9b" Oct 30 09:34:17.411891 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:17.411843 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07d2eae4fd38949486594f677cab355a35c411e53d804c78081372e1d0527f9b"} err="failed to get container status \"07d2eae4fd38949486594f677cab355a35c411e53d804c78081372e1d0527f9b\": rpc error: code = NotFound desc = could not find container \"07d2eae4fd38949486594f677cab355a35c411e53d804c78081372e1d0527f9b\": container with ID starting with 07d2eae4fd38949486594f677cab355a35c411e53d804c78081372e1d0527f9b not found: ID does not exist" Oct 30 09:34:17.455992 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:17.455967 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g"] Oct 30 09:34:17.471663 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:17.471452 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-ldx9g"] Oct 30 09:34:17.875638 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:17.875596 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d2f5f60-9f83-49d0-aac6-94d56afc2d56" path="/var/lib/kubelet/pods/0d2f5f60-9f83-49d0-aac6-94d56afc2d56/volumes" Oct 30 09:34:18.343473 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:18.343445 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-1413c8-76f699c45f-rv8gt" Oct 30 09:34:18.343879 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:18.343518 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-1413c8-6d66d75c58-ncqbq" Oct 30 09:34:26.643893 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:26.643859 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-1413c8-6566657db8-lcr4t"] Oct 30 09:34:26.644769 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:26.644747 4562 cpu_manager.go:414] "containerMap: removing container" podUID="0d2f5f60-9f83-49d0-aac6-94d56afc2d56" containerName="istio-proxy" Oct 30 09:34:26.644911 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:26.644770 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d2f5f60-9f83-49d0-aac6-94d56afc2d56" containerName="istio-proxy" Oct 30 09:34:26.644992 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:26.644979 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d2f5f60-9f83-49d0-aac6-94d56afc2d56" containerName="istio-proxy" Oct 30 09:34:26.650142 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:26.650119 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-1413c8-6566657db8-lcr4t" Oct 30 09:34:26.669254 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:26.669224 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-1413c8-6566657db8-lcr4t"] Oct 30 09:34:26.677609 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:26.677575 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-1413c8-serving-cert\"" Oct 30 09:34:26.681684 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:26.681650 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a009b29f-625d-4074-9a24-55f2af08717d-maas-api-tls\") pod \"maas-api-e2e-explicit-1413c8-6566657db8-lcr4t\" (UID: \"a009b29f-625d-4074-9a24-55f2af08717d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-1413c8-6566657db8-lcr4t" Oct 30 09:34:26.681852 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:26.681700 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbjt8\" (UniqueName: \"kubernetes.io/projected/a009b29f-625d-4074-9a24-55f2af08717d-kube-api-access-kbjt8\") pod \"maas-api-e2e-explicit-1413c8-6566657db8-lcr4t\" (UID: \"a009b29f-625d-4074-9a24-55f2af08717d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-1413c8-6566657db8-lcr4t" Oct 30 09:34:26.681852 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:26.681747 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a009b29f-625d-4074-9a24-55f2af08717d-openshift-service-ca\") pod \"maas-api-e2e-explicit-1413c8-6566657db8-lcr4t\" (UID: \"a009b29f-625d-4074-9a24-55f2af08717d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-1413c8-6566657db8-lcr4t" Oct 30 09:34:26.682007 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:26.681986 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a009b29f-625d-4074-9a24-55f2af08717d-metrics-certs\") pod \"maas-api-e2e-explicit-1413c8-6566657db8-lcr4t\" (UID: \"a009b29f-625d-4074-9a24-55f2af08717d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-1413c8-6566657db8-lcr4t" Oct 30 09:34:26.783687 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:26.783649 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a009b29f-625d-4074-9a24-55f2af08717d-metrics-certs\") pod \"maas-api-e2e-explicit-1413c8-6566657db8-lcr4t\" (UID: \"a009b29f-625d-4074-9a24-55f2af08717d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-1413c8-6566657db8-lcr4t" Oct 30 09:34:26.783879 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:26.783726 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a009b29f-625d-4074-9a24-55f2af08717d-maas-api-tls\") pod \"maas-api-e2e-explicit-1413c8-6566657db8-lcr4t\" (UID: \"a009b29f-625d-4074-9a24-55f2af08717d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-1413c8-6566657db8-lcr4t" Oct 30 09:34:26.783879 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:26.783744 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbjt8\" (UniqueName: \"kubernetes.io/projected/a009b29f-625d-4074-9a24-55f2af08717d-kube-api-access-kbjt8\") pod \"maas-api-e2e-explicit-1413c8-6566657db8-lcr4t\" (UID: \"a009b29f-625d-4074-9a24-55f2af08717d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-1413c8-6566657db8-lcr4t" Oct 30 09:34:26.783879 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:34:26.783852 4562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-1413c8-serving-cert: secret "maas-api-e2e-explicit-1413c8-serving-cert" not found Oct 30 09:34:26.784355 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:26.784321 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a009b29f-625d-4074-9a24-55f2af08717d-openshift-service-ca\") pod \"maas-api-e2e-explicit-1413c8-6566657db8-lcr4t\" (UID: \"a009b29f-625d-4074-9a24-55f2af08717d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-1413c8-6566657db8-lcr4t" Oct 30 09:34:26.784578 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:34:26.784432 4562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a009b29f-625d-4074-9a24-55f2af08717d-maas-api-tls podName:a009b29f-625d-4074-9a24-55f2af08717d nodeName:}" failed. No retries permitted until 2026-10-30 09:34:27.284358782 +0000 UTC m=+2427.545919815 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a009b29f-625d-4074-9a24-55f2af08717d-maas-api-tls") pod "maas-api-e2e-explicit-1413c8-6566657db8-lcr4t" (UID: "a009b29f-625d-4074-9a24-55f2af08717d") : secret "maas-api-e2e-explicit-1413c8-serving-cert" not found Oct 30 09:34:26.787419 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:26.785505 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a009b29f-625d-4074-9a24-55f2af08717d-openshift-service-ca\") pod \"maas-api-e2e-explicit-1413c8-6566657db8-lcr4t\" (UID: \"a009b29f-625d-4074-9a24-55f2af08717d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-1413c8-6566657db8-lcr4t" Oct 30 09:34:26.796766 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:26.796742 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a009b29f-625d-4074-9a24-55f2af08717d-metrics-certs\") pod \"maas-api-e2e-explicit-1413c8-6566657db8-lcr4t\" (UID: \"a009b29f-625d-4074-9a24-55f2af08717d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-1413c8-6566657db8-lcr4t" Oct 30 09:34:26.804807 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:26.804779 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbjt8\" (UniqueName: \"kubernetes.io/projected/a009b29f-625d-4074-9a24-55f2af08717d-kube-api-access-kbjt8\") pod \"maas-api-e2e-explicit-1413c8-6566657db8-lcr4t\" (UID: \"a009b29f-625d-4074-9a24-55f2af08717d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-1413c8-6566657db8-lcr4t" Oct 30 09:34:27.291434 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:27.291361 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a009b29f-625d-4074-9a24-55f2af08717d-maas-api-tls\") pod \"maas-api-e2e-explicit-1413c8-6566657db8-lcr4t\" (UID: \"a009b29f-625d-4074-9a24-55f2af08717d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-1413c8-6566657db8-lcr4t" Oct 30 09:34:27.293784 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:27.293754 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a009b29f-625d-4074-9a24-55f2af08717d-maas-api-tls\") pod \"maas-api-e2e-explicit-1413c8-6566657db8-lcr4t\" (UID: \"a009b29f-625d-4074-9a24-55f2af08717d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-1413c8-6566657db8-lcr4t" Oct 30 09:34:27.566326 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:27.566232 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-1413c8-6566657db8-lcr4t" Oct 30 09:34:27.882696 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:34:27.882664 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda009b29f_625d_4074_9a24_55f2af08717d.slice/crio-c4fbc88636deccf7348ebe34ec5169fce5c9c2dd421ec84a0c1844fa8fda2bf7 WatchSource:0}: Error finding container c4fbc88636deccf7348ebe34ec5169fce5c9c2dd421ec84a0c1844fa8fda2bf7: Status 404 returned error can't find the container with id c4fbc88636deccf7348ebe34ec5169fce5c9c2dd421ec84a0c1844fa8fda2bf7 Oct 30 09:34:27.884170 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:27.884145 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-1413c8-6566657db8-lcr4t"] Oct 30 09:34:28.566644 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:28.566600 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-1413c8-6566657db8-lcr4t" event={"ID":"a009b29f-625d-4074-9a24-55f2af08717d","Type":"ContainerStarted","Data":"2b9edfc3a8fac46e2d9cb71ff42bb7215f6ecf52aeb69d6281f93360b8ba3997"} Oct 30 09:34:28.566839 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:28.566667 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-1413c8-6566657db8-lcr4t" event={"ID":"a009b29f-625d-4074-9a24-55f2af08717d","Type":"ContainerStarted","Data":"c4fbc88636deccf7348ebe34ec5169fce5c9c2dd421ec84a0c1844fa8fda2bf7"} Oct 30 09:34:28.566839 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:28.566694 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-1413c8-6566657db8-lcr4t" Oct 30 09:34:28.597843 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:34:28.597802 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:34:28.607534 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:28.607442 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-1413c8-6566657db8-lcr4t" podStartSLOduration=2.206319301 podStartE2EDuration="2.607412683s" totalImagesPullingTime="401.093382ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:34:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:34:28.606902006 +0000 UTC m=+2428.868462995" watchObservedRunningTime="2026-10-30 09:34:28.607412683 +0000 UTC m=+2428.868973672" Oct 30 09:34:31.980939 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:31.980906 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd"] Oct 30 09:34:31.985237 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:31.985206 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd" Oct 30 09:34:32.098445 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:32.011989 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd"] Oct 30 09:34:32.098445 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:32.048394 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fca872d-4848-4886-925d-e06a5bcc2526-workload-certs\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd\" (UID: \"2fca872d-4848-4886-925d-e06a5bcc2526\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd" Oct 30 09:34:32.098445 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:32.048438 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fca872d-4848-4886-925d-e06a5bcc2526-istio-podinfo\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd\" (UID: \"2fca872d-4848-4886-925d-e06a5bcc2526\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd" Oct 30 09:34:32.098445 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:32.048481 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-882g7\" (UniqueName: \"kubernetes.io/projected/2fca872d-4848-4886-925d-e06a5bcc2526-kube-api-access-882g7\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd\" (UID: \"2fca872d-4848-4886-925d-e06a5bcc2526\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd" Oct 30 09:34:32.098445 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:32.048516 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fca872d-4848-4886-925d-e06a5bcc2526-istio-envoy\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd\" (UID: \"2fca872d-4848-4886-925d-e06a5bcc2526\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd" Oct 30 09:34:32.098445 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:32.067451 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fca872d-4848-4886-925d-e06a5bcc2526-istiod-ca-cert\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd\" (UID: \"2fca872d-4848-4886-925d-e06a5bcc2526\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd" Oct 30 09:34:32.098445 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:32.067576 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fca872d-4848-4886-925d-e06a5bcc2526-credential-socket\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd\" (UID: \"2fca872d-4848-4886-925d-e06a5bcc2526\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd" Oct 30 09:34:32.098445 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:32.067691 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fca872d-4848-4886-925d-e06a5bcc2526-workload-socket\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd\" (UID: \"2fca872d-4848-4886-925d-e06a5bcc2526\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd" Oct 30 09:34:32.098445 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:32.067774 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fca872d-4848-4886-925d-e06a5bcc2526-istio-token\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd\" (UID: \"2fca872d-4848-4886-925d-e06a5bcc2526\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd" Oct 30 09:34:32.098445 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:32.067813 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fca872d-4848-4886-925d-e06a5bcc2526-istio-data\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd\" (UID: \"2fca872d-4848-4886-925d-e06a5bcc2526\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd" Oct 30 09:34:32.169188 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:32.169157 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fca872d-4848-4886-925d-e06a5bcc2526-istio-token\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd\" (UID: \"2fca872d-4848-4886-925d-e06a5bcc2526\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd" Oct 30 09:34:32.169352 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:32.169200 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fca872d-4848-4886-925d-e06a5bcc2526-istio-data\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd\" (UID: \"2fca872d-4848-4886-925d-e06a5bcc2526\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd" Oct 30 09:34:32.169352 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:32.169253 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fca872d-4848-4886-925d-e06a5bcc2526-workload-certs\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd\" (UID: \"2fca872d-4848-4886-925d-e06a5bcc2526\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd" Oct 30 09:34:32.169352 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:32.169274 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fca872d-4848-4886-925d-e06a5bcc2526-istio-podinfo\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd\" (UID: \"2fca872d-4848-4886-925d-e06a5bcc2526\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd" Oct 30 09:34:32.169352 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:32.169300 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-882g7\" (UniqueName: \"kubernetes.io/projected/2fca872d-4848-4886-925d-e06a5bcc2526-kube-api-access-882g7\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd\" (UID: \"2fca872d-4848-4886-925d-e06a5bcc2526\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd" Oct 30 09:34:32.169570 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:32.169353 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fca872d-4848-4886-925d-e06a5bcc2526-istio-envoy\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd\" (UID: \"2fca872d-4848-4886-925d-e06a5bcc2526\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd" Oct 30 09:34:32.169570 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:32.169472 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fca872d-4848-4886-925d-e06a5bcc2526-istiod-ca-cert\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd\" (UID: \"2fca872d-4848-4886-925d-e06a5bcc2526\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd" Oct 30 09:34:32.169685 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:32.169568 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fca872d-4848-4886-925d-e06a5bcc2526-credential-socket\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd\" (UID: \"2fca872d-4848-4886-925d-e06a5bcc2526\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd" Oct 30 09:34:32.169738 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:32.169682 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fca872d-4848-4886-925d-e06a5bcc2526-workload-socket\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd\" (UID: \"2fca872d-4848-4886-925d-e06a5bcc2526\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd" Oct 30 09:34:32.169847 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:32.169818 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fca872d-4848-4886-925d-e06a5bcc2526-workload-certs\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd\" (UID: \"2fca872d-4848-4886-925d-e06a5bcc2526\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd" Oct 30 09:34:32.169936 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:32.169856 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fca872d-4848-4886-925d-e06a5bcc2526-istio-data\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd\" (UID: \"2fca872d-4848-4886-925d-e06a5bcc2526\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd" Oct 30 09:34:32.170038 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:32.170014 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fca872d-4848-4886-925d-e06a5bcc2526-credential-socket\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd\" (UID: \"2fca872d-4848-4886-925d-e06a5bcc2526\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd" Oct 30 09:34:32.170038 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:32.170026 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fca872d-4848-4886-925d-e06a5bcc2526-workload-socket\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd\" (UID: \"2fca872d-4848-4886-925d-e06a5bcc2526\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd" Oct 30 09:34:32.170356 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:32.170331 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fca872d-4848-4886-925d-e06a5bcc2526-istiod-ca-cert\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd\" (UID: \"2fca872d-4848-4886-925d-e06a5bcc2526\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd" Oct 30 09:34:32.171836 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:32.171807 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fca872d-4848-4886-925d-e06a5bcc2526-istio-envoy\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd\" (UID: \"2fca872d-4848-4886-925d-e06a5bcc2526\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd" Oct 30 09:34:32.172372 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:32.172351 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fca872d-4848-4886-925d-e06a5bcc2526-istio-podinfo\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd\" (UID: \"2fca872d-4848-4886-925d-e06a5bcc2526\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd" Oct 30 09:34:32.188970 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:32.188945 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fca872d-4848-4886-925d-e06a5bcc2526-istio-token\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd\" (UID: \"2fca872d-4848-4886-925d-e06a5bcc2526\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd" Oct 30 09:34:32.189141 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:32.189122 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-882g7\" (UniqueName: \"kubernetes.io/projected/2fca872d-4848-4886-925d-e06a5bcc2526-kube-api-access-882g7\") pod \"e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd\" (UID: \"2fca872d-4848-4886-925d-e06a5bcc2526\") " pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd" Oct 30 09:34:32.299634 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:32.299549 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd" Oct 30 09:34:32.361186 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:32.355493 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-aa2fdf/auth-test-model-aa2fdf-kserve-699ff98945-zq96m"] Oct 30 09:34:32.363904 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:32.363880 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-aa2fdf/auth-test-model-aa2fdf-kserve-699ff98945-zq96m" Oct 30 09:34:32.383268 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:32.382286 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-aa2fdf/auth-test-model-aa2fdf-kserve-699ff98945-zq96m"] Oct 30 09:34:32.384579 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:32.383455 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-aa2fdf\"/\"auth-test-model-aa2fdf-kserve-self-signed-certs\"" Oct 30 09:34:32.384579 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:32.383581 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-aa2fdf\"/\"default-dockercfg-nxdfv\"" Oct 30 09:34:32.409503 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:32.409479 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-aa2fdf\"/\"openshift-service-ca.crt\"" Oct 30 09:34:32.409640 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:32.409531 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-aa2fdf\"/\"kube-root-ca.crt\"" Oct 30 09:34:32.484775 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:32.484742 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00d5c9cf-9dae-425f-8189-870602061926-dshm\") pod \"auth-test-model-aa2fdf-kserve-699ff98945-zq96m\" (UID: \"00d5c9cf-9dae-425f-8189-870602061926\") " pod="ai-tenant-e2e-shared-b-w1-aa2fdf/auth-test-model-aa2fdf-kserve-699ff98945-zq96m" Oct 30 09:34:32.484960 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:32.484787 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zvpf\" (UniqueName: \"kubernetes.io/projected/00d5c9cf-9dae-425f-8189-870602061926-kube-api-access-8zvpf\") pod \"auth-test-model-aa2fdf-kserve-699ff98945-zq96m\" (UID: \"00d5c9cf-9dae-425f-8189-870602061926\") " pod="ai-tenant-e2e-shared-b-w1-aa2fdf/auth-test-model-aa2fdf-kserve-699ff98945-zq96m" Oct 30 09:34:32.484960 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:32.484831 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00d5c9cf-9dae-425f-8189-870602061926-model-cache\") pod \"auth-test-model-aa2fdf-kserve-699ff98945-zq96m\" (UID: \"00d5c9cf-9dae-425f-8189-870602061926\") " pod="ai-tenant-e2e-shared-b-w1-aa2fdf/auth-test-model-aa2fdf-kserve-699ff98945-zq96m" Oct 30 09:34:32.485084 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:32.484989 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00d5c9cf-9dae-425f-8189-870602061926-tmp-dir\") pod \"auth-test-model-aa2fdf-kserve-699ff98945-zq96m\" (UID: \"00d5c9cf-9dae-425f-8189-870602061926\") " pod="ai-tenant-e2e-shared-b-w1-aa2fdf/auth-test-model-aa2fdf-kserve-699ff98945-zq96m" Oct 30 09:34:32.485084 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:32.485059 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00d5c9cf-9dae-425f-8189-870602061926-home\") pod \"auth-test-model-aa2fdf-kserve-699ff98945-zq96m\" (UID: \"00d5c9cf-9dae-425f-8189-870602061926\") " pod="ai-tenant-e2e-shared-b-w1-aa2fdf/auth-test-model-aa2fdf-kserve-699ff98945-zq96m" Oct 30 09:34:32.485173 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:32.485133 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00d5c9cf-9dae-425f-8189-870602061926-tls-certs\") pod \"auth-test-model-aa2fdf-kserve-699ff98945-zq96m\" (UID: \"00d5c9cf-9dae-425f-8189-870602061926\") " pod="ai-tenant-e2e-shared-b-w1-aa2fdf/auth-test-model-aa2fdf-kserve-699ff98945-zq96m" Oct 30 09:34:32.599458 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:32.595591 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00d5c9cf-9dae-425f-8189-870602061926-dshm\") pod \"auth-test-model-aa2fdf-kserve-699ff98945-zq96m\" (UID: \"00d5c9cf-9dae-425f-8189-870602061926\") " pod="ai-tenant-e2e-shared-b-w1-aa2fdf/auth-test-model-aa2fdf-kserve-699ff98945-zq96m" Oct 30 09:34:32.599458 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:32.595662 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zvpf\" (UniqueName: \"kubernetes.io/projected/00d5c9cf-9dae-425f-8189-870602061926-kube-api-access-8zvpf\") pod \"auth-test-model-aa2fdf-kserve-699ff98945-zq96m\" (UID: \"00d5c9cf-9dae-425f-8189-870602061926\") " pod="ai-tenant-e2e-shared-b-w1-aa2fdf/auth-test-model-aa2fdf-kserve-699ff98945-zq96m" Oct 30 09:34:32.599458 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:32.595731 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00d5c9cf-9dae-425f-8189-870602061926-model-cache\") pod \"auth-test-model-aa2fdf-kserve-699ff98945-zq96m\" (UID: \"00d5c9cf-9dae-425f-8189-870602061926\") " pod="ai-tenant-e2e-shared-b-w1-aa2fdf/auth-test-model-aa2fdf-kserve-699ff98945-zq96m" Oct 30 09:34:32.599458 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:32.595823 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00d5c9cf-9dae-425f-8189-870602061926-tmp-dir\") pod \"auth-test-model-aa2fdf-kserve-699ff98945-zq96m\" (UID: \"00d5c9cf-9dae-425f-8189-870602061926\") " pod="ai-tenant-e2e-shared-b-w1-aa2fdf/auth-test-model-aa2fdf-kserve-699ff98945-zq96m" Oct 30 09:34:32.599458 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:32.595880 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00d5c9cf-9dae-425f-8189-870602061926-home\") pod \"auth-test-model-aa2fdf-kserve-699ff98945-zq96m\" (UID: \"00d5c9cf-9dae-425f-8189-870602061926\") " pod="ai-tenant-e2e-shared-b-w1-aa2fdf/auth-test-model-aa2fdf-kserve-699ff98945-zq96m" Oct 30 09:34:32.599458 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:32.595954 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00d5c9cf-9dae-425f-8189-870602061926-tls-certs\") pod \"auth-test-model-aa2fdf-kserve-699ff98945-zq96m\" (UID: \"00d5c9cf-9dae-425f-8189-870602061926\") " pod="ai-tenant-e2e-shared-b-w1-aa2fdf/auth-test-model-aa2fdf-kserve-699ff98945-zq96m" Oct 30 09:34:32.599458 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:32.596895 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00d5c9cf-9dae-425f-8189-870602061926-model-cache\") pod \"auth-test-model-aa2fdf-kserve-699ff98945-zq96m\" (UID: \"00d5c9cf-9dae-425f-8189-870602061926\") " pod="ai-tenant-e2e-shared-b-w1-aa2fdf/auth-test-model-aa2fdf-kserve-699ff98945-zq96m" Oct 30 09:34:32.599458 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:32.597892 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00d5c9cf-9dae-425f-8189-870602061926-tmp-dir\") pod \"auth-test-model-aa2fdf-kserve-699ff98945-zq96m\" (UID: \"00d5c9cf-9dae-425f-8189-870602061926\") " pod="ai-tenant-e2e-shared-b-w1-aa2fdf/auth-test-model-aa2fdf-kserve-699ff98945-zq96m" Oct 30 09:34:32.599458 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:32.598168 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00d5c9cf-9dae-425f-8189-870602061926-home\") pod \"auth-test-model-aa2fdf-kserve-699ff98945-zq96m\" (UID: \"00d5c9cf-9dae-425f-8189-870602061926\") " pod="ai-tenant-e2e-shared-b-w1-aa2fdf/auth-test-model-aa2fdf-kserve-699ff98945-zq96m" Oct 30 09:34:32.617450 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:32.614833 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00d5c9cf-9dae-425f-8189-870602061926-dshm\") pod \"auth-test-model-aa2fdf-kserve-699ff98945-zq96m\" (UID: \"00d5c9cf-9dae-425f-8189-870602061926\") " pod="ai-tenant-e2e-shared-b-w1-aa2fdf/auth-test-model-aa2fdf-kserve-699ff98945-zq96m" Oct 30 09:34:32.643769 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:32.641761 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00d5c9cf-9dae-425f-8189-870602061926-tls-certs\") pod \"auth-test-model-aa2fdf-kserve-699ff98945-zq96m\" (UID: \"00d5c9cf-9dae-425f-8189-870602061926\") " pod="ai-tenant-e2e-shared-b-w1-aa2fdf/auth-test-model-aa2fdf-kserve-699ff98945-zq96m" Oct 30 09:34:32.652974 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:32.652307 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd"] Oct 30 09:34:32.671747 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:32.671620 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zvpf\" (UniqueName: \"kubernetes.io/projected/00d5c9cf-9dae-425f-8189-870602061926-kube-api-access-8zvpf\") pod \"auth-test-model-aa2fdf-kserve-699ff98945-zq96m\" (UID: \"00d5c9cf-9dae-425f-8189-870602061926\") " pod="ai-tenant-e2e-shared-b-w1-aa2fdf/auth-test-model-aa2fdf-kserve-699ff98945-zq96m" Oct 30 09:34:32.689993 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:32.689892 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-aa2fdf/auth-test-model-aa2fdf-kserve-699ff98945-zq96m" Oct 30 09:34:32.692398 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:32.692345 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:34:32.692660 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:32.692635 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:34:32.692819 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:32.692802 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:34:33.030576 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:34:33.030543 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod00d5c9cf_9dae_425f_8189_870602061926.slice/crio-c93c0918a0b3df1f52cfeb7fe56c18b72596ac22380472bd31b76fe04749648f WatchSource:0}: Error finding container c93c0918a0b3df1f52cfeb7fe56c18b72596ac22380472bd31b76fe04749648f: Status 404 returned error can't find the container with id c93c0918a0b3df1f52cfeb7fe56c18b72596ac22380472bd31b76fe04749648f Oct 30 09:34:33.031649 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:33.031623 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-aa2fdf/auth-test-model-aa2fdf-kserve-699ff98945-zq96m"] Oct 30 09:34:33.681028 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:33.680987 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd" event={"ID":"2fca872d-4848-4886-925d-e06a5bcc2526","Type":"ContainerStarted","Data":"0d8ed9c24ec64c9580dd8e6a22b23007907dc8e934b7fdfd476d2e8ec0ffc701"} Oct 30 09:34:33.681290 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:33.681270 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd" event={"ID":"2fca872d-4848-4886-925d-e06a5bcc2526","Type":"ContainerStarted","Data":"edaed962556cca9ef72cb59367ec9ad421426f62b31af1d95f1e386482ae2fe9"} Oct 30 09:34:33.685280 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:33.685250 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-aa2fdf/auth-test-model-aa2fdf-kserve-699ff98945-zq96m" event={"ID":"00d5c9cf-9dae-425f-8189-870602061926","Type":"ContainerStarted","Data":"d80fd8b11c285fd2e2dd627d81427dc3443373a63b90b31cdca5ae043c917ff5"} Oct 30 09:34:33.685663 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:33.685638 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-aa2fdf/auth-test-model-aa2fdf-kserve-699ff98945-zq96m" event={"ID":"00d5c9cf-9dae-425f-8189-870602061926","Type":"ContainerStarted","Data":"c93c0918a0b3df1f52cfeb7fe56c18b72596ac22380472bd31b76fe04749648f"} Oct 30 09:34:33.691513 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:33.691482 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-aa2fdf/auth-test-model-aa2fdf-kserve-699ff98945-zq96m" Oct 30 09:34:33.734594 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:33.734536 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd" podStartSLOduration=2.734522531 podStartE2EDuration="2.734522531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:34:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:34:33.728947731 +0000 UTC m=+2433.990508722" watchObservedRunningTime="2026-10-30 09:34:33.734522531 +0000 UTC m=+2433.996083521" Oct 30 09:34:33.781311 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:33.781240 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-aa2fdf/auth-test-model-aa2fdf-kserve-699ff98945-zq96m" podStartSLOduration=1.781220885 podStartE2EDuration="1.781220885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:34:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:34:33.776527121 +0000 UTC m=+2434.038088109" watchObservedRunningTime="2026-10-30 09:34:33.781220885 +0000 UTC m=+2434.042781874" Oct 30 09:34:34.300521 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:34.300488 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd" Oct 30 09:34:34.302485 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:34.302459 4562 patch_prober.go:29] interesting pod/e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Oct 30 09:34:34.302625 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:34.302510 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd" podUID="2fca872d-4848-4886-925d-e06a5bcc2526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Oct 30 09:34:34.581092 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:34.581013 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-1413c8-6566657db8-lcr4t" Oct 30 09:34:35.302545 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:35.302503 4562 patch_prober.go:29] interesting pod/e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Oct 30 09:34:35.303160 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:35.303122 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd" podUID="2fca872d-4848-4886-925d-e06a5bcc2526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Oct 30 09:34:36.300206 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:36.300167 4562 patch_prober.go:29] interesting pod/e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Oct 30 09:34:36.300421 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:36.300224 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd" podUID="2fca872d-4848-4886-925d-e06a5bcc2526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Oct 30 09:34:37.300495 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:37.300455 4562 patch_prober.go:29] interesting pod/e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Oct 30 09:34:37.300939 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:37.300512 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd" podUID="2fca872d-4848-4886-925d-e06a5bcc2526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Oct 30 09:34:38.300213 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:38.300175 4562 patch_prober.go:29] interesting pod/e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Oct 30 09:34:38.300428 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:38.300243 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd" podUID="2fca872d-4848-4886-925d-e06a5bcc2526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Oct 30 09:34:39.304792 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:39.304759 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd" Oct 30 09:34:39.305290 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:39.305170 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd" Oct 30 09:34:39.305864 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:39.305833 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd" Oct 30 09:34:42.690928 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:42.690890 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-aa2fdf/auth-test-model-aa2fdf-kserve-699ff98945-zq96m" Oct 30 09:34:42.703721 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:42.703698 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-aa2fdf/auth-test-model-aa2fdf-kserve-699ff98945-zq96m" Oct 30 09:34:42.786659 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:34:42.786634 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-aa2fdf/auth-test-model-aa2fdf-kserve-699ff98945-zq96m" Oct 30 09:34:58.596762 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:34:58.596724 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:35:00.585101 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:00.585068 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-1413c8/explicit-model-1413c8-kserve-745ffb9459-9vvts"] Oct 30 09:35:00.590336 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:00.590299 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-1413c8/explicit-model-1413c8-kserve-745ffb9459-9vvts" Oct 30 09:35:00.609581 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:00.609547 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-1413c8/explicit-model-1413c8-kserve-745ffb9459-9vvts"] Oct 30 09:35:00.616923 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:00.616885 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-1413c8\"/\"kube-root-ca.crt\"" Oct 30 09:35:00.617092 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:00.617065 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-1413c8\"/\"explicit-model-1413c8-kserve-self-signed-certs\"" Oct 30 09:35:00.619207 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:00.619171 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-1413c8\"/\"openshift-service-ca.crt\"" Oct 30 09:35:00.619337 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:00.619253 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-1413c8\"/\"default-dockercfg-cfdxv\"" Oct 30 09:35:00.655055 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:00.655024 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7350869e-8e0b-41cd-b0fc-97686ccd12e4-dshm\") pod \"explicit-model-1413c8-kserve-745ffb9459-9vvts\" (UID: \"7350869e-8e0b-41cd-b0fc-97686ccd12e4\") " pod="ai-tenant-e2e-explicit-1413c8/explicit-model-1413c8-kserve-745ffb9459-9vvts" Oct 30 09:35:00.655180 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:00.655085 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7350869e-8e0b-41cd-b0fc-97686ccd12e4-home\") pod \"explicit-model-1413c8-kserve-745ffb9459-9vvts\" (UID: \"7350869e-8e0b-41cd-b0fc-97686ccd12e4\") " pod="ai-tenant-e2e-explicit-1413c8/explicit-model-1413c8-kserve-745ffb9459-9vvts" Oct 30 09:35:00.655180 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:00.655105 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7350869e-8e0b-41cd-b0fc-97686ccd12e4-tmp-dir\") pod \"explicit-model-1413c8-kserve-745ffb9459-9vvts\" (UID: \"7350869e-8e0b-41cd-b0fc-97686ccd12e4\") " pod="ai-tenant-e2e-explicit-1413c8/explicit-model-1413c8-kserve-745ffb9459-9vvts" Oct 30 09:35:00.655298 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:00.655273 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7350869e-8e0b-41cd-b0fc-97686ccd12e4-tls-certs\") pod \"explicit-model-1413c8-kserve-745ffb9459-9vvts\" (UID: \"7350869e-8e0b-41cd-b0fc-97686ccd12e4\") " pod="ai-tenant-e2e-explicit-1413c8/explicit-model-1413c8-kserve-745ffb9459-9vvts" Oct 30 09:35:00.655436 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:00.655422 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqmbs\" (UniqueName: \"kubernetes.io/projected/7350869e-8e0b-41cd-b0fc-97686ccd12e4-kube-api-access-vqmbs\") pod \"explicit-model-1413c8-kserve-745ffb9459-9vvts\" (UID: \"7350869e-8e0b-41cd-b0fc-97686ccd12e4\") " pod="ai-tenant-e2e-explicit-1413c8/explicit-model-1413c8-kserve-745ffb9459-9vvts" Oct 30 09:35:00.655541 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:00.655529 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7350869e-8e0b-41cd-b0fc-97686ccd12e4-model-cache\") pod \"explicit-model-1413c8-kserve-745ffb9459-9vvts\" (UID: \"7350869e-8e0b-41cd-b0fc-97686ccd12e4\") " pod="ai-tenant-e2e-explicit-1413c8/explicit-model-1413c8-kserve-745ffb9459-9vvts" Oct 30 09:35:00.756643 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:00.756604 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7350869e-8e0b-41cd-b0fc-97686ccd12e4-dshm\") pod \"explicit-model-1413c8-kserve-745ffb9459-9vvts\" (UID: \"7350869e-8e0b-41cd-b0fc-97686ccd12e4\") " pod="ai-tenant-e2e-explicit-1413c8/explicit-model-1413c8-kserve-745ffb9459-9vvts" Oct 30 09:35:00.756824 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:00.756661 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7350869e-8e0b-41cd-b0fc-97686ccd12e4-home\") pod \"explicit-model-1413c8-kserve-745ffb9459-9vvts\" (UID: \"7350869e-8e0b-41cd-b0fc-97686ccd12e4\") " pod="ai-tenant-e2e-explicit-1413c8/explicit-model-1413c8-kserve-745ffb9459-9vvts" Oct 30 09:35:00.756824 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:00.756689 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7350869e-8e0b-41cd-b0fc-97686ccd12e4-tmp-dir\") pod \"explicit-model-1413c8-kserve-745ffb9459-9vvts\" (UID: \"7350869e-8e0b-41cd-b0fc-97686ccd12e4\") " pod="ai-tenant-e2e-explicit-1413c8/explicit-model-1413c8-kserve-745ffb9459-9vvts" Oct 30 09:35:00.756824 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:00.756731 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7350869e-8e0b-41cd-b0fc-97686ccd12e4-tls-certs\") pod \"explicit-model-1413c8-kserve-745ffb9459-9vvts\" (UID: \"7350869e-8e0b-41cd-b0fc-97686ccd12e4\") " pod="ai-tenant-e2e-explicit-1413c8/explicit-model-1413c8-kserve-745ffb9459-9vvts" Oct 30 09:35:00.756972 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:00.756828 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqmbs\" (UniqueName: \"kubernetes.io/projected/7350869e-8e0b-41cd-b0fc-97686ccd12e4-kube-api-access-vqmbs\") pod \"explicit-model-1413c8-kserve-745ffb9459-9vvts\" (UID: \"7350869e-8e0b-41cd-b0fc-97686ccd12e4\") " pod="ai-tenant-e2e-explicit-1413c8/explicit-model-1413c8-kserve-745ffb9459-9vvts" Oct 30 09:35:00.756972 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:00.756957 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7350869e-8e0b-41cd-b0fc-97686ccd12e4-model-cache\") pod \"explicit-model-1413c8-kserve-745ffb9459-9vvts\" (UID: \"7350869e-8e0b-41cd-b0fc-97686ccd12e4\") " pod="ai-tenant-e2e-explicit-1413c8/explicit-model-1413c8-kserve-745ffb9459-9vvts" Oct 30 09:35:00.757181 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:00.757155 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7350869e-8e0b-41cd-b0fc-97686ccd12e4-tmp-dir\") pod \"explicit-model-1413c8-kserve-745ffb9459-9vvts\" (UID: \"7350869e-8e0b-41cd-b0fc-97686ccd12e4\") " pod="ai-tenant-e2e-explicit-1413c8/explicit-model-1413c8-kserve-745ffb9459-9vvts" Oct 30 09:35:00.757284 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:00.757203 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7350869e-8e0b-41cd-b0fc-97686ccd12e4-home\") pod \"explicit-model-1413c8-kserve-745ffb9459-9vvts\" (UID: \"7350869e-8e0b-41cd-b0fc-97686ccd12e4\") " pod="ai-tenant-e2e-explicit-1413c8/explicit-model-1413c8-kserve-745ffb9459-9vvts" Oct 30 09:35:00.757341 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:00.757281 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7350869e-8e0b-41cd-b0fc-97686ccd12e4-model-cache\") pod \"explicit-model-1413c8-kserve-745ffb9459-9vvts\" (UID: \"7350869e-8e0b-41cd-b0fc-97686ccd12e4\") " pod="ai-tenant-e2e-explicit-1413c8/explicit-model-1413c8-kserve-745ffb9459-9vvts" Oct 30 09:35:00.759040 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:00.759018 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7350869e-8e0b-41cd-b0fc-97686ccd12e4-dshm\") pod \"explicit-model-1413c8-kserve-745ffb9459-9vvts\" (UID: \"7350869e-8e0b-41cd-b0fc-97686ccd12e4\") " pod="ai-tenant-e2e-explicit-1413c8/explicit-model-1413c8-kserve-745ffb9459-9vvts" Oct 30 09:35:00.759511 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:00.759488 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7350869e-8e0b-41cd-b0fc-97686ccd12e4-tls-certs\") pod \"explicit-model-1413c8-kserve-745ffb9459-9vvts\" (UID: \"7350869e-8e0b-41cd-b0fc-97686ccd12e4\") " pod="ai-tenant-e2e-explicit-1413c8/explicit-model-1413c8-kserve-745ffb9459-9vvts" Oct 30 09:35:00.788843 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:00.781706 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqmbs\" (UniqueName: \"kubernetes.io/projected/7350869e-8e0b-41cd-b0fc-97686ccd12e4-kube-api-access-vqmbs\") pod \"explicit-model-1413c8-kserve-745ffb9459-9vvts\" (UID: \"7350869e-8e0b-41cd-b0fc-97686ccd12e4\") " pod="ai-tenant-e2e-explicit-1413c8/explicit-model-1413c8-kserve-745ffb9459-9vvts" Oct 30 09:35:00.919493 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:00.919399 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-1413c8/explicit-model-1413c8-kserve-745ffb9459-9vvts" Oct 30 09:35:01.253241 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:35:01.253204 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7350869e_8e0b_41cd_b0fc_97686ccd12e4.slice/crio-1b929c18f8f1155dae8db954c99dfc270a235e846afe125d04144d14db506ce9 WatchSource:0}: Error finding container 1b929c18f8f1155dae8db954c99dfc270a235e846afe125d04144d14db506ce9: Status 404 returned error can't find the container with id 1b929c18f8f1155dae8db954c99dfc270a235e846afe125d04144d14db506ce9 Oct 30 09:35:01.255068 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:01.255031 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-1413c8/explicit-model-1413c8-kserve-745ffb9459-9vvts"] Oct 30 09:35:01.973463 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:01.973415 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-1413c8/explicit-model-1413c8-kserve-745ffb9459-9vvts" event={"ID":"7350869e-8e0b-41cd-b0fc-97686ccd12e4","Type":"ContainerStarted","Data":"cca417ec08dacd4b54fb900f373667a28d429c7102815692b5d25ae352fdcc1b"} Oct 30 09:35:01.973954 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:01.973469 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-1413c8/explicit-model-1413c8-kserve-745ffb9459-9vvts" event={"ID":"7350869e-8e0b-41cd-b0fc-97686ccd12e4","Type":"ContainerStarted","Data":"1b929c18f8f1155dae8db954c99dfc270a235e846afe125d04144d14db506ce9"} Oct 30 09:35:02.016032 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:02.015985 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-1413c8/explicit-model-1413c8-kserve-745ffb9459-9vvts" podStartSLOduration=2.015971149 podStartE2EDuration="2.015971149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:35:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:35:02.013494395 +0000 UTC m=+2462.275055383" watchObservedRunningTime="2026-10-30 09:35:02.015971149 +0000 UTC m=+2462.277532178" Oct 30 09:35:02.920487 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:02.920447 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-1413c8/explicit-model-1413c8-kserve-745ffb9459-9vvts" Oct 30 09:35:10.919644 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:10.919610 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-1413c8/explicit-model-1413c8-kserve-745ffb9459-9vvts" Oct 30 09:35:10.933763 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:10.933733 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-1413c8/explicit-model-1413c8-kserve-745ffb9459-9vvts" Oct 30 09:35:11.053359 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:11.053321 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-1413c8/explicit-model-1413c8-kserve-745ffb9459-9vvts" Oct 30 09:35:15.027456 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:15.027357 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c421e892-9pkx9"] Oct 30 09:35:15.041576 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:15.041536 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c421e892-9pkx9"] Oct 30 09:35:15.876314 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:15.876283 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecab733b-751b-4448-a556-4a70875db2e5" path="/var/lib/kubelet/pods/ecab733b-751b-4448-a556-4a70875db2e5/volumes" Oct 30 09:35:22.961943 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:22.961911 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-1413c8/explicit-model-1413c8-kserve-745ffb9459-9vvts"] Oct 30 09:35:28.597398 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:35:28.597350 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:35:33.796176 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:33.796137 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-1413c8-6d66d75c58-ncqbq"] Oct 30 09:35:33.828705 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:33.828671 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-1413c8-76f699c45f-rv8gt"] Oct 30 09:35:37.985135 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:37.985098 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-1413c8/explicit-model-1413c8-kserve-745ffb9459-9vvts" podUID="7350869e-8e0b-41cd-b0fc-97686ccd12e4" containerName="main" containerID="cri-o://cca417ec08dacd4b54fb900f373667a28d429c7102815692b5d25ae352fdcc1b" gracePeriod=45 Oct 30 09:35:38.041947 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:38.041911 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-explicit-1413c8/explicit-model-1413c8-kserve-745ffb9459-9vvts" podUID="7350869e-8e0b-41cd-b0fc-97686ccd12e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.191:8000/ready\": dial tcp 10.133.0.191:8000: connect: connection refused" Oct 30 09:35:38.258903 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:38.258828 4562 generic.go:356] "Generic (PLEG): container finished" podID="7350869e-8e0b-41cd-b0fc-97686ccd12e4" containerID="cca417ec08dacd4b54fb900f373667a28d429c7102815692b5d25ae352fdcc1b" exitCode=0 Oct 30 09:35:38.259069 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:38.258897 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-1413c8/explicit-model-1413c8-kserve-745ffb9459-9vvts" event={"ID":"7350869e-8e0b-41cd-b0fc-97686ccd12e4","Type":"ContainerDied","Data":"cca417ec08dacd4b54fb900f373667a28d429c7102815692b5d25ae352fdcc1b"} Oct 30 09:35:38.302500 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:38.302475 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-1413c8/explicit-model-1413c8-kserve-745ffb9459-9vvts" Oct 30 09:35:38.397692 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:38.397653 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7350869e-8e0b-41cd-b0fc-97686ccd12e4-tls-certs\" (UniqueName: \"kubernetes.io/secret/7350869e-8e0b-41cd-b0fc-97686ccd12e4-tls-certs\") pod \"7350869e-8e0b-41cd-b0fc-97686ccd12e4\" (UID: \"7350869e-8e0b-41cd-b0fc-97686ccd12e4\") " Oct 30 09:35:38.397890 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:38.397765 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7350869e-8e0b-41cd-b0fc-97686ccd12e4-home\" (UniqueName: \"kubernetes.io/empty-dir/7350869e-8e0b-41cd-b0fc-97686ccd12e4-home\") pod \"7350869e-8e0b-41cd-b0fc-97686ccd12e4\" (UID: \"7350869e-8e0b-41cd-b0fc-97686ccd12e4\") " Oct 30 09:35:38.397890 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:38.397825 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7350869e-8e0b-41cd-b0fc-97686ccd12e4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7350869e-8e0b-41cd-b0fc-97686ccd12e4-tmp-dir\") pod \"7350869e-8e0b-41cd-b0fc-97686ccd12e4\" (UID: \"7350869e-8e0b-41cd-b0fc-97686ccd12e4\") " Oct 30 09:35:38.398014 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:38.397891 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7350869e-8e0b-41cd-b0fc-97686ccd12e4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7350869e-8e0b-41cd-b0fc-97686ccd12e4-model-cache\") pod \"7350869e-8e0b-41cd-b0fc-97686ccd12e4\" (UID: \"7350869e-8e0b-41cd-b0fc-97686ccd12e4\") " Oct 30 09:35:38.398014 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:38.397948 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7350869e-8e0b-41cd-b0fc-97686ccd12e4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7350869e-8e0b-41cd-b0fc-97686ccd12e4-dshm\") pod \"7350869e-8e0b-41cd-b0fc-97686ccd12e4\" (UID: \"7350869e-8e0b-41cd-b0fc-97686ccd12e4\") " Oct 30 09:35:38.398114 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:38.398090 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7350869e-8e0b-41cd-b0fc-97686ccd12e4-home" pod "7350869e-8e0b-41cd-b0fc-97686ccd12e4" (UID: "7350869e-8e0b-41cd-b0fc-97686ccd12e4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:35:38.398171 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:38.398126 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7350869e-8e0b-41cd-b0fc-97686ccd12e4-kube-api-access-vqmbs\" (UniqueName: \"kubernetes.io/projected/7350869e-8e0b-41cd-b0fc-97686ccd12e4-kube-api-access-vqmbs\") pod \"7350869e-8e0b-41cd-b0fc-97686ccd12e4\" (UID: \"7350869e-8e0b-41cd-b0fc-97686ccd12e4\") " Oct 30 09:35:38.398171 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:38.398135 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7350869e-8e0b-41cd-b0fc-97686ccd12e4-tmp-dir" pod "7350869e-8e0b-41cd-b0fc-97686ccd12e4" (UID: "7350869e-8e0b-41cd-b0fc-97686ccd12e4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:35:38.398276 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:38.398230 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7350869e-8e0b-41cd-b0fc-97686ccd12e4-model-cache" pod "7350869e-8e0b-41cd-b0fc-97686ccd12e4" (UID: "7350869e-8e0b-41cd-b0fc-97686ccd12e4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:35:38.398842 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:38.398818 4562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7350869e-8e0b-41cd-b0fc-97686ccd12e4-home\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:35:38.399060 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:38.399034 4562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7350869e-8e0b-41cd-b0fc-97686ccd12e4-tmp-dir\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:35:38.399060 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:38.399063 4562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7350869e-8e0b-41cd-b0fc-97686ccd12e4-model-cache\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:35:38.400477 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:38.400446 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7350869e-8e0b-41cd-b0fc-97686ccd12e4-dshm" pod "7350869e-8e0b-41cd-b0fc-97686ccd12e4" (UID: "7350869e-8e0b-41cd-b0fc-97686ccd12e4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:35:38.400477 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:38.400464 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7350869e-8e0b-41cd-b0fc-97686ccd12e4-tls-certs" pod "7350869e-8e0b-41cd-b0fc-97686ccd12e4" (UID: "7350869e-8e0b-41cd-b0fc-97686ccd12e4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:35:38.400740 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:38.400716 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7350869e-8e0b-41cd-b0fc-97686ccd12e4-kube-api-access-vqmbs" pod "7350869e-8e0b-41cd-b0fc-97686ccd12e4" (UID: "7350869e-8e0b-41cd-b0fc-97686ccd12e4"). InnerVolumeSpecName "kube-api-access-vqmbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:35:38.500979 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:38.500946 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqmbs\" (UniqueName: \"kubernetes.io/projected/7350869e-8e0b-41cd-b0fc-97686ccd12e4-kube-api-access-vqmbs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:35:38.500979 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:38.500978 4562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7350869e-8e0b-41cd-b0fc-97686ccd12e4-tls-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:35:38.500979 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:38.500989 4562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7350869e-8e0b-41cd-b0fc-97686ccd12e4-dshm\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:35:38.796760 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:38.796697 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-1413c8-6d66d75c58-ncqbq" podUID="4084b690-89cf-4f47-911e-a49aa5368653" containerName="payload-processing" containerID="cri-o://1c4753e6d584c6f971a048d687cab572d054a22aa0b189ea2f9962bf4c706193" gracePeriod=25 Oct 30 09:35:38.829019 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:38.828968 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-1413c8-76f699c45f-rv8gt" podUID="23788da7-9c3c-45aa-b327-5f3f05c35cb6" containerName="payload-pre-processing" containerID="cri-o://5fe063f4ebaa9701fa8eef5294797dece6f568b69bcf8d70e80c1075b02c6655" gracePeriod=25 Oct 30 09:35:39.169971 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:39.169941 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-1413c8-6d66d75c58-ncqbq" Oct 30 09:35:39.203584 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:39.203549 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-1413c8-76f699c45f-rv8gt" Oct 30 09:35:39.272824 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:39.272779 4562 generic.go:356] "Generic (PLEG): container finished" podID="4084b690-89cf-4f47-911e-a49aa5368653" containerID="1c4753e6d584c6f971a048d687cab572d054a22aa0b189ea2f9962bf4c706193" exitCode=0 Oct 30 09:35:39.273078 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:39.272835 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-1413c8-6d66d75c58-ncqbq" Oct 30 09:35:39.273524 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:39.272855 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-1413c8-6d66d75c58-ncqbq" event={"ID":"4084b690-89cf-4f47-911e-a49aa5368653","Type":"ContainerDied","Data":"1c4753e6d584c6f971a048d687cab572d054a22aa0b189ea2f9962bf4c706193"} Oct 30 09:35:39.273621 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:39.273602 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-1413c8-6d66d75c58-ncqbq" event={"ID":"4084b690-89cf-4f47-911e-a49aa5368653","Type":"ContainerDied","Data":"a14dda18406b2d62ccdc0894efc1328fe050a65d3fab5165bc66364f088f6af0"} Oct 30 09:35:39.273693 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:39.273640 4562 scope.go:122] "RemoveContainer" containerID="1c4753e6d584c6f971a048d687cab572d054a22aa0b189ea2f9962bf4c706193" Oct 30 09:35:39.275901 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:39.275867 4562 generic.go:356] "Generic (PLEG): container finished" podID="23788da7-9c3c-45aa-b327-5f3f05c35cb6" containerID="5fe063f4ebaa9701fa8eef5294797dece6f568b69bcf8d70e80c1075b02c6655" exitCode=0 Oct 30 09:35:39.281429 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:39.276009 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-1413c8-76f699c45f-rv8gt" event={"ID":"23788da7-9c3c-45aa-b327-5f3f05c35cb6","Type":"ContainerDied","Data":"5fe063f4ebaa9701fa8eef5294797dece6f568b69bcf8d70e80c1075b02c6655"} Oct 30 09:35:39.281429 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:39.276062 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-1413c8-76f699c45f-rv8gt" event={"ID":"23788da7-9c3c-45aa-b327-5f3f05c35cb6","Type":"ContainerDied","Data":"af5e2ff2cce2578848e89db3a929653a9245a2770a839c4828b86200f4c3a424"} Oct 30 09:35:39.281429 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:39.276171 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-1413c8-76f699c45f-rv8gt" Oct 30 09:35:39.281429 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:39.277928 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-1413c8/explicit-model-1413c8-kserve-745ffb9459-9vvts" event={"ID":"7350869e-8e0b-41cd-b0fc-97686ccd12e4","Type":"ContainerDied","Data":"1b929c18f8f1155dae8db954c99dfc270a235e846afe125d04144d14db506ce9"} Oct 30 09:35:39.281429 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:39.278015 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-1413c8/explicit-model-1413c8-kserve-745ffb9459-9vvts" Oct 30 09:35:39.294948 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:39.294907 4562 scope.go:122] "RemoveContainer" containerID="1c4753e6d584c6f971a048d687cab572d054a22aa0b189ea2f9962bf4c706193" Oct 30 09:35:39.295557 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:35:39.295531 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c4753e6d584c6f971a048d687cab572d054a22aa0b189ea2f9962bf4c706193\": container with ID starting with 1c4753e6d584c6f971a048d687cab572d054a22aa0b189ea2f9962bf4c706193 not found: ID does not exist" containerID="1c4753e6d584c6f971a048d687cab572d054a22aa0b189ea2f9962bf4c706193" Oct 30 09:35:39.295668 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:39.295566 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c4753e6d584c6f971a048d687cab572d054a22aa0b189ea2f9962bf4c706193"} err="failed to get container status \"1c4753e6d584c6f971a048d687cab572d054a22aa0b189ea2f9962bf4c706193\": rpc error: code = NotFound desc = could not find container \"1c4753e6d584c6f971a048d687cab572d054a22aa0b189ea2f9962bf4c706193\": container with ID starting with 1c4753e6d584c6f971a048d687cab572d054a22aa0b189ea2f9962bf4c706193 not found: ID does not exist" Oct 30 09:35:39.295668 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:39.295590 4562 scope.go:122] "RemoveContainer" containerID="5fe063f4ebaa9701fa8eef5294797dece6f568b69bcf8d70e80c1075b02c6655" Oct 30 09:35:39.311045 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:39.311013 4562 scope.go:122] "RemoveContainer" containerID="5fe063f4ebaa9701fa8eef5294797dece6f568b69bcf8d70e80c1075b02c6655" Oct 30 09:35:39.311463 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:35:39.311439 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fe063f4ebaa9701fa8eef5294797dece6f568b69bcf8d70e80c1075b02c6655\": container with ID starting with 5fe063f4ebaa9701fa8eef5294797dece6f568b69bcf8d70e80c1075b02c6655 not found: ID does not exist" containerID="5fe063f4ebaa9701fa8eef5294797dece6f568b69bcf8d70e80c1075b02c6655" Oct 30 09:35:39.311591 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:39.311467 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fe063f4ebaa9701fa8eef5294797dece6f568b69bcf8d70e80c1075b02c6655"} err="failed to get container status \"5fe063f4ebaa9701fa8eef5294797dece6f568b69bcf8d70e80c1075b02c6655\": rpc error: code = NotFound desc = could not find container \"5fe063f4ebaa9701fa8eef5294797dece6f568b69bcf8d70e80c1075b02c6655\": container with ID starting with 5fe063f4ebaa9701fa8eef5294797dece6f568b69bcf8d70e80c1075b02c6655 not found: ID does not exist" Oct 30 09:35:39.311591 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:39.311484 4562 scope.go:122] "RemoveContainer" containerID="cca417ec08dacd4b54fb900f373667a28d429c7102815692b5d25ae352fdcc1b" Oct 30 09:35:39.312031 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:39.312007 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23788da7-9c3c-45aa-b327-5f3f05c35cb6-kube-api-access-4p8dx\" (UniqueName: \"kubernetes.io/projected/23788da7-9c3c-45aa-b327-5f3f05c35cb6-kube-api-access-4p8dx\") pod \"23788da7-9c3c-45aa-b327-5f3f05c35cb6\" (UID: \"23788da7-9c3c-45aa-b327-5f3f05c35cb6\") " Oct 30 09:35:39.312126 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:39.312070 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4084b690-89cf-4f47-911e-a49aa5368653-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4084b690-89cf-4f47-911e-a49aa5368653-plugins-config-volume\") pod \"4084b690-89cf-4f47-911e-a49aa5368653\" (UID: \"4084b690-89cf-4f47-911e-a49aa5368653\") " Oct 30 09:35:39.312211 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:39.312197 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23788da7-9c3c-45aa-b327-5f3f05c35cb6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23788da7-9c3c-45aa-b327-5f3f05c35cb6-plugins-config-volume\") pod \"23788da7-9c3c-45aa-b327-5f3f05c35cb6\" (UID: \"23788da7-9c3c-45aa-b327-5f3f05c35cb6\") " Oct 30 09:35:39.312277 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:39.312250 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4084b690-89cf-4f47-911e-a49aa5368653-kube-api-access-r427g\" (UniqueName: \"kubernetes.io/projected/4084b690-89cf-4f47-911e-a49aa5368653-kube-api-access-r427g\") pod \"4084b690-89cf-4f47-911e-a49aa5368653\" (UID: \"4084b690-89cf-4f47-911e-a49aa5368653\") " Oct 30 09:35:39.313372 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:39.313344 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4084b690-89cf-4f47-911e-a49aa5368653-plugins-config-volume" pod "4084b690-89cf-4f47-911e-a49aa5368653" (UID: "4084b690-89cf-4f47-911e-a49aa5368653"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:35:39.313831 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:39.313810 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23788da7-9c3c-45aa-b327-5f3f05c35cb6-plugins-config-volume" pod "23788da7-9c3c-45aa-b327-5f3f05c35cb6" (UID: "23788da7-9c3c-45aa-b327-5f3f05c35cb6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:35:39.316180 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:39.316143 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4084b690-89cf-4f47-911e-a49aa5368653-kube-api-access-r427g" pod "4084b690-89cf-4f47-911e-a49aa5368653" (UID: "4084b690-89cf-4f47-911e-a49aa5368653"). InnerVolumeSpecName "kube-api-access-r427g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:35:39.317003 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:39.316971 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23788da7-9c3c-45aa-b327-5f3f05c35cb6-kube-api-access-4p8dx" pod "23788da7-9c3c-45aa-b327-5f3f05c35cb6" (UID: "23788da7-9c3c-45aa-b327-5f3f05c35cb6"). InnerVolumeSpecName "kube-api-access-4p8dx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:35:39.360405 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:39.360269 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-1413c8/explicit-model-1413c8-kserve-745ffb9459-9vvts"] Oct 30 09:35:39.367129 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:39.367095 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-1413c8/explicit-model-1413c8-kserve-745ffb9459-9vvts"] Oct 30 09:35:39.381929 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:39.381897 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-1413c8-6566657db8-lcr4t"] Oct 30 09:35:39.382134 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:39.382110 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-1413c8-6566657db8-lcr4t" podUID="a009b29f-625d-4074-9a24-55f2af08717d" containerName="maas-api" containerID="cri-o://2b9edfc3a8fac46e2d9cb71ff42bb7215f6ecf52aeb69d6281f93360b8ba3997" gracePeriod=30 Oct 30 09:35:39.415141 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:39.415097 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4p8dx\" (UniqueName: \"kubernetes.io/projected/23788da7-9c3c-45aa-b327-5f3f05c35cb6-kube-api-access-4p8dx\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:35:39.415141 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:39.415140 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4084b690-89cf-4f47-911e-a49aa5368653-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:35:39.415412 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:39.415159 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23788da7-9c3c-45aa-b327-5f3f05c35cb6-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:35:39.415412 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:39.415175 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r427g\" (UniqueName: \"kubernetes.io/projected/4084b690-89cf-4f47-911e-a49aa5368653-kube-api-access-r427g\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:35:39.642160 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:39.642081 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-1413c8-6d66d75c58-ncqbq"] Oct 30 09:35:39.645863 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:39.645834 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-1413c8-6d66d75c58-ncqbq"] Oct 30 09:35:39.659427 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:39.659350 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-1413c8-6566657db8-lcr4t" Oct 30 09:35:39.683453 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:39.683416 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-1413c8-76f699c45f-rv8gt"] Oct 30 09:35:39.688614 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:39.688584 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-1413c8-76f699c45f-rv8gt"] Oct 30 09:35:39.956499 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:39.949832 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a009b29f-625d-4074-9a24-55f2af08717d-kube-api-access-kbjt8\" (UniqueName: \"kubernetes.io/projected/a009b29f-625d-4074-9a24-55f2af08717d-kube-api-access-kbjt8\") pod \"a009b29f-625d-4074-9a24-55f2af08717d\" (UID: \"a009b29f-625d-4074-9a24-55f2af08717d\") " Oct 30 09:35:39.956499 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:39.949925 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a009b29f-625d-4074-9a24-55f2af08717d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a009b29f-625d-4074-9a24-55f2af08717d-metrics-certs\") pod \"a009b29f-625d-4074-9a24-55f2af08717d\" (UID: \"a009b29f-625d-4074-9a24-55f2af08717d\") " Oct 30 09:35:39.956499 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:39.950180 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a009b29f-625d-4074-9a24-55f2af08717d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a009b29f-625d-4074-9a24-55f2af08717d-maas-api-tls\") pod \"a009b29f-625d-4074-9a24-55f2af08717d\" (UID: \"a009b29f-625d-4074-9a24-55f2af08717d\") " Oct 30 09:35:39.956499 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:39.950307 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a009b29f-625d-4074-9a24-55f2af08717d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a009b29f-625d-4074-9a24-55f2af08717d-openshift-service-ca\") pod \"a009b29f-625d-4074-9a24-55f2af08717d\" (UID: \"a009b29f-625d-4074-9a24-55f2af08717d\") " Oct 30 09:35:39.956499 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:39.951106 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a009b29f-625d-4074-9a24-55f2af08717d-openshift-service-ca" pod "a009b29f-625d-4074-9a24-55f2af08717d" (UID: "a009b29f-625d-4074-9a24-55f2af08717d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:35:39.956499 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:39.954945 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a009b29f-625d-4074-9a24-55f2af08717d-kube-api-access-kbjt8" pod "a009b29f-625d-4074-9a24-55f2af08717d" (UID: "a009b29f-625d-4074-9a24-55f2af08717d"). InnerVolumeSpecName "kube-api-access-kbjt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:35:39.956499 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:39.956186 4562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a009b29f-625d-4074-9a24-55f2af08717d-openshift-service-ca\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:35:39.956499 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:39.956209 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbjt8\" (UniqueName: \"kubernetes.io/projected/a009b29f-625d-4074-9a24-55f2af08717d-kube-api-access-kbjt8\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:35:39.981402 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:39.978074 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23788da7-9c3c-45aa-b327-5f3f05c35cb6" path="/var/lib/kubelet/pods/23788da7-9c3c-45aa-b327-5f3f05c35cb6/volumes" Oct 30 09:35:39.981402 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:39.978963 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4084b690-89cf-4f47-911e-a49aa5368653" path="/var/lib/kubelet/pods/4084b690-89cf-4f47-911e-a49aa5368653/volumes" Oct 30 09:35:39.981402 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:39.979558 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7350869e-8e0b-41cd-b0fc-97686ccd12e4" path="/var/lib/kubelet/pods/7350869e-8e0b-41cd-b0fc-97686ccd12e4/volumes" Oct 30 09:35:39.983505 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:39.983455 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a009b29f-625d-4074-9a24-55f2af08717d-maas-api-tls" pod "a009b29f-625d-4074-9a24-55f2af08717d" (UID: "a009b29f-625d-4074-9a24-55f2af08717d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:35:39.993913 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:39.993860 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a009b29f-625d-4074-9a24-55f2af08717d-metrics-certs" pod "a009b29f-625d-4074-9a24-55f2af08717d" (UID: "a009b29f-625d-4074-9a24-55f2af08717d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:35:40.058095 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:40.058065 4562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a009b29f-625d-4074-9a24-55f2af08717d-maas-api-tls\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:35:40.058095 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:40.058092 4562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a009b29f-625d-4074-9a24-55f2af08717d-metrics-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:35:40.292344 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:40.292246 4562 generic.go:356] "Generic (PLEG): container finished" podID="a009b29f-625d-4074-9a24-55f2af08717d" containerID="2b9edfc3a8fac46e2d9cb71ff42bb7215f6ecf52aeb69d6281f93360b8ba3997" exitCode=0 Oct 30 09:35:40.292344 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:40.292305 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-1413c8-6566657db8-lcr4t" Oct 30 09:35:40.292886 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:40.292333 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-1413c8-6566657db8-lcr4t" event={"ID":"a009b29f-625d-4074-9a24-55f2af08717d","Type":"ContainerDied","Data":"2b9edfc3a8fac46e2d9cb71ff42bb7215f6ecf52aeb69d6281f93360b8ba3997"} Oct 30 09:35:40.292886 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:40.292468 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-1413c8-6566657db8-lcr4t" event={"ID":"a009b29f-625d-4074-9a24-55f2af08717d","Type":"ContainerDied","Data":"c4fbc88636deccf7348ebe34ec5169fce5c9c2dd421ec84a0c1844fa8fda2bf7"} Oct 30 09:35:40.292886 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:40.292496 4562 scope.go:122] "RemoveContainer" containerID="2b9edfc3a8fac46e2d9cb71ff42bb7215f6ecf52aeb69d6281f93360b8ba3997" Oct 30 09:35:40.308436 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:40.308367 4562 scope.go:122] "RemoveContainer" containerID="2b9edfc3a8fac46e2d9cb71ff42bb7215f6ecf52aeb69d6281f93360b8ba3997" Oct 30 09:35:40.308787 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:35:40.308762 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b9edfc3a8fac46e2d9cb71ff42bb7215f6ecf52aeb69d6281f93360b8ba3997\": container with ID starting with 2b9edfc3a8fac46e2d9cb71ff42bb7215f6ecf52aeb69d6281f93360b8ba3997 not found: ID does not exist" containerID="2b9edfc3a8fac46e2d9cb71ff42bb7215f6ecf52aeb69d6281f93360b8ba3997" Oct 30 09:35:40.308949 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:40.308799 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b9edfc3a8fac46e2d9cb71ff42bb7215f6ecf52aeb69d6281f93360b8ba3997"} err="failed to get container status \"2b9edfc3a8fac46e2d9cb71ff42bb7215f6ecf52aeb69d6281f93360b8ba3997\": rpc error: code = NotFound desc = could not find container \"2b9edfc3a8fac46e2d9cb71ff42bb7215f6ecf52aeb69d6281f93360b8ba3997\": container with ID starting with 2b9edfc3a8fac46e2d9cb71ff42bb7215f6ecf52aeb69d6281f93360b8ba3997 not found: ID does not exist" Oct 30 09:35:40.398407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:40.394222 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-1413c8-6566657db8-lcr4t"] Oct 30 09:35:40.402406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:40.399569 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-1413c8-6566657db8-lcr4t"] Oct 30 09:35:41.876355 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:41.876319 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a009b29f-625d-4074-9a24-55f2af08717d" path="/var/lib/kubelet/pods/a009b29f-625d-4074-9a24-55f2af08717d/volumes" Oct 30 09:35:42.585013 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:42.584972 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-1413c8-6566657db8-lcr4t" podUID="a009b29f-625d-4074-9a24-55f2af08717d" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.188:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:35:45.276847 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:45.276810 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-7a213d/auth-test-model-7a213d-kserve-54db88fccc-j8ldx"] Oct 30 09:35:45.277824 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:45.277802 4562 cpu_manager.go:414] "containerMap: removing container" podUID="23788da7-9c3c-45aa-b327-5f3f05c35cb6" containerName="payload-pre-processing" Oct 30 09:35:45.277824 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:45.277824 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23788da7-9c3c-45aa-b327-5f3f05c35cb6" containerName="payload-pre-processing" Oct 30 09:35:45.278014 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:45.277843 4562 cpu_manager.go:414] "containerMap: removing container" podUID="4084b690-89cf-4f47-911e-a49aa5368653" containerName="payload-processing" Oct 30 09:35:45.278014 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:45.277851 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4084b690-89cf-4f47-911e-a49aa5368653" containerName="payload-processing" Oct 30 09:35:45.278014 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:45.277890 4562 cpu_manager.go:414] "containerMap: removing container" podUID="7350869e-8e0b-41cd-b0fc-97686ccd12e4" containerName="main" Oct 30 09:35:45.278014 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:45.277897 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7350869e-8e0b-41cd-b0fc-97686ccd12e4" containerName="main" Oct 30 09:35:45.278014 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:45.277924 4562 cpu_manager.go:414] "containerMap: removing container" podUID="a009b29f-625d-4074-9a24-55f2af08717d" containerName="maas-api" Oct 30 09:35:45.278014 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:45.277931 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a009b29f-625d-4074-9a24-55f2af08717d" containerName="maas-api" Oct 30 09:35:45.278238 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:45.278087 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="7350869e-8e0b-41cd-b0fc-97686ccd12e4" containerName="main" Oct 30 09:35:45.278238 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:45.278101 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="a009b29f-625d-4074-9a24-55f2af08717d" containerName="maas-api" Oct 30 09:35:45.278238 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:45.278117 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="23788da7-9c3c-45aa-b327-5f3f05c35cb6" containerName="payload-pre-processing" Oct 30 09:35:45.278238 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:45.278128 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="4084b690-89cf-4f47-911e-a49aa5368653" containerName="payload-processing" Oct 30 09:35:45.284036 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:45.284012 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7a213d/auth-test-model-7a213d-kserve-54db88fccc-j8ldx" Oct 30 09:35:45.300573 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:45.300544 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7a213d/auth-test-model-7a213d-kserve-54db88fccc-j8ldx"] Oct 30 09:35:45.308939 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:45.308907 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7a213d\"/\"kube-root-ca.crt\"" Oct 30 09:35:45.309068 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:45.308953 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7a213d\"/\"default-dockercfg-f6z9s\"" Oct 30 09:35:45.313994 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:45.313957 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7a213d\"/\"auth-test-model-7a213d-kserve-self-signed-certs\"" Oct 30 09:35:45.314123 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:45.314052 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7a213d\"/\"openshift-service-ca.crt\"" Oct 30 09:35:45.347830 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:45.347803 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqwrw\" (UniqueName: \"kubernetes.io/projected/75eb9d8e-5d05-47d3-b923-06404edb57b2-kube-api-access-nqwrw\") pod \"auth-test-model-7a213d-kserve-54db88fccc-j8ldx\" (UID: \"75eb9d8e-5d05-47d3-b923-06404edb57b2\") " pod="ai-tenant-e2e-shared-b-w3-7a213d/auth-test-model-7a213d-kserve-54db88fccc-j8ldx" Oct 30 09:35:45.347959 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:45.347864 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75eb9d8e-5d05-47d3-b923-06404edb57b2-home\") pod \"auth-test-model-7a213d-kserve-54db88fccc-j8ldx\" (UID: \"75eb9d8e-5d05-47d3-b923-06404edb57b2\") " pod="ai-tenant-e2e-shared-b-w3-7a213d/auth-test-model-7a213d-kserve-54db88fccc-j8ldx" Oct 30 09:35:45.347959 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:45.347935 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75eb9d8e-5d05-47d3-b923-06404edb57b2-tmp-dir\") pod \"auth-test-model-7a213d-kserve-54db88fccc-j8ldx\" (UID: \"75eb9d8e-5d05-47d3-b923-06404edb57b2\") " pod="ai-tenant-e2e-shared-b-w3-7a213d/auth-test-model-7a213d-kserve-54db88fccc-j8ldx" Oct 30 09:35:45.348110 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:45.348093 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75eb9d8e-5d05-47d3-b923-06404edb57b2-model-cache\") pod \"auth-test-model-7a213d-kserve-54db88fccc-j8ldx\" (UID: \"75eb9d8e-5d05-47d3-b923-06404edb57b2\") " pod="ai-tenant-e2e-shared-b-w3-7a213d/auth-test-model-7a213d-kserve-54db88fccc-j8ldx" Oct 30 09:35:45.348173 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:45.348146 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75eb9d8e-5d05-47d3-b923-06404edb57b2-dshm\") pod \"auth-test-model-7a213d-kserve-54db88fccc-j8ldx\" (UID: \"75eb9d8e-5d05-47d3-b923-06404edb57b2\") " pod="ai-tenant-e2e-shared-b-w3-7a213d/auth-test-model-7a213d-kserve-54db88fccc-j8ldx" Oct 30 09:35:45.348298 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:45.348230 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75eb9d8e-5d05-47d3-b923-06404edb57b2-tls-certs\") pod \"auth-test-model-7a213d-kserve-54db88fccc-j8ldx\" (UID: \"75eb9d8e-5d05-47d3-b923-06404edb57b2\") " pod="ai-tenant-e2e-shared-b-w3-7a213d/auth-test-model-7a213d-kserve-54db88fccc-j8ldx" Oct 30 09:35:45.450217 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:45.450177 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75eb9d8e-5d05-47d3-b923-06404edb57b2-model-cache\") pod \"auth-test-model-7a213d-kserve-54db88fccc-j8ldx\" (UID: \"75eb9d8e-5d05-47d3-b923-06404edb57b2\") " pod="ai-tenant-e2e-shared-b-w3-7a213d/auth-test-model-7a213d-kserve-54db88fccc-j8ldx" Oct 30 09:35:45.450428 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:45.450268 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75eb9d8e-5d05-47d3-b923-06404edb57b2-dshm\") pod \"auth-test-model-7a213d-kserve-54db88fccc-j8ldx\" (UID: \"75eb9d8e-5d05-47d3-b923-06404edb57b2\") " pod="ai-tenant-e2e-shared-b-w3-7a213d/auth-test-model-7a213d-kserve-54db88fccc-j8ldx" Oct 30 09:35:45.450428 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:45.450311 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75eb9d8e-5d05-47d3-b923-06404edb57b2-tls-certs\") pod \"auth-test-model-7a213d-kserve-54db88fccc-j8ldx\" (UID: \"75eb9d8e-5d05-47d3-b923-06404edb57b2\") " pod="ai-tenant-e2e-shared-b-w3-7a213d/auth-test-model-7a213d-kserve-54db88fccc-j8ldx" Oct 30 09:35:45.450530 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:45.450433 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqwrw\" (UniqueName: \"kubernetes.io/projected/75eb9d8e-5d05-47d3-b923-06404edb57b2-kube-api-access-nqwrw\") pod \"auth-test-model-7a213d-kserve-54db88fccc-j8ldx\" (UID: \"75eb9d8e-5d05-47d3-b923-06404edb57b2\") " pod="ai-tenant-e2e-shared-b-w3-7a213d/auth-test-model-7a213d-kserve-54db88fccc-j8ldx" Oct 30 09:35:45.450530 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:45.450513 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75eb9d8e-5d05-47d3-b923-06404edb57b2-home\") pod \"auth-test-model-7a213d-kserve-54db88fccc-j8ldx\" (UID: \"75eb9d8e-5d05-47d3-b923-06404edb57b2\") " pod="ai-tenant-e2e-shared-b-w3-7a213d/auth-test-model-7a213d-kserve-54db88fccc-j8ldx" Oct 30 09:35:45.450614 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:45.450550 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75eb9d8e-5d05-47d3-b923-06404edb57b2-tmp-dir\") pod \"auth-test-model-7a213d-kserve-54db88fccc-j8ldx\" (UID: \"75eb9d8e-5d05-47d3-b923-06404edb57b2\") " pod="ai-tenant-e2e-shared-b-w3-7a213d/auth-test-model-7a213d-kserve-54db88fccc-j8ldx" Oct 30 09:35:45.450696 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:45.450622 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75eb9d8e-5d05-47d3-b923-06404edb57b2-model-cache\") pod \"auth-test-model-7a213d-kserve-54db88fccc-j8ldx\" (UID: \"75eb9d8e-5d05-47d3-b923-06404edb57b2\") " pod="ai-tenant-e2e-shared-b-w3-7a213d/auth-test-model-7a213d-kserve-54db88fccc-j8ldx" Oct 30 09:35:45.450961 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:45.450933 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75eb9d8e-5d05-47d3-b923-06404edb57b2-home\") pod \"auth-test-model-7a213d-kserve-54db88fccc-j8ldx\" (UID: \"75eb9d8e-5d05-47d3-b923-06404edb57b2\") " pod="ai-tenant-e2e-shared-b-w3-7a213d/auth-test-model-7a213d-kserve-54db88fccc-j8ldx" Oct 30 09:35:45.451443 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:45.451415 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75eb9d8e-5d05-47d3-b923-06404edb57b2-tmp-dir\") pod \"auth-test-model-7a213d-kserve-54db88fccc-j8ldx\" (UID: \"75eb9d8e-5d05-47d3-b923-06404edb57b2\") " pod="ai-tenant-e2e-shared-b-w3-7a213d/auth-test-model-7a213d-kserve-54db88fccc-j8ldx" Oct 30 09:35:45.453215 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:45.453186 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75eb9d8e-5d05-47d3-b923-06404edb57b2-dshm\") pod \"auth-test-model-7a213d-kserve-54db88fccc-j8ldx\" (UID: \"75eb9d8e-5d05-47d3-b923-06404edb57b2\") " pod="ai-tenant-e2e-shared-b-w3-7a213d/auth-test-model-7a213d-kserve-54db88fccc-j8ldx" Oct 30 09:35:45.453318 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:45.453227 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75eb9d8e-5d05-47d3-b923-06404edb57b2-tls-certs\") pod \"auth-test-model-7a213d-kserve-54db88fccc-j8ldx\" (UID: \"75eb9d8e-5d05-47d3-b923-06404edb57b2\") " pod="ai-tenant-e2e-shared-b-w3-7a213d/auth-test-model-7a213d-kserve-54db88fccc-j8ldx" Oct 30 09:35:45.470905 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:45.470884 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqwrw\" (UniqueName: \"kubernetes.io/projected/75eb9d8e-5d05-47d3-b923-06404edb57b2-kube-api-access-nqwrw\") pod \"auth-test-model-7a213d-kserve-54db88fccc-j8ldx\" (UID: \"75eb9d8e-5d05-47d3-b923-06404edb57b2\") " pod="ai-tenant-e2e-shared-b-w3-7a213d/auth-test-model-7a213d-kserve-54db88fccc-j8ldx" Oct 30 09:35:45.604715 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:45.603414 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7a213d/auth-test-model-7a213d-kserve-54db88fccc-j8ldx" Oct 30 09:35:45.935914 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:35:45.935885 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75eb9d8e_5d05_47d3_b923_06404edb57b2.slice/crio-aa0bfede847e016de3aeef16fc69f11fc4153395b2801851f8f4c5e5e8b9e774 WatchSource:0}: Error finding container aa0bfede847e016de3aeef16fc69f11fc4153395b2801851f8f4c5e5e8b9e774: Status 404 returned error can't find the container with id aa0bfede847e016de3aeef16fc69f11fc4153395b2801851f8f4c5e5e8b9e774 Oct 30 09:35:45.937280 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:45.937250 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7a213d/auth-test-model-7a213d-kserve-54db88fccc-j8ldx"] Oct 30 09:35:46.337727 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:46.337692 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7a213d/auth-test-model-7a213d-kserve-54db88fccc-j8ldx" event={"ID":"75eb9d8e-5d05-47d3-b923-06404edb57b2","Type":"ContainerStarted","Data":"68fdb6dcde6cb7a89b57259897d851965854add2e91f6ea4636168b922c46ac3"} Oct 30 09:35:46.337727 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:46.337734 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7a213d/auth-test-model-7a213d-kserve-54db88fccc-j8ldx" event={"ID":"75eb9d8e-5d05-47d3-b923-06404edb57b2","Type":"ContainerStarted","Data":"aa0bfede847e016de3aeef16fc69f11fc4153395b2801851f8f4c5e5e8b9e774"} Oct 30 09:35:46.382516 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:46.382463 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-7a213d/auth-test-model-7a213d-kserve-54db88fccc-j8ldx" podStartSLOduration=1.382451369 podStartE2EDuration="1.382451369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:35:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:35:46.380978773 +0000 UTC m=+2506.642539761" watchObservedRunningTime="2026-10-30 09:35:46.382451369 +0000 UTC m=+2506.644012356" Oct 30 09:35:46.604690 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:46.604598 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-7a213d/auth-test-model-7a213d-kserve-54db88fccc-j8ldx" Oct 30 09:35:53.877977 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:53.877941 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-d3047c/noten-model-d3047c-kserve-657778cddf-w5vxs"] Oct 30 09:35:53.883469 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:53.883443 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-d3047c/noten-model-d3047c-kserve-657778cddf-w5vxs" Oct 30 09:35:53.896136 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:53.895228 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-d3047c/noten-model-d3047c-kserve-657778cddf-w5vxs"] Oct 30 09:35:53.901766 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:53.901738 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-d3047c\"/\"openshift-service-ca.crt\"" Oct 30 09:35:53.913489 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:53.912362 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-d3047c\"/\"noten-model-d3047c-kserve-self-signed-certs\"" Oct 30 09:35:53.913489 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:53.912443 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-d3047c\"/\"kube-root-ca.crt\"" Oct 30 09:35:53.913489 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:53.912531 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-d3047c\"/\"default-dockercfg-74wss\"" Oct 30 09:35:54.023215 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:54.023185 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab15174a-6d56-4183-a923-25fa16ff9b4d-dshm\") pod \"noten-model-d3047c-kserve-657778cddf-w5vxs\" (UID: \"ab15174a-6d56-4183-a923-25fa16ff9b4d\") " pod="ai-tenant-e2e-noten-d3047c/noten-model-d3047c-kserve-657778cddf-w5vxs" Oct 30 09:35:54.023215 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:54.023217 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab15174a-6d56-4183-a923-25fa16ff9b4d-home\") pod \"noten-model-d3047c-kserve-657778cddf-w5vxs\" (UID: \"ab15174a-6d56-4183-a923-25fa16ff9b4d\") " pod="ai-tenant-e2e-noten-d3047c/noten-model-d3047c-kserve-657778cddf-w5vxs" Oct 30 09:35:54.023478 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:54.023242 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab15174a-6d56-4183-a923-25fa16ff9b4d-tls-certs\") pod \"noten-model-d3047c-kserve-657778cddf-w5vxs\" (UID: \"ab15174a-6d56-4183-a923-25fa16ff9b4d\") " pod="ai-tenant-e2e-noten-d3047c/noten-model-d3047c-kserve-657778cddf-w5vxs" Oct 30 09:35:54.023478 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:54.023439 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab15174a-6d56-4183-a923-25fa16ff9b4d-model-cache\") pod \"noten-model-d3047c-kserve-657778cddf-w5vxs\" (UID: \"ab15174a-6d56-4183-a923-25fa16ff9b4d\") " pod="ai-tenant-e2e-noten-d3047c/noten-model-d3047c-kserve-657778cddf-w5vxs" Oct 30 09:35:54.023478 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:54.023474 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab15174a-6d56-4183-a923-25fa16ff9b4d-tmp-dir\") pod \"noten-model-d3047c-kserve-657778cddf-w5vxs\" (UID: \"ab15174a-6d56-4183-a923-25fa16ff9b4d\") " pod="ai-tenant-e2e-noten-d3047c/noten-model-d3047c-kserve-657778cddf-w5vxs" Oct 30 09:35:54.023648 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:54.023630 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq2v5\" (UniqueName: \"kubernetes.io/projected/ab15174a-6d56-4183-a923-25fa16ff9b4d-kube-api-access-xq2v5\") pod \"noten-model-d3047c-kserve-657778cddf-w5vxs\" (UID: \"ab15174a-6d56-4183-a923-25fa16ff9b4d\") " pod="ai-tenant-e2e-noten-d3047c/noten-model-d3047c-kserve-657778cddf-w5vxs" Oct 30 09:35:54.125552 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:54.125522 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab15174a-6d56-4183-a923-25fa16ff9b4d-dshm\") pod \"noten-model-d3047c-kserve-657778cddf-w5vxs\" (UID: \"ab15174a-6d56-4183-a923-25fa16ff9b4d\") " pod="ai-tenant-e2e-noten-d3047c/noten-model-d3047c-kserve-657778cddf-w5vxs" Oct 30 09:35:54.125552 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:54.125556 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab15174a-6d56-4183-a923-25fa16ff9b4d-home\") pod \"noten-model-d3047c-kserve-657778cddf-w5vxs\" (UID: \"ab15174a-6d56-4183-a923-25fa16ff9b4d\") " pod="ai-tenant-e2e-noten-d3047c/noten-model-d3047c-kserve-657778cddf-w5vxs" Oct 30 09:35:54.125800 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:54.125579 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab15174a-6d56-4183-a923-25fa16ff9b4d-tls-certs\") pod \"noten-model-d3047c-kserve-657778cddf-w5vxs\" (UID: \"ab15174a-6d56-4183-a923-25fa16ff9b4d\") " pod="ai-tenant-e2e-noten-d3047c/noten-model-d3047c-kserve-657778cddf-w5vxs" Oct 30 09:35:54.125800 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:54.125687 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab15174a-6d56-4183-a923-25fa16ff9b4d-model-cache\") pod \"noten-model-d3047c-kserve-657778cddf-w5vxs\" (UID: \"ab15174a-6d56-4183-a923-25fa16ff9b4d\") " pod="ai-tenant-e2e-noten-d3047c/noten-model-d3047c-kserve-657778cddf-w5vxs" Oct 30 09:35:54.125800 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:54.125712 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab15174a-6d56-4183-a923-25fa16ff9b4d-tmp-dir\") pod \"noten-model-d3047c-kserve-657778cddf-w5vxs\" (UID: \"ab15174a-6d56-4183-a923-25fa16ff9b4d\") " pod="ai-tenant-e2e-noten-d3047c/noten-model-d3047c-kserve-657778cddf-w5vxs" Oct 30 09:35:54.125982 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:54.125893 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq2v5\" (UniqueName: \"kubernetes.io/projected/ab15174a-6d56-4183-a923-25fa16ff9b4d-kube-api-access-xq2v5\") pod \"noten-model-d3047c-kserve-657778cddf-w5vxs\" (UID: \"ab15174a-6d56-4183-a923-25fa16ff9b4d\") " pod="ai-tenant-e2e-noten-d3047c/noten-model-d3047c-kserve-657778cddf-w5vxs" Oct 30 09:35:54.125982 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:54.125924 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab15174a-6d56-4183-a923-25fa16ff9b4d-home\") pod \"noten-model-d3047c-kserve-657778cddf-w5vxs\" (UID: \"ab15174a-6d56-4183-a923-25fa16ff9b4d\") " pod="ai-tenant-e2e-noten-d3047c/noten-model-d3047c-kserve-657778cddf-w5vxs" Oct 30 09:35:54.126085 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:54.126051 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab15174a-6d56-4183-a923-25fa16ff9b4d-model-cache\") pod \"noten-model-d3047c-kserve-657778cddf-w5vxs\" (UID: \"ab15174a-6d56-4183-a923-25fa16ff9b4d\") " pod="ai-tenant-e2e-noten-d3047c/noten-model-d3047c-kserve-657778cddf-w5vxs" Oct 30 09:35:54.126125 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:54.126080 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab15174a-6d56-4183-a923-25fa16ff9b4d-tmp-dir\") pod \"noten-model-d3047c-kserve-657778cddf-w5vxs\" (UID: \"ab15174a-6d56-4183-a923-25fa16ff9b4d\") " pod="ai-tenant-e2e-noten-d3047c/noten-model-d3047c-kserve-657778cddf-w5vxs" Oct 30 09:35:54.127829 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:54.127810 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab15174a-6d56-4183-a923-25fa16ff9b4d-dshm\") pod \"noten-model-d3047c-kserve-657778cddf-w5vxs\" (UID: \"ab15174a-6d56-4183-a923-25fa16ff9b4d\") " pod="ai-tenant-e2e-noten-d3047c/noten-model-d3047c-kserve-657778cddf-w5vxs" Oct 30 09:35:54.128234 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:54.128184 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab15174a-6d56-4183-a923-25fa16ff9b4d-tls-certs\") pod \"noten-model-d3047c-kserve-657778cddf-w5vxs\" (UID: \"ab15174a-6d56-4183-a923-25fa16ff9b4d\") " pod="ai-tenant-e2e-noten-d3047c/noten-model-d3047c-kserve-657778cddf-w5vxs" Oct 30 09:35:54.172408 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:54.169879 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq2v5\" (UniqueName: \"kubernetes.io/projected/ab15174a-6d56-4183-a923-25fa16ff9b4d-kube-api-access-xq2v5\") pod \"noten-model-d3047c-kserve-657778cddf-w5vxs\" (UID: \"ab15174a-6d56-4183-a923-25fa16ff9b4d\") " pod="ai-tenant-e2e-noten-d3047c/noten-model-d3047c-kserve-657778cddf-w5vxs" Oct 30 09:35:54.218424 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:54.210608 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-d3047c/noten-model-d3047c-kserve-657778cddf-w5vxs" Oct 30 09:35:54.569255 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:54.569221 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-d3047c/noten-model-d3047c-kserve-657778cddf-w5vxs"] Oct 30 09:35:54.576014 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:35:54.575985 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab15174a_6d56_4183_a923_25fa16ff9b4d.slice/crio-ab9100b29f41d806afa119fc7fae6afd162f6091f4bba297fd7482cb244ea1a8 WatchSource:0}: Error finding container ab9100b29f41d806afa119fc7fae6afd162f6091f4bba297fd7482cb244ea1a8: Status 404 returned error can't find the container with id ab9100b29f41d806afa119fc7fae6afd162f6091f4bba297fd7482cb244ea1a8 Oct 30 09:35:55.398619 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:55.398576 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-d3047c/noten-model-d3047c-kserve-657778cddf-w5vxs" event={"ID":"ab15174a-6d56-4183-a923-25fa16ff9b4d","Type":"ContainerStarted","Data":"6c5da69f6d892138f4aa439b3ef19ebef8595e09246a693538480bf8426f9b72"} Oct 30 09:35:55.398619 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:55.398624 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-d3047c/noten-model-d3047c-kserve-657778cddf-w5vxs" event={"ID":"ab15174a-6d56-4183-a923-25fa16ff9b4d","Type":"ContainerStarted","Data":"ab9100b29f41d806afa119fc7fae6afd162f6091f4bba297fd7482cb244ea1a8"} Oct 30 09:35:55.440226 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:55.440173 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-d3047c/noten-model-d3047c-kserve-657778cddf-w5vxs" podStartSLOduration=2.440160572 podStartE2EDuration="2.440160572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:35:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:35:55.439550112 +0000 UTC m=+2515.701111106" watchObservedRunningTime="2026-10-30 09:35:55.440160572 +0000 UTC m=+2515.701721560" Oct 30 09:35:55.605037 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:55.605003 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-7a213d/auth-test-model-7a213d-kserve-54db88fccc-j8ldx" Oct 30 09:35:55.624598 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:55.624567 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-7a213d/auth-test-model-7a213d-kserve-54db88fccc-j8ldx" Oct 30 09:35:56.211493 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:56.211456 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-d3047c/noten-model-d3047c-kserve-657778cddf-w5vxs" Oct 30 09:35:56.416930 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:35:56.416897 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-7a213d/auth-test-model-7a213d-kserve-54db88fccc-j8ldx" Oct 30 09:35:58.597581 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:35:58.597544 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:36:04.211339 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:36:04.211302 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-d3047c/noten-model-d3047c-kserve-657778cddf-w5vxs" Oct 30 09:36:04.224689 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:36:04.224662 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-d3047c/noten-model-d3047c-kserve-657778cddf-w5vxs" Oct 30 09:36:04.512289 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:36:04.512215 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-d3047c/noten-model-d3047c-kserve-657778cddf-w5vxs" Oct 30 09:36:15.162096 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:36:15.162051 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-d3047c/noten-model-d3047c-kserve-657778cddf-w5vxs"] Oct 30 09:36:28.696719 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:36:28.696656 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:36:30.186289 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:36:30.186240 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-d3047c/noten-model-d3047c-kserve-657778cddf-w5vxs" podUID="ab15174a-6d56-4183-a923-25fa16ff9b4d" containerName="main" containerID="cri-o://6c5da69f6d892138f4aa439b3ef19ebef8595e09246a693538480bf8426f9b72" gracePeriod=45 Oct 30 09:36:30.532266 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:36:30.532232 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-d3047c/noten-model-d3047c-kserve-657778cddf-w5vxs" Oct 30 09:36:30.592218 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:36:30.592188 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab15174a-6d56-4183-a923-25fa16ff9b4d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab15174a-6d56-4183-a923-25fa16ff9b4d-tmp-dir\") pod \"ab15174a-6d56-4183-a923-25fa16ff9b4d\" (UID: \"ab15174a-6d56-4183-a923-25fa16ff9b4d\") " Oct 30 09:36:30.592433 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:36:30.592274 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab15174a-6d56-4183-a923-25fa16ff9b4d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab15174a-6d56-4183-a923-25fa16ff9b4d-model-cache\") pod \"ab15174a-6d56-4183-a923-25fa16ff9b4d\" (UID: \"ab15174a-6d56-4183-a923-25fa16ff9b4d\") " Oct 30 09:36:30.592433 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:36:30.592320 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab15174a-6d56-4183-a923-25fa16ff9b4d-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab15174a-6d56-4183-a923-25fa16ff9b4d-tls-certs\") pod \"ab15174a-6d56-4183-a923-25fa16ff9b4d\" (UID: \"ab15174a-6d56-4183-a923-25fa16ff9b4d\") " Oct 30 09:36:30.592572 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:36:30.592469 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab15174a-6d56-4183-a923-25fa16ff9b4d-home\" (UniqueName: \"kubernetes.io/empty-dir/ab15174a-6d56-4183-a923-25fa16ff9b4d-home\") pod \"ab15174a-6d56-4183-a923-25fa16ff9b4d\" (UID: \"ab15174a-6d56-4183-a923-25fa16ff9b4d\") " Oct 30 09:36:30.592670 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:36:30.592580 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab15174a-6d56-4183-a923-25fa16ff9b4d-tmp-dir" pod "ab15174a-6d56-4183-a923-25fa16ff9b4d" (UID: "ab15174a-6d56-4183-a923-25fa16ff9b4d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:36:30.592670 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:36:30.592603 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab15174a-6d56-4183-a923-25fa16ff9b4d-model-cache" pod "ab15174a-6d56-4183-a923-25fa16ff9b4d" (UID: "ab15174a-6d56-4183-a923-25fa16ff9b4d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:36:30.592670 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:36:30.592635 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab15174a-6d56-4183-a923-25fa16ff9b4d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab15174a-6d56-4183-a923-25fa16ff9b4d-dshm\") pod \"ab15174a-6d56-4183-a923-25fa16ff9b4d\" (UID: \"ab15174a-6d56-4183-a923-25fa16ff9b4d\") " Oct 30 09:36:30.592822 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:36:30.592693 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab15174a-6d56-4183-a923-25fa16ff9b4d-kube-api-access-xq2v5\" (UniqueName: \"kubernetes.io/projected/ab15174a-6d56-4183-a923-25fa16ff9b4d-kube-api-access-xq2v5\") pod \"ab15174a-6d56-4183-a923-25fa16ff9b4d\" (UID: \"ab15174a-6d56-4183-a923-25fa16ff9b4d\") " Oct 30 09:36:30.593399 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:36:30.592994 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab15174a-6d56-4183-a923-25fa16ff9b4d-home" pod "ab15174a-6d56-4183-a923-25fa16ff9b4d" (UID: "ab15174a-6d56-4183-a923-25fa16ff9b4d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:36:30.593399 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:36:30.593325 4562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab15174a-6d56-4183-a923-25fa16ff9b4d-tmp-dir\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:36:30.593399 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:36:30.593345 4562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab15174a-6d56-4183-a923-25fa16ff9b4d-model-cache\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:36:30.593399 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:36:30.593357 4562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab15174a-6d56-4183-a923-25fa16ff9b4d-home\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:36:30.595124 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:36:30.595096 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab15174a-6d56-4183-a923-25fa16ff9b4d-dshm" pod "ab15174a-6d56-4183-a923-25fa16ff9b4d" (UID: "ab15174a-6d56-4183-a923-25fa16ff9b4d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:36:30.595442 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:36:30.595363 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab15174a-6d56-4183-a923-25fa16ff9b4d-kube-api-access-xq2v5" pod "ab15174a-6d56-4183-a923-25fa16ff9b4d" (UID: "ab15174a-6d56-4183-a923-25fa16ff9b4d"). InnerVolumeSpecName "kube-api-access-xq2v5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:36:30.595803 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:36:30.595777 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab15174a-6d56-4183-a923-25fa16ff9b4d-tls-certs" pod "ab15174a-6d56-4183-a923-25fa16ff9b4d" (UID: "ab15174a-6d56-4183-a923-25fa16ff9b4d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:36:30.694868 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:36:30.694830 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xq2v5\" (UniqueName: \"kubernetes.io/projected/ab15174a-6d56-4183-a923-25fa16ff9b4d-kube-api-access-xq2v5\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:36:30.694868 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:36:30.694862 4562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab15174a-6d56-4183-a923-25fa16ff9b4d-tls-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:36:30.694868 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:36:30.694872 4562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab15174a-6d56-4183-a923-25fa16ff9b4d-dshm\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:36:30.760319 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:36:30.760229 4562 generic.go:356] "Generic (PLEG): container finished" podID="ab15174a-6d56-4183-a923-25fa16ff9b4d" containerID="6c5da69f6d892138f4aa439b3ef19ebef8595e09246a693538480bf8426f9b72" exitCode=0 Oct 30 09:36:30.760319 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:36:30.760278 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-d3047c/noten-model-d3047c-kserve-657778cddf-w5vxs" Oct 30 09:36:30.760609 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:36:30.760321 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-d3047c/noten-model-d3047c-kserve-657778cddf-w5vxs" event={"ID":"ab15174a-6d56-4183-a923-25fa16ff9b4d","Type":"ContainerDied","Data":"6c5da69f6d892138f4aa439b3ef19ebef8595e09246a693538480bf8426f9b72"} Oct 30 09:36:30.760609 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:36:30.760375 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-d3047c/noten-model-d3047c-kserve-657778cddf-w5vxs" event={"ID":"ab15174a-6d56-4183-a923-25fa16ff9b4d","Type":"ContainerDied","Data":"ab9100b29f41d806afa119fc7fae6afd162f6091f4bba297fd7482cb244ea1a8"} Oct 30 09:36:30.760609 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:36:30.760417 4562 scope.go:122] "RemoveContainer" containerID="6c5da69f6d892138f4aa439b3ef19ebef8595e09246a693538480bf8426f9b72" Oct 30 09:36:30.771693 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:36:30.771674 4562 scope.go:122] "RemoveContainer" containerID="6c5da69f6d892138f4aa439b3ef19ebef8595e09246a693538480bf8426f9b72" Oct 30 09:36:30.771939 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:36:30.771918 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c5da69f6d892138f4aa439b3ef19ebef8595e09246a693538480bf8426f9b72\": container with ID starting with 6c5da69f6d892138f4aa439b3ef19ebef8595e09246a693538480bf8426f9b72 not found: ID does not exist" containerID="6c5da69f6d892138f4aa439b3ef19ebef8595e09246a693538480bf8426f9b72" Oct 30 09:36:30.771987 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:36:30.771947 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c5da69f6d892138f4aa439b3ef19ebef8595e09246a693538480bf8426f9b72"} err="failed to get container status \"6c5da69f6d892138f4aa439b3ef19ebef8595e09246a693538480bf8426f9b72\": rpc error: code = NotFound desc = could not find container \"6c5da69f6d892138f4aa439b3ef19ebef8595e09246a693538480bf8426f9b72\": container with ID starting with 6c5da69f6d892138f4aa439b3ef19ebef8595e09246a693538480bf8426f9b72 not found: ID does not exist" Oct 30 09:36:30.816533 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:36:30.816494 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-d3047c/noten-model-d3047c-kserve-657778cddf-w5vxs"] Oct 30 09:36:30.820342 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:36:30.820318 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-d3047c/noten-model-d3047c-kserve-657778cddf-w5vxs"] Oct 30 09:36:31.502018 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:36:31.501972 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-noten-d3047c/noten-model-d3047c-kserve-657778cddf-w5vxs" podUID="ab15174a-6d56-4183-a923-25fa16ff9b4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.193:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 09:36:31.876065 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:36:31.875983 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab15174a-6d56-4183-a923-25fa16ff9b4d" path="/var/lib/kubelet/pods/ab15174a-6d56-4183-a923-25fa16ff9b4d/volumes" Oct 30 09:36:58.596541 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:36:58.596458 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:37:02.874659 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:02.874623 4562 scope.go:122] "RemoveContainer" containerID="390867dc8e77f223d0ad16c2e13467809a7ff6279076ed3e5cff3cd4704ede34" Oct 30 09:37:08.229885 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:08.229851 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8ebd65-7cb68f4d79-z5vx4"] Oct 30 09:37:08.230598 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:08.230580 4562 cpu_manager.go:414] "containerMap: removing container" podUID="ab15174a-6d56-4183-a923-25fa16ff9b4d" containerName="main" Oct 30 09:37:08.230598 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:08.230597 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab15174a-6d56-4183-a923-25fa16ff9b4d" containerName="main" Oct 30 09:37:08.230708 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:08.230699 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab15174a-6d56-4183-a923-25fa16ff9b4d" containerName="main" Oct 30 09:37:08.235550 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:08.235533 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8ebd65-7cb68f4d79-z5vx4" Oct 30 09:37:08.253318 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:08.253293 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8ebd65-7cb68f4d79-z5vx4"] Oct 30 09:37:08.260766 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:08.260743 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-8ebd65\"" Oct 30 09:37:08.322415 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:08.318834 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-8ebd65-dockercfg-5zk7s\"" Oct 30 09:37:08.386407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:08.365709 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vggbw\" (UniqueName: \"kubernetes.io/projected/aadf57f9-3206-4de6-87af-c42d14e3af5f-kube-api-access-vggbw\") pod \"payload-pre-processing-e2e-shared-a-w0-8ebd65-7cb68f4d79-z5vx4\" (UID: \"aadf57f9-3206-4de6-87af-c42d14e3af5f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8ebd65-7cb68f4d79-z5vx4" Oct 30 09:37:08.386407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:08.365861 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aadf57f9-3206-4de6-87af-c42d14e3af5f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-8ebd65-7cb68f4d79-z5vx4\" (UID: \"aadf57f9-3206-4de6-87af-c42d14e3af5f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8ebd65-7cb68f4d79-z5vx4" Oct 30 09:37:08.485454 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:08.484570 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-8ebd65-84fd6bbf6d-6t7tb"] Oct 30 09:37:08.495402 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:08.495349 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vggbw\" (UniqueName: \"kubernetes.io/projected/aadf57f9-3206-4de6-87af-c42d14e3af5f-kube-api-access-vggbw\") pod \"payload-pre-processing-e2e-shared-a-w0-8ebd65-7cb68f4d79-z5vx4\" (UID: \"aadf57f9-3206-4de6-87af-c42d14e3af5f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8ebd65-7cb68f4d79-z5vx4" Oct 30 09:37:08.495888 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:08.495857 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aadf57f9-3206-4de6-87af-c42d14e3af5f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-8ebd65-7cb68f4d79-z5vx4\" (UID: \"aadf57f9-3206-4de6-87af-c42d14e3af5f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8ebd65-7cb68f4d79-z5vx4" Oct 30 09:37:08.497198 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:08.497171 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aadf57f9-3206-4de6-87af-c42d14e3af5f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-8ebd65-7cb68f4d79-z5vx4\" (UID: \"aadf57f9-3206-4de6-87af-c42d14e3af5f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8ebd65-7cb68f4d79-z5vx4" Oct 30 09:37:08.503158 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:08.503126 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8ebd65-84fd6bbf6d-6t7tb" Oct 30 09:37:08.532134 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:08.532100 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vggbw\" (UniqueName: \"kubernetes.io/projected/aadf57f9-3206-4de6-87af-c42d14e3af5f-kube-api-access-vggbw\") pod \"payload-pre-processing-e2e-shared-a-w0-8ebd65-7cb68f4d79-z5vx4\" (UID: \"aadf57f9-3206-4de6-87af-c42d14e3af5f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8ebd65-7cb68f4d79-z5vx4" Oct 30 09:37:08.532752 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:08.532724 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-8ebd65-84fd6bbf6d-6t7tb"] Oct 30 09:37:08.553684 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:08.553652 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8ebd65-7cb68f4d79-z5vx4" Oct 30 09:37:08.597490 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:08.597453 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74fa48b5-9049-4bb8-978d-5b7773f0b93e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-8ebd65-84fd6bbf6d-6t7tb\" (UID: \"74fa48b5-9049-4bb8-978d-5b7773f0b93e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8ebd65-84fd6bbf6d-6t7tb" Oct 30 09:37:08.597718 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:08.597682 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7chlk\" (UniqueName: \"kubernetes.io/projected/74fa48b5-9049-4bb8-978d-5b7773f0b93e-kube-api-access-7chlk\") pod \"payload-processing-e2e-shared-a-w0-8ebd65-84fd6bbf6d-6t7tb\" (UID: \"74fa48b5-9049-4bb8-978d-5b7773f0b93e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8ebd65-84fd6bbf6d-6t7tb" Oct 30 09:37:08.699220 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:08.699182 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7chlk\" (UniqueName: \"kubernetes.io/projected/74fa48b5-9049-4bb8-978d-5b7773f0b93e-kube-api-access-7chlk\") pod \"payload-processing-e2e-shared-a-w0-8ebd65-84fd6bbf6d-6t7tb\" (UID: \"74fa48b5-9049-4bb8-978d-5b7773f0b93e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8ebd65-84fd6bbf6d-6t7tb" Oct 30 09:37:08.699455 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:08.699359 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74fa48b5-9049-4bb8-978d-5b7773f0b93e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-8ebd65-84fd6bbf6d-6t7tb\" (UID: \"74fa48b5-9049-4bb8-978d-5b7773f0b93e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8ebd65-84fd6bbf6d-6t7tb" Oct 30 09:37:08.700285 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:08.700252 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74fa48b5-9049-4bb8-978d-5b7773f0b93e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-8ebd65-84fd6bbf6d-6t7tb\" (UID: \"74fa48b5-9049-4bb8-978d-5b7773f0b93e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8ebd65-84fd6bbf6d-6t7tb" Oct 30 09:37:08.719236 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:08.719206 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7chlk\" (UniqueName: \"kubernetes.io/projected/74fa48b5-9049-4bb8-978d-5b7773f0b93e-kube-api-access-7chlk\") pod \"payload-processing-e2e-shared-a-w0-8ebd65-84fd6bbf6d-6t7tb\" (UID: \"74fa48b5-9049-4bb8-978d-5b7773f0b93e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8ebd65-84fd6bbf6d-6t7tb" Oct 30 09:37:08.863644 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:08.863616 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8ebd65-84fd6bbf6d-6t7tb" Oct 30 09:37:08.863787 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:37:08.863764 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaadf57f9_3206_4de6_87af_c42d14e3af5f.slice/crio-e8d5e752e24e8cd94d2436ba000e3ae8c1cf998e6a2e46e3c70b75493bc2f853 WatchSource:0}: Error finding container e8d5e752e24e8cd94d2436ba000e3ae8c1cf998e6a2e46e3c70b75493bc2f853: Status 404 returned error can't find the container with id e8d5e752e24e8cd94d2436ba000e3ae8c1cf998e6a2e46e3c70b75493bc2f853 Oct 30 09:37:08.867930 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:08.867902 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8ebd65-7cb68f4d79-z5vx4"] Oct 30 09:37:09.030180 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:09.030135 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8ebd65-7cb68f4d79-z5vx4" event={"ID":"aadf57f9-3206-4de6-87af-c42d14e3af5f","Type":"ContainerStarted","Data":"e9e03a5a690efa822a04db830e494a779dd97fddaa48b5d46d60e3e3515e5536"} Oct 30 09:37:09.030405 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:09.030211 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8ebd65-7cb68f4d79-z5vx4" event={"ID":"aadf57f9-3206-4de6-87af-c42d14e3af5f","Type":"ContainerStarted","Data":"e8d5e752e24e8cd94d2436ba000e3ae8c1cf998e6a2e46e3c70b75493bc2f853"} Oct 30 09:37:09.030405 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:09.030295 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8ebd65-7cb68f4d79-z5vx4" Oct 30 09:37:09.071231 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:09.071171 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8ebd65-7cb68f4d79-z5vx4" podStartSLOduration=1.0711522470000001 podStartE2EDuration="1.071152247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:37:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:37:09.069170265 +0000 UTC m=+2589.330731255" watchObservedRunningTime="2026-10-30 09:37:09.071152247 +0000 UTC m=+2589.332713236" Oct 30 09:37:09.227745 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:09.223460 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-8ebd65-84fd6bbf6d-6t7tb"] Oct 30 09:37:10.038251 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:10.038213 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8ebd65-84fd6bbf6d-6t7tb" event={"ID":"74fa48b5-9049-4bb8-978d-5b7773f0b93e","Type":"ContainerStarted","Data":"6ceec0c628cceb12390d6fde7406af207acf002324fb1b08e54168782215119c"} Oct 30 09:37:10.038750 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:10.038272 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8ebd65-84fd6bbf6d-6t7tb" event={"ID":"74fa48b5-9049-4bb8-978d-5b7773f0b93e","Type":"ContainerStarted","Data":"e78f61201edb971d9e84d2ac59f64eae21e244402a958ca455013cff4a42e30e"} Oct 30 09:37:10.038750 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:10.038294 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8ebd65-84fd6bbf6d-6t7tb" Oct 30 09:37:10.082943 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:10.082888 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8ebd65-84fd6bbf6d-6t7tb" podStartSLOduration=2.08287105 podStartE2EDuration="2.08287105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:37:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:37:10.080462511 +0000 UTC m=+2590.342023501" watchObservedRunningTime="2026-10-30 09:37:10.08287105 +0000 UTC m=+2590.344432040" Oct 30 09:37:15.864231 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:15.864195 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11390-379eb6"] Oct 30 09:37:15.868916 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:15.868885 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11390-379eb6" Oct 30 09:37:15.884352 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:15.884322 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11390-379eb6"] Oct 30 09:37:15.893513 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:15.893490 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-jjt9f\"" Oct 30 09:37:16.045815 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:16.045780 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndgsr\" (UniqueName: \"kubernetes.io/projected/07038022-1dc8-4ac6-9bf6-07df45bb1da4-kube-api-access-ndgsr\") pod \"test-curl-11390-379eb6\" (UID: \"07038022-1dc8-4ac6-9bf6-07df45bb1da4\") " pod="opendatahub/test-curl-11390-379eb6" Oct 30 09:37:16.147618 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:16.147524 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndgsr\" (UniqueName: \"kubernetes.io/projected/07038022-1dc8-4ac6-9bf6-07df45bb1da4-kube-api-access-ndgsr\") pod \"test-curl-11390-379eb6\" (UID: \"07038022-1dc8-4ac6-9bf6-07df45bb1da4\") " pod="opendatahub/test-curl-11390-379eb6" Oct 30 09:37:16.168522 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:16.168491 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndgsr\" (UniqueName: \"kubernetes.io/projected/07038022-1dc8-4ac6-9bf6-07df45bb1da4-kube-api-access-ndgsr\") pod \"test-curl-11390-379eb6\" (UID: \"07038022-1dc8-4ac6-9bf6-07df45bb1da4\") " pod="opendatahub/test-curl-11390-379eb6" Oct 30 09:37:16.184534 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:16.184497 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11390-379eb6" Oct 30 09:37:16.546794 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:37:16.546753 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod07038022_1dc8_4ac6_9bf6_07df45bb1da4.slice/crio-cdf4193ecd825fa553facc0f944803cd9d69ead41d6398b6b4d873f731179bd2 WatchSource:0}: Error finding container cdf4193ecd825fa553facc0f944803cd9d69ead41d6398b6b4d873f731179bd2: Status 404 returned error can't find the container with id cdf4193ecd825fa553facc0f944803cd9d69ead41d6398b6b4d873f731179bd2 Oct 30 09:37:16.548234 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:16.548207 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11390-379eb6"] Oct 30 09:37:17.120656 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:17.120612 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11390-379eb6" event={"ID":"07038022-1dc8-4ac6-9bf6-07df45bb1da4","Type":"ContainerStarted","Data":"4b07c1f9a0357f23d155f8b3d1fe84bb654ba4b8d2bcb2e3f5306039aeada16c"} Oct 30 09:37:17.120656 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:17.120653 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11390-379eb6" event={"ID":"07038022-1dc8-4ac6-9bf6-07df45bb1da4","Type":"ContainerStarted","Data":"cdf4193ecd825fa553facc0f944803cd9d69ead41d6398b6b4d873f731179bd2"} Oct 30 09:37:17.164512 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:17.164444 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11390-379eb6" podStartSLOduration=2.164426828 podStartE2EDuration="2.164426828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:37:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:37:17.162615153 +0000 UTC m=+2597.424176142" watchObservedRunningTime="2026-10-30 09:37:17.164426828 +0000 UTC m=+2597.425987826" Oct 30 09:37:17.997191 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:17.997160 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11390-379eb6"] Oct 30 09:37:18.002594 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:18.002569 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11390-379eb6"] Oct 30 09:37:19.133944 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:19.133900 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11390-379eb6" podUID="07038022-1dc8-4ac6-9bf6-07df45bb1da4" containerName="test-curl-11390-379eb6" containerID="cri-o://4b07c1f9a0357f23d155f8b3d1fe84bb654ba4b8d2bcb2e3f5306039aeada16c" gracePeriod=2 Oct 30 09:37:19.148252 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:37:19.148218 4562 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11390-379eb6\" is forbidden: User \"system:node:ip-10-0-143-244.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-244.ec2.internal' and this object" podUID="07038022-1dc8-4ac6-9bf6-07df45bb1da4" pod="opendatahub/test-curl-11390-379eb6" Oct 30 09:37:19.885835 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:37:19.885751 4562 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11390-379eb6\" is forbidden: User \"system:node:ip-10-0-143-244.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-244.ec2.internal' and this object" podUID="07038022-1dc8-4ac6-9bf6-07df45bb1da4" pod="opendatahub/test-curl-11390-379eb6" Oct 30 09:37:20.042340 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:20.042312 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8ebd65-7cb68f4d79-z5vx4" Oct 30 09:37:21.049444 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:21.049412 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8ebd65-84fd6bbf6d-6t7tb" Oct 30 09:37:21.332641 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:21.332448 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11390-379eb6" Oct 30 09:37:21.434214 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:21.434181 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07038022-1dc8-4ac6-9bf6-07df45bb1da4-kube-api-access-ndgsr\" (UniqueName: \"kubernetes.io/projected/07038022-1dc8-4ac6-9bf6-07df45bb1da4-kube-api-access-ndgsr\") pod \"07038022-1dc8-4ac6-9bf6-07df45bb1da4\" (UID: \"07038022-1dc8-4ac6-9bf6-07df45bb1da4\") " Oct 30 09:37:21.436323 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:21.436297 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07038022-1dc8-4ac6-9bf6-07df45bb1da4-kube-api-access-ndgsr" pod "07038022-1dc8-4ac6-9bf6-07df45bb1da4" (UID: "07038022-1dc8-4ac6-9bf6-07df45bb1da4"). InnerVolumeSpecName "kube-api-access-ndgsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:37:21.538866 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:21.537582 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndgsr\" (UniqueName: \"kubernetes.io/projected/07038022-1dc8-4ac6-9bf6-07df45bb1da4-kube-api-access-ndgsr\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:37:21.876042 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:21.876009 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07038022-1dc8-4ac6-9bf6-07df45bb1da4" path="/var/lib/kubelet/pods/07038022-1dc8-4ac6-9bf6-07df45bb1da4/volumes" Oct 30 09:37:22.158238 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:22.158153 4562 generic.go:356] "Generic (PLEG): container finished" podID="07038022-1dc8-4ac6-9bf6-07df45bb1da4" containerID="4b07c1f9a0357f23d155f8b3d1fe84bb654ba4b8d2bcb2e3f5306039aeada16c" exitCode=137 Oct 30 09:37:22.158238 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:22.158197 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11390-379eb6" Oct 30 09:37:22.158764 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:22.158249 4562 scope.go:122] "RemoveContainer" containerID="4b07c1f9a0357f23d155f8b3d1fe84bb654ba4b8d2bcb2e3f5306039aeada16c" Oct 30 09:37:22.171274 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:22.171255 4562 scope.go:122] "RemoveContainer" containerID="4b07c1f9a0357f23d155f8b3d1fe84bb654ba4b8d2bcb2e3f5306039aeada16c" Oct 30 09:37:22.171577 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:37:22.171554 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b07c1f9a0357f23d155f8b3d1fe84bb654ba4b8d2bcb2e3f5306039aeada16c\": container with ID starting with 4b07c1f9a0357f23d155f8b3d1fe84bb654ba4b8d2bcb2e3f5306039aeada16c not found: ID does not exist" containerID="4b07c1f9a0357f23d155f8b3d1fe84bb654ba4b8d2bcb2e3f5306039aeada16c" Oct 30 09:37:22.171659 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:22.171587 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b07c1f9a0357f23d155f8b3d1fe84bb654ba4b8d2bcb2e3f5306039aeada16c"} err="failed to get container status \"4b07c1f9a0357f23d155f8b3d1fe84bb654ba4b8d2bcb2e3f5306039aeada16c\": rpc error: code = NotFound desc = could not find container \"4b07c1f9a0357f23d155f8b3d1fe84bb654ba4b8d2bcb2e3f5306039aeada16c\": container with ID starting with 4b07c1f9a0357f23d155f8b3d1fe84bb654ba4b8d2bcb2e3f5306039aeada16c not found: ID does not exist" Oct 30 09:37:28.597188 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:37:28.597148 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:37:52.647308 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:52.647272 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb"] Oct 30 09:37:52.647877 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:52.647503 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb" podUID="94202efb-a6b4-4260-bd94-16473a214a5c" containerName="istio-proxy" containerID="cri-o://101f0b8ae7a931c275b34237e93dca0566e73cdd34f4889ea503aeac9911e3a0" gracePeriod=30 Oct 30 09:37:57.922617 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:57.922561 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb" Oct 30 09:37:58.011550 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:58.011514 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94202efb-a6b4-4260-bd94-16473a214a5c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94202efb-a6b4-4260-bd94-16473a214a5c-workload-socket\") pod \"94202efb-a6b4-4260-bd94-16473a214a5c\" (UID: \"94202efb-a6b4-4260-bd94-16473a214a5c\") " Oct 30 09:37:58.011747 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:58.011576 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94202efb-a6b4-4260-bd94-16473a214a5c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94202efb-a6b4-4260-bd94-16473a214a5c-istio-data\") pod \"94202efb-a6b4-4260-bd94-16473a214a5c\" (UID: \"94202efb-a6b4-4260-bd94-16473a214a5c\") " Oct 30 09:37:58.011747 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:58.011617 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94202efb-a6b4-4260-bd94-16473a214a5c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94202efb-a6b4-4260-bd94-16473a214a5c-istio-envoy\") pod \"94202efb-a6b4-4260-bd94-16473a214a5c\" (UID: \"94202efb-a6b4-4260-bd94-16473a214a5c\") " Oct 30 09:37:58.011747 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:58.011709 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94202efb-a6b4-4260-bd94-16473a214a5c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94202efb-a6b4-4260-bd94-16473a214a5c-istiod-ca-cert\") pod \"94202efb-a6b4-4260-bd94-16473a214a5c\" (UID: \"94202efb-a6b4-4260-bd94-16473a214a5c\") " Oct 30 09:37:58.011951 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:58.011748 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94202efb-a6b4-4260-bd94-16473a214a5c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94202efb-a6b4-4260-bd94-16473a214a5c-workload-certs\") pod \"94202efb-a6b4-4260-bd94-16473a214a5c\" (UID: \"94202efb-a6b4-4260-bd94-16473a214a5c\") " Oct 30 09:37:58.011951 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:58.011795 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94202efb-a6b4-4260-bd94-16473a214a5c-istio-token\" (UniqueName: \"kubernetes.io/projected/94202efb-a6b4-4260-bd94-16473a214a5c-istio-token\") pod \"94202efb-a6b4-4260-bd94-16473a214a5c\" (UID: \"94202efb-a6b4-4260-bd94-16473a214a5c\") " Oct 30 09:37:58.011951 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:58.011844 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94202efb-a6b4-4260-bd94-16473a214a5c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94202efb-a6b4-4260-bd94-16473a214a5c-credential-socket\") pod \"94202efb-a6b4-4260-bd94-16473a214a5c\" (UID: \"94202efb-a6b4-4260-bd94-16473a214a5c\") " Oct 30 09:37:58.011951 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:58.011838 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94202efb-a6b4-4260-bd94-16473a214a5c-workload-socket" pod "94202efb-a6b4-4260-bd94-16473a214a5c" (UID: "94202efb-a6b4-4260-bd94-16473a214a5c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:37:58.011951 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:58.011882 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/94202efb-a6b4-4260-bd94-16473a214a5c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94202efb-a6b4-4260-bd94-16473a214a5c-istio-podinfo\") pod \"94202efb-a6b4-4260-bd94-16473a214a5c\" (UID: \"94202efb-a6b4-4260-bd94-16473a214a5c\") " Oct 30 09:37:58.011951 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:58.011895 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94202efb-a6b4-4260-bd94-16473a214a5c-istio-data" pod "94202efb-a6b4-4260-bd94-16473a214a5c" (UID: "94202efb-a6b4-4260-bd94-16473a214a5c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:37:58.011951 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:58.011946 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94202efb-a6b4-4260-bd94-16473a214a5c-kube-api-access-x9rv7\" (UniqueName: \"kubernetes.io/projected/94202efb-a6b4-4260-bd94-16473a214a5c-kube-api-access-x9rv7\") pod \"94202efb-a6b4-4260-bd94-16473a214a5c\" (UID: \"94202efb-a6b4-4260-bd94-16473a214a5c\") " Oct 30 09:37:58.012337 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:58.012099 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94202efb-a6b4-4260-bd94-16473a214a5c-workload-certs" pod "94202efb-a6b4-4260-bd94-16473a214a5c" (UID: "94202efb-a6b4-4260-bd94-16473a214a5c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:37:58.012441 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:58.012420 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94202efb-a6b4-4260-bd94-16473a214a5c-istiod-ca-cert" pod "94202efb-a6b4-4260-bd94-16473a214a5c" (UID: "94202efb-a6b4-4260-bd94-16473a214a5c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:37:58.012704 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:58.012682 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94202efb-a6b4-4260-bd94-16473a214a5c-credential-socket" pod "94202efb-a6b4-4260-bd94-16473a214a5c" (UID: "94202efb-a6b4-4260-bd94-16473a214a5c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:37:58.013030 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:58.013008 4562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94202efb-a6b4-4260-bd94-16473a214a5c-workload-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:37:58.013156 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:58.013142 4562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94202efb-a6b4-4260-bd94-16473a214a5c-istio-data\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:37:58.013254 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:58.013242 4562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94202efb-a6b4-4260-bd94-16473a214a5c-istiod-ca-cert\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:37:58.013344 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:58.013332 4562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94202efb-a6b4-4260-bd94-16473a214a5c-workload-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:37:58.013455 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:58.013441 4562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94202efb-a6b4-4260-bd94-16473a214a5c-credential-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:37:58.015440 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:58.015401 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94202efb-a6b4-4260-bd94-16473a214a5c-kube-api-access-x9rv7" pod "94202efb-a6b4-4260-bd94-16473a214a5c" (UID: "94202efb-a6b4-4260-bd94-16473a214a5c"). InnerVolumeSpecName "kube-api-access-x9rv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:37:58.017054 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:58.017026 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/94202efb-a6b4-4260-bd94-16473a214a5c-istio-podinfo" pod "94202efb-a6b4-4260-bd94-16473a214a5c" (UID: "94202efb-a6b4-4260-bd94-16473a214a5c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:37:58.017520 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:58.017494 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94202efb-a6b4-4260-bd94-16473a214a5c-istio-token" pod "94202efb-a6b4-4260-bd94-16473a214a5c" (UID: "94202efb-a6b4-4260-bd94-16473a214a5c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:37:58.017699 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:58.017677 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94202efb-a6b4-4260-bd94-16473a214a5c-istio-envoy" pod "94202efb-a6b4-4260-bd94-16473a214a5c" (UID: "94202efb-a6b4-4260-bd94-16473a214a5c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:37:58.115145 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:58.115106 4562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94202efb-a6b4-4260-bd94-16473a214a5c-istio-token\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:37:58.115145 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:58.115142 4562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94202efb-a6b4-4260-bd94-16473a214a5c-istio-podinfo\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:37:58.115431 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:58.115156 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9rv7\" (UniqueName: \"kubernetes.io/projected/94202efb-a6b4-4260-bd94-16473a214a5c-kube-api-access-x9rv7\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:37:58.115431 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:58.115171 4562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94202efb-a6b4-4260-bd94-16473a214a5c-istio-envoy\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:37:58.565229 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:58.565194 4562 generic.go:356] "Generic (PLEG): container finished" podID="94202efb-a6b4-4260-bd94-16473a214a5c" containerID="101f0b8ae7a931c275b34237e93dca0566e73cdd34f4889ea503aeac9911e3a0" exitCode=0 Oct 30 09:37:58.565416 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:58.565237 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb" Oct 30 09:37:58.565416 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:58.565271 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb" event={"ID":"94202efb-a6b4-4260-bd94-16473a214a5c","Type":"ContainerDied","Data":"101f0b8ae7a931c275b34237e93dca0566e73cdd34f4889ea503aeac9911e3a0"} Oct 30 09:37:58.565416 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:58.565323 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb" event={"ID":"94202efb-a6b4-4260-bd94-16473a214a5c","Type":"ContainerDied","Data":"27dfba87591a74416c61b8a66b8da0f83046881f0936fcc31aee8ff6072f7499"} Oct 30 09:37:58.565416 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:58.565348 4562 scope.go:122] "RemoveContainer" containerID="101f0b8ae7a931c275b34237e93dca0566e73cdd34f4889ea503aeac9911e3a0" Oct 30 09:37:58.582619 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:58.582598 4562 scope.go:122] "RemoveContainer" containerID="101f0b8ae7a931c275b34237e93dca0566e73cdd34f4889ea503aeac9911e3a0" Oct 30 09:37:58.582899 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:37:58.582881 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"101f0b8ae7a931c275b34237e93dca0566e73cdd34f4889ea503aeac9911e3a0\": container with ID starting with 101f0b8ae7a931c275b34237e93dca0566e73cdd34f4889ea503aeac9911e3a0 not found: ID does not exist" containerID="101f0b8ae7a931c275b34237e93dca0566e73cdd34f4889ea503aeac9911e3a0" Oct 30 09:37:58.582999 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:58.582905 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"101f0b8ae7a931c275b34237e93dca0566e73cdd34f4889ea503aeac9911e3a0"} err="failed to get container status \"101f0b8ae7a931c275b34237e93dca0566e73cdd34f4889ea503aeac9911e3a0\": rpc error: code = NotFound desc = could not find container \"101f0b8ae7a931c275b34237e93dca0566e73cdd34f4889ea503aeac9911e3a0\": container with ID starting with 101f0b8ae7a931c275b34237e93dca0566e73cdd34f4889ea503aeac9911e3a0 not found: ID does not exist" Oct 30 09:37:58.603038 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:37:58.603013 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:37:58.623619 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:58.623596 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb"] Oct 30 09:37:58.626801 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:58.626780 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-hhlrb"] Oct 30 09:37:59.876157 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:37:59.876124 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94202efb-a6b4-4260-bd94-16473a214a5c" path="/var/lib/kubelet/pods/94202efb-a6b4-4260-bd94-16473a214a5c/volumes" Oct 30 09:38:04.175024 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:04.174981 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-388653-56dfb7cb47-8th58"] Oct 30 09:38:04.175846 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:04.175824 4562 cpu_manager.go:414] "containerMap: removing container" podUID="94202efb-a6b4-4260-bd94-16473a214a5c" containerName="istio-proxy" Oct 30 09:38:04.175953 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:04.175846 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94202efb-a6b4-4260-bd94-16473a214a5c" containerName="istio-proxy" Oct 30 09:38:04.175953 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:04.175918 4562 cpu_manager.go:414] "containerMap: removing container" podUID="07038022-1dc8-4ac6-9bf6-07df45bb1da4" containerName="test-curl-11390-379eb6" Oct 30 09:38:04.175953 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:04.175928 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07038022-1dc8-4ac6-9bf6-07df45bb1da4" containerName="test-curl-11390-379eb6" Oct 30 09:38:04.176111 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:04.176096 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="07038022-1dc8-4ac6-9bf6-07df45bb1da4" containerName="test-curl-11390-379eb6" Oct 30 09:38:04.176167 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:04.176113 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="94202efb-a6b4-4260-bd94-16473a214a5c" containerName="istio-proxy" Oct 30 09:38:04.181202 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:04.181181 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-388653-56dfb7cb47-8th58" Oct 30 09:38:04.196765 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:04.196735 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-388653-56dfb7cb47-8th58"] Oct 30 09:38:04.205150 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:04.205121 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-388653-dockercfg-j2d7d\"" Oct 30 09:38:04.205150 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:04.205120 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-388653\"" Oct 30 09:38:04.337408 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:04.323478 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23911a6c-f976-496a-b75d-759bbc201b44-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-388653-56dfb7cb47-8th58\" (UID: \"23911a6c-f976-496a-b75d-759bbc201b44\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-388653-56dfb7cb47-8th58" Oct 30 09:38:04.342403 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:04.338220 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m6k7\" (UniqueName: \"kubernetes.io/projected/23911a6c-f976-496a-b75d-759bbc201b44-kube-api-access-9m6k7\") pod \"payload-pre-processing-e2e-shared-b-w0-388653-56dfb7cb47-8th58\" (UID: \"23911a6c-f976-496a-b75d-759bbc201b44\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-388653-56dfb7cb47-8th58" Oct 30 09:38:04.435496 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:04.416839 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-388653-576db78cbd-t8ncz"] Oct 30 09:38:04.449962 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:04.449895 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m6k7\" (UniqueName: \"kubernetes.io/projected/23911a6c-f976-496a-b75d-759bbc201b44-kube-api-access-9m6k7\") pod \"payload-pre-processing-e2e-shared-b-w0-388653-56dfb7cb47-8th58\" (UID: \"23911a6c-f976-496a-b75d-759bbc201b44\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-388653-56dfb7cb47-8th58" Oct 30 09:38:04.449962 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:04.449954 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23911a6c-f976-496a-b75d-759bbc201b44-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-388653-56dfb7cb47-8th58\" (UID: \"23911a6c-f976-496a-b75d-759bbc201b44\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-388653-56dfb7cb47-8th58" Oct 30 09:38:04.452125 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:04.451244 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23911a6c-f976-496a-b75d-759bbc201b44-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-388653-56dfb7cb47-8th58\" (UID: \"23911a6c-f976-496a-b75d-759bbc201b44\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-388653-56dfb7cb47-8th58" Oct 30 09:38:04.453556 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:04.453532 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-388653-576db78cbd-t8ncz"] Oct 30 09:38:04.455038 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:04.455017 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-388653-576db78cbd-t8ncz" Oct 30 09:38:04.486945 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:04.486911 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m6k7\" (UniqueName: \"kubernetes.io/projected/23911a6c-f976-496a-b75d-759bbc201b44-kube-api-access-9m6k7\") pod \"payload-pre-processing-e2e-shared-b-w0-388653-56dfb7cb47-8th58\" (UID: \"23911a6c-f976-496a-b75d-759bbc201b44\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-388653-56dfb7cb47-8th58" Oct 30 09:38:04.497917 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:04.497885 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-388653-56dfb7cb47-8th58" Oct 30 09:38:04.551892 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:04.551851 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f367c180-d0c3-4266-ba10-d982197e5bd8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-388653-576db78cbd-t8ncz\" (UID: \"f367c180-d0c3-4266-ba10-d982197e5bd8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-388653-576db78cbd-t8ncz" Oct 30 09:38:04.552073 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:04.551951 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t4n4\" (UniqueName: \"kubernetes.io/projected/f367c180-d0c3-4266-ba10-d982197e5bd8-kube-api-access-5t4n4\") pod \"payload-processing-e2e-shared-b-w0-388653-576db78cbd-t8ncz\" (UID: \"f367c180-d0c3-4266-ba10-d982197e5bd8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-388653-576db78cbd-t8ncz" Oct 30 09:38:04.652965 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:04.652928 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f367c180-d0c3-4266-ba10-d982197e5bd8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-388653-576db78cbd-t8ncz\" (UID: \"f367c180-d0c3-4266-ba10-d982197e5bd8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-388653-576db78cbd-t8ncz" Oct 30 09:38:04.653181 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:04.652990 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t4n4\" (UniqueName: \"kubernetes.io/projected/f367c180-d0c3-4266-ba10-d982197e5bd8-kube-api-access-5t4n4\") pod \"payload-processing-e2e-shared-b-w0-388653-576db78cbd-t8ncz\" (UID: \"f367c180-d0c3-4266-ba10-d982197e5bd8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-388653-576db78cbd-t8ncz" Oct 30 09:38:04.653691 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:04.653669 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f367c180-d0c3-4266-ba10-d982197e5bd8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-388653-576db78cbd-t8ncz\" (UID: \"f367c180-d0c3-4266-ba10-d982197e5bd8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-388653-576db78cbd-t8ncz" Oct 30 09:38:04.673312 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:04.673278 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t4n4\" (UniqueName: \"kubernetes.io/projected/f367c180-d0c3-4266-ba10-d982197e5bd8-kube-api-access-5t4n4\") pod \"payload-processing-e2e-shared-b-w0-388653-576db78cbd-t8ncz\" (UID: \"f367c180-d0c3-4266-ba10-d982197e5bd8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-388653-576db78cbd-t8ncz" Oct 30 09:38:04.795134 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:04.795109 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-388653-576db78cbd-t8ncz" Oct 30 09:38:04.799161 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:38:04.799130 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23911a6c_f976_496a_b75d_759bbc201b44.slice/crio-c8f1f707bb9a4eb63c17400649217c7dbda04645c33c78b87297071ad62f7fce WatchSource:0}: Error finding container c8f1f707bb9a4eb63c17400649217c7dbda04645c33c78b87297071ad62f7fce: Status 404 returned error can't find the container with id c8f1f707bb9a4eb63c17400649217c7dbda04645c33c78b87297071ad62f7fce Oct 30 09:38:04.805144 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:04.803308 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-388653-56dfb7cb47-8th58"] Oct 30 09:38:05.104807 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:38:05.104774 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf367c180_d0c3_4266_ba10_d982197e5bd8.slice/crio-bc1dfbf406d99b72894318beb1f3332d7e451f0b5184f93ee1b1718b4441019f WatchSource:0}: Error finding container bc1dfbf406d99b72894318beb1f3332d7e451f0b5184f93ee1b1718b4441019f: Status 404 returned error can't find the container with id bc1dfbf406d99b72894318beb1f3332d7e451f0b5184f93ee1b1718b4441019f Oct 30 09:38:05.108066 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:05.107961 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-388653-576db78cbd-t8ncz"] Oct 30 09:38:05.666501 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:05.666169 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-388653-576db78cbd-t8ncz" event={"ID":"f367c180-d0c3-4266-ba10-d982197e5bd8","Type":"ContainerStarted","Data":"ccdde6a697f8d77a059a68bddbb243cac687f59b6e2d3b284bac520075869962"} Oct 30 09:38:05.666970 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:05.666530 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-388653-576db78cbd-t8ncz" event={"ID":"f367c180-d0c3-4266-ba10-d982197e5bd8","Type":"ContainerStarted","Data":"bc1dfbf406d99b72894318beb1f3332d7e451f0b5184f93ee1b1718b4441019f"} Oct 30 09:38:05.666970 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:05.666553 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-388653-576db78cbd-t8ncz" Oct 30 09:38:05.668460 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:05.668362 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-388653-56dfb7cb47-8th58" event={"ID":"23911a6c-f976-496a-b75d-759bbc201b44","Type":"ContainerStarted","Data":"030fb21e03d183dbc0933944c3b21947a9e1db492d2614dffe0db0bf02a16ae8"} Oct 30 09:38:05.668460 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:05.668436 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-388653-56dfb7cb47-8th58" Oct 30 09:38:05.668460 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:05.668455 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-388653-56dfb7cb47-8th58" event={"ID":"23911a6c-f976-496a-b75d-759bbc201b44","Type":"ContainerStarted","Data":"c8f1f707bb9a4eb63c17400649217c7dbda04645c33c78b87297071ad62f7fce"} Oct 30 09:38:05.706173 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:05.706120 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-388653-576db78cbd-t8ncz" podStartSLOduration=1.706106631 podStartE2EDuration="1.706106631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:38:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:38:05.705202776 +0000 UTC m=+2645.966763765" watchObservedRunningTime="2026-10-30 09:38:05.706106631 +0000 UTC m=+2645.967667618" Oct 30 09:38:05.747211 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:05.747160 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-388653-56dfb7cb47-8th58" podStartSLOduration=1.747144102 podStartE2EDuration="1.747144102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:38:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:38:05.746225889 +0000 UTC m=+2646.007786859" watchObservedRunningTime="2026-10-30 09:38:05.747144102 +0000 UTC m=+2646.008705090" Oct 30 09:38:05.966562 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:05.966528 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7"] Oct 30 09:38:05.966771 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:05.966719 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7" podUID="6c176252-e0bb-44a8-93d9-4902b0afcb06" containerName="istio-proxy" containerID="cri-o://9bd6b4f47ac172758390f3f1e0c060358816211371a863c9bdfe060980f85c34" gracePeriod=30 Oct 30 09:38:05.979539 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:05.979514 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl"] Oct 30 09:38:05.979716 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:05.979695 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl" podUID="21455e45-a059-4b1a-81db-c4bba40c8442" containerName="istio-proxy" containerID="cri-o://1dc51e04524037602c25b3a193f6613599961ec7d4b01a772fb093a62f172415" gracePeriod=30 Oct 30 09:38:05.984513 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:05.984486 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh"] Oct 30 09:38:05.984513 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:05.984678 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh" podUID="7a745f00-5456-4a63-86f9-19c03a808c05" containerName="istio-proxy" containerID="cri-o://c58ae56cc4d8ae95c0a4519182658ff9f2c15c0321ecb20fc7dcd0aab8ba6cad" gracePeriod=30 Oct 30 09:38:05.989073 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:05.989051 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh"] Oct 30 09:38:05.989272 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:05.989248 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh" podUID="9491d089-4927-4270-82f2-fecd0f108311" containerName="istio-proxy" containerID="cri-o://5ad2bc72a04755052f3f98d927116b4d65e50d9a8d9f3d6f37017c84c6f7d18e" gracePeriod=30 Oct 30 09:38:10.603579 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:10.603544 4562 patch_prober.go:29] interesting pod/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 09:38:10.603956 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:10.603602 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7" podUID="6c176252-e0bb-44a8-93d9-4902b0afcb06" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 09:38:11.287954 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.287922 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl" Oct 30 09:38:11.310371 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.310341 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7" Oct 30 09:38:11.356465 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.356436 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh" Oct 30 09:38:11.360257 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.360229 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh" Oct 30 09:38:11.370629 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.370606 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21455e45-a059-4b1a-81db-c4bba40c8442-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21455e45-a059-4b1a-81db-c4bba40c8442-credential-socket\") pod \"21455e45-a059-4b1a-81db-c4bba40c8442\" (UID: \"21455e45-a059-4b1a-81db-c4bba40c8442\") " Oct 30 09:38:11.370734 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.370688 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c176252-e0bb-44a8-93d9-4902b0afcb06-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c176252-e0bb-44a8-93d9-4902b0afcb06-istio-envoy\") pod \"6c176252-e0bb-44a8-93d9-4902b0afcb06\" (UID: \"6c176252-e0bb-44a8-93d9-4902b0afcb06\") " Oct 30 09:38:11.370734 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.370714 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21455e45-a059-4b1a-81db-c4bba40c8442-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21455e45-a059-4b1a-81db-c4bba40c8442-workload-certs\") pod \"21455e45-a059-4b1a-81db-c4bba40c8442\" (UID: \"21455e45-a059-4b1a-81db-c4bba40c8442\") " Oct 30 09:38:11.370867 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.370755 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21455e45-a059-4b1a-81db-c4bba40c8442-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21455e45-a059-4b1a-81db-c4bba40c8442-workload-socket\") pod \"21455e45-a059-4b1a-81db-c4bba40c8442\" (UID: \"21455e45-a059-4b1a-81db-c4bba40c8442\") " Oct 30 09:38:11.370867 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.370815 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6c176252-e0bb-44a8-93d9-4902b0afcb06-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c176252-e0bb-44a8-93d9-4902b0afcb06-istio-podinfo\") pod \"6c176252-e0bb-44a8-93d9-4902b0afcb06\" (UID: \"6c176252-e0bb-44a8-93d9-4902b0afcb06\") " Oct 30 09:38:11.370980 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.370909 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/21455e45-a059-4b1a-81db-c4bba40c8442-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21455e45-a059-4b1a-81db-c4bba40c8442-istio-podinfo\") pod \"21455e45-a059-4b1a-81db-c4bba40c8442\" (UID: \"21455e45-a059-4b1a-81db-c4bba40c8442\") " Oct 30 09:38:11.370980 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.370925 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21455e45-a059-4b1a-81db-c4bba40c8442-credential-socket" pod "21455e45-a059-4b1a-81db-c4bba40c8442" (UID: "21455e45-a059-4b1a-81db-c4bba40c8442"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:38:11.371091 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.371044 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21455e45-a059-4b1a-81db-c4bba40c8442-workload-certs" pod "21455e45-a059-4b1a-81db-c4bba40c8442" (UID: "21455e45-a059-4b1a-81db-c4bba40c8442"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:38:11.371091 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.371066 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21455e45-a059-4b1a-81db-c4bba40c8442-workload-socket" pod "21455e45-a059-4b1a-81db-c4bba40c8442" (UID: "21455e45-a059-4b1a-81db-c4bba40c8442"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:38:11.371198 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.371132 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c176252-e0bb-44a8-93d9-4902b0afcb06-kube-api-access-cj8dp\" (UniqueName: \"kubernetes.io/projected/6c176252-e0bb-44a8-93d9-4902b0afcb06-kube-api-access-cj8dp\") pod \"6c176252-e0bb-44a8-93d9-4902b0afcb06\" (UID: \"6c176252-e0bb-44a8-93d9-4902b0afcb06\") " Oct 30 09:38:11.371588 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.371284 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21455e45-a059-4b1a-81db-c4bba40c8442-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21455e45-a059-4b1a-81db-c4bba40c8442-istio-envoy\") pod \"21455e45-a059-4b1a-81db-c4bba40c8442\" (UID: \"21455e45-a059-4b1a-81db-c4bba40c8442\") " Oct 30 09:38:11.371722 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.371688 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21455e45-a059-4b1a-81db-c4bba40c8442-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21455e45-a059-4b1a-81db-c4bba40c8442-istiod-ca-cert\") pod \"21455e45-a059-4b1a-81db-c4bba40c8442\" (UID: \"21455e45-a059-4b1a-81db-c4bba40c8442\") " Oct 30 09:38:11.372113 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.372089 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21455e45-a059-4b1a-81db-c4bba40c8442-istiod-ca-cert" pod "21455e45-a059-4b1a-81db-c4bba40c8442" (UID: "21455e45-a059-4b1a-81db-c4bba40c8442"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:38:11.372209 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.372191 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c176252-e0bb-44a8-93d9-4902b0afcb06-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c176252-e0bb-44a8-93d9-4902b0afcb06-workload-socket\") pod \"6c176252-e0bb-44a8-93d9-4902b0afcb06\" (UID: \"6c176252-e0bb-44a8-93d9-4902b0afcb06\") " Oct 30 09:38:11.372547 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.372522 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c176252-e0bb-44a8-93d9-4902b0afcb06-workload-socket" pod "6c176252-e0bb-44a8-93d9-4902b0afcb06" (UID: "6c176252-e0bb-44a8-93d9-4902b0afcb06"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:38:11.372635 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.372575 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21455e45-a059-4b1a-81db-c4bba40c8442-istio-token\" (UniqueName: \"kubernetes.io/projected/21455e45-a059-4b1a-81db-c4bba40c8442-istio-token\") pod \"21455e45-a059-4b1a-81db-c4bba40c8442\" (UID: \"21455e45-a059-4b1a-81db-c4bba40c8442\") " Oct 30 09:38:11.374406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.372943 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c176252-e0bb-44a8-93d9-4902b0afcb06-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c176252-e0bb-44a8-93d9-4902b0afcb06-workload-certs\") pod \"6c176252-e0bb-44a8-93d9-4902b0afcb06\" (UID: \"6c176252-e0bb-44a8-93d9-4902b0afcb06\") " Oct 30 09:38:11.374406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.373003 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21455e45-a059-4b1a-81db-c4bba40c8442-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21455e45-a059-4b1a-81db-c4bba40c8442-istio-data\") pod \"21455e45-a059-4b1a-81db-c4bba40c8442\" (UID: \"21455e45-a059-4b1a-81db-c4bba40c8442\") " Oct 30 09:38:11.374406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.373040 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c176252-e0bb-44a8-93d9-4902b0afcb06-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c176252-e0bb-44a8-93d9-4902b0afcb06-istiod-ca-cert\") pod \"6c176252-e0bb-44a8-93d9-4902b0afcb06\" (UID: \"6c176252-e0bb-44a8-93d9-4902b0afcb06\") " Oct 30 09:38:11.374406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.373078 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c176252-e0bb-44a8-93d9-4902b0afcb06-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c176252-e0bb-44a8-93d9-4902b0afcb06-credential-socket\") pod \"6c176252-e0bb-44a8-93d9-4902b0afcb06\" (UID: \"6c176252-e0bb-44a8-93d9-4902b0afcb06\") " Oct 30 09:38:11.374406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.373165 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21455e45-a059-4b1a-81db-c4bba40c8442-kube-api-access-h85mp\" (UniqueName: \"kubernetes.io/projected/21455e45-a059-4b1a-81db-c4bba40c8442-kube-api-access-h85mp\") pod \"21455e45-a059-4b1a-81db-c4bba40c8442\" (UID: \"21455e45-a059-4b1a-81db-c4bba40c8442\") " Oct 30 09:38:11.374406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.373195 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c176252-e0bb-44a8-93d9-4902b0afcb06-workload-certs" pod "6c176252-e0bb-44a8-93d9-4902b0afcb06" (UID: "6c176252-e0bb-44a8-93d9-4902b0afcb06"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:38:11.374406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.373235 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c176252-e0bb-44a8-93d9-4902b0afcb06-istio-token\" (UniqueName: \"kubernetes.io/projected/6c176252-e0bb-44a8-93d9-4902b0afcb06-istio-token\") pod \"6c176252-e0bb-44a8-93d9-4902b0afcb06\" (UID: \"6c176252-e0bb-44a8-93d9-4902b0afcb06\") " Oct 30 09:38:11.374406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.373252 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c176252-e0bb-44a8-93d9-4902b0afcb06-istio-envoy" pod "6c176252-e0bb-44a8-93d9-4902b0afcb06" (UID: "6c176252-e0bb-44a8-93d9-4902b0afcb06"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:38:11.374406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.373431 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21455e45-a059-4b1a-81db-c4bba40c8442-istio-data" pod "21455e45-a059-4b1a-81db-c4bba40c8442" (UID: "21455e45-a059-4b1a-81db-c4bba40c8442"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:38:11.374406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.373576 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c176252-e0bb-44a8-93d9-4902b0afcb06-credential-socket" pod "6c176252-e0bb-44a8-93d9-4902b0afcb06" (UID: "6c176252-e0bb-44a8-93d9-4902b0afcb06"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:38:11.374406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.373628 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c176252-e0bb-44a8-93d9-4902b0afcb06-istiod-ca-cert" pod "6c176252-e0bb-44a8-93d9-4902b0afcb06" (UID: "6c176252-e0bb-44a8-93d9-4902b0afcb06"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:38:11.374406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.373677 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c176252-e0bb-44a8-93d9-4902b0afcb06-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c176252-e0bb-44a8-93d9-4902b0afcb06-istio-data\") pod \"6c176252-e0bb-44a8-93d9-4902b0afcb06\" (UID: \"6c176252-e0bb-44a8-93d9-4902b0afcb06\") " Oct 30 09:38:11.374406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.374110 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c176252-e0bb-44a8-93d9-4902b0afcb06-istio-podinfo" pod "6c176252-e0bb-44a8-93d9-4902b0afcb06" (UID: "6c176252-e0bb-44a8-93d9-4902b0afcb06"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:38:11.374406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.374319 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c176252-e0bb-44a8-93d9-4902b0afcb06-istio-data" pod "6c176252-e0bb-44a8-93d9-4902b0afcb06" (UID: "6c176252-e0bb-44a8-93d9-4902b0afcb06"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:38:11.375256 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.374501 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21455e45-a059-4b1a-81db-c4bba40c8442-istio-envoy" pod "21455e45-a059-4b1a-81db-c4bba40c8442" (UID: "21455e45-a059-4b1a-81db-c4bba40c8442"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:38:11.375256 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.374547 4562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21455e45-a059-4b1a-81db-c4bba40c8442-istiod-ca-cert\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:38:11.375256 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.374565 4562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c176252-e0bb-44a8-93d9-4902b0afcb06-workload-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:38:11.375256 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.374579 4562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c176252-e0bb-44a8-93d9-4902b0afcb06-workload-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:38:11.375256 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.374593 4562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21455e45-a059-4b1a-81db-c4bba40c8442-istio-data\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:38:11.375256 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.374606 4562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c176252-e0bb-44a8-93d9-4902b0afcb06-istiod-ca-cert\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:38:11.375256 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.374617 4562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c176252-e0bb-44a8-93d9-4902b0afcb06-credential-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:38:11.375256 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.374629 4562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c176252-e0bb-44a8-93d9-4902b0afcb06-istio-data\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:38:11.375256 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.374641 4562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21455e45-a059-4b1a-81db-c4bba40c8442-credential-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:38:11.375256 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.374652 4562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c176252-e0bb-44a8-93d9-4902b0afcb06-istio-envoy\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:38:11.375256 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.374665 4562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21455e45-a059-4b1a-81db-c4bba40c8442-workload-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:38:11.375256 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.374677 4562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21455e45-a059-4b1a-81db-c4bba40c8442-workload-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:38:11.375256 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.374695 4562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c176252-e0bb-44a8-93d9-4902b0afcb06-istio-podinfo\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:38:11.375256 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.374876 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21455e45-a059-4b1a-81db-c4bba40c8442-istio-podinfo" pod "21455e45-a059-4b1a-81db-c4bba40c8442" (UID: "21455e45-a059-4b1a-81db-c4bba40c8442"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:38:11.375256 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.375077 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c176252-e0bb-44a8-93d9-4902b0afcb06-kube-api-access-cj8dp" pod "6c176252-e0bb-44a8-93d9-4902b0afcb06" (UID: "6c176252-e0bb-44a8-93d9-4902b0afcb06"). InnerVolumeSpecName "kube-api-access-cj8dp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:38:11.376091 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.375455 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21455e45-a059-4b1a-81db-c4bba40c8442-istio-token" pod "21455e45-a059-4b1a-81db-c4bba40c8442" (UID: "21455e45-a059-4b1a-81db-c4bba40c8442"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:38:11.376091 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.375468 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c176252-e0bb-44a8-93d9-4902b0afcb06-istio-token" pod "6c176252-e0bb-44a8-93d9-4902b0afcb06" (UID: "6c176252-e0bb-44a8-93d9-4902b0afcb06"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:38:11.376443 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.376421 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21455e45-a059-4b1a-81db-c4bba40c8442-kube-api-access-h85mp" pod "21455e45-a059-4b1a-81db-c4bba40c8442" (UID: "21455e45-a059-4b1a-81db-c4bba40c8442"). InnerVolumeSpecName "kube-api-access-h85mp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:38:11.476092 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.476057 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a745f00-5456-4a63-86f9-19c03a808c05-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a745f00-5456-4a63-86f9-19c03a808c05-istio-data\") pod \"7a745f00-5456-4a63-86f9-19c03a808c05\" (UID: \"7a745f00-5456-4a63-86f9-19c03a808c05\") " Oct 30 09:38:11.476263 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.476125 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9491d089-4927-4270-82f2-fecd0f108311-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9491d089-4927-4270-82f2-fecd0f108311-workload-socket\") pod \"9491d089-4927-4270-82f2-fecd0f108311\" (UID: \"9491d089-4927-4270-82f2-fecd0f108311\") " Oct 30 09:38:11.476263 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.476188 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7a745f00-5456-4a63-86f9-19c03a808c05-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a745f00-5456-4a63-86f9-19c03a808c05-istio-podinfo\") pod \"7a745f00-5456-4a63-86f9-19c03a808c05\" (UID: \"7a745f00-5456-4a63-86f9-19c03a808c05\") " Oct 30 09:38:11.476263 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.476225 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9491d089-4927-4270-82f2-fecd0f108311-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9491d089-4927-4270-82f2-fecd0f108311-workload-certs\") pod \"9491d089-4927-4270-82f2-fecd0f108311\" (UID: \"9491d089-4927-4270-82f2-fecd0f108311\") " Oct 30 09:38:11.476469 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.476301 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9491d089-4927-4270-82f2-fecd0f108311-kube-api-access-twwrq\" (UniqueName: \"kubernetes.io/projected/9491d089-4927-4270-82f2-fecd0f108311-kube-api-access-twwrq\") pod \"9491d089-4927-4270-82f2-fecd0f108311\" (UID: \"9491d089-4927-4270-82f2-fecd0f108311\") " Oct 30 09:38:11.476469 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.476329 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a745f00-5456-4a63-86f9-19c03a808c05-kube-api-access-292fn\" (UniqueName: \"kubernetes.io/projected/7a745f00-5456-4a63-86f9-19c03a808c05-kube-api-access-292fn\") pod \"7a745f00-5456-4a63-86f9-19c03a808c05\" (UID: \"7a745f00-5456-4a63-86f9-19c03a808c05\") " Oct 30 09:38:11.476469 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.476364 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9491d089-4927-4270-82f2-fecd0f108311-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9491d089-4927-4270-82f2-fecd0f108311-istio-data\") pod \"9491d089-4927-4270-82f2-fecd0f108311\" (UID: \"9491d089-4927-4270-82f2-fecd0f108311\") " Oct 30 09:38:11.476469 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.476439 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9491d089-4927-4270-82f2-fecd0f108311-workload-socket" pod "9491d089-4927-4270-82f2-fecd0f108311" (UID: "9491d089-4927-4270-82f2-fecd0f108311"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:38:11.476469 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.476446 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a745f00-5456-4a63-86f9-19c03a808c05-istio-data" pod "7a745f00-5456-4a63-86f9-19c03a808c05" (UID: "7a745f00-5456-4a63-86f9-19c03a808c05"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:38:11.476728 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.476486 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a745f00-5456-4a63-86f9-19c03a808c05-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a745f00-5456-4a63-86f9-19c03a808c05-istiod-ca-cert\") pod \"7a745f00-5456-4a63-86f9-19c03a808c05\" (UID: \"7a745f00-5456-4a63-86f9-19c03a808c05\") " Oct 30 09:38:11.476728 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.476540 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a745f00-5456-4a63-86f9-19c03a808c05-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a745f00-5456-4a63-86f9-19c03a808c05-workload-certs\") pod \"7a745f00-5456-4a63-86f9-19c03a808c05\" (UID: \"7a745f00-5456-4a63-86f9-19c03a808c05\") " Oct 30 09:38:11.476728 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.476623 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9491d089-4927-4270-82f2-fecd0f108311-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9491d089-4927-4270-82f2-fecd0f108311-credential-socket\") pod \"9491d089-4927-4270-82f2-fecd0f108311\" (UID: \"9491d089-4927-4270-82f2-fecd0f108311\") " Oct 30 09:38:11.476728 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.476707 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9491d089-4927-4270-82f2-fecd0f108311-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9491d089-4927-4270-82f2-fecd0f108311-istio-envoy\") pod \"9491d089-4927-4270-82f2-fecd0f108311\" (UID: \"9491d089-4927-4270-82f2-fecd0f108311\") " Oct 30 09:38:11.476728 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.476720 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9491d089-4927-4270-82f2-fecd0f108311-workload-certs" pod "9491d089-4927-4270-82f2-fecd0f108311" (UID: "9491d089-4927-4270-82f2-fecd0f108311"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:38:11.477015 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.476749 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a745f00-5456-4a63-86f9-19c03a808c05-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a745f00-5456-4a63-86f9-19c03a808c05-workload-socket\") pod \"7a745f00-5456-4a63-86f9-19c03a808c05\" (UID: \"7a745f00-5456-4a63-86f9-19c03a808c05\") " Oct 30 09:38:11.477015 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.476801 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9491d089-4927-4270-82f2-fecd0f108311-istio-token\" (UniqueName: \"kubernetes.io/projected/9491d089-4927-4270-82f2-fecd0f108311-istio-token\") pod \"9491d089-4927-4270-82f2-fecd0f108311\" (UID: \"9491d089-4927-4270-82f2-fecd0f108311\") " Oct 30 09:38:11.477015 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.476834 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9491d089-4927-4270-82f2-fecd0f108311-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9491d089-4927-4270-82f2-fecd0f108311-istiod-ca-cert\") pod \"9491d089-4927-4270-82f2-fecd0f108311\" (UID: \"9491d089-4927-4270-82f2-fecd0f108311\") " Oct 30 09:38:11.477015 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.476866 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a745f00-5456-4a63-86f9-19c03a808c05-istio-token\" (UniqueName: \"kubernetes.io/projected/7a745f00-5456-4a63-86f9-19c03a808c05-istio-token\") pod \"7a745f00-5456-4a63-86f9-19c03a808c05\" (UID: \"7a745f00-5456-4a63-86f9-19c03a808c05\") " Oct 30 09:38:11.477015 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.476866 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9491d089-4927-4270-82f2-fecd0f108311-istio-data" pod "9491d089-4927-4270-82f2-fecd0f108311" (UID: "9491d089-4927-4270-82f2-fecd0f108311"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:38:11.477015 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.476901 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a745f00-5456-4a63-86f9-19c03a808c05-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a745f00-5456-4a63-86f9-19c03a808c05-istio-envoy\") pod \"7a745f00-5456-4a63-86f9-19c03a808c05\" (UID: \"7a745f00-5456-4a63-86f9-19c03a808c05\") " Oct 30 09:38:11.477015 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.476941 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9491d089-4927-4270-82f2-fecd0f108311-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9491d089-4927-4270-82f2-fecd0f108311-istio-podinfo\") pod \"9491d089-4927-4270-82f2-fecd0f108311\" (UID: \"9491d089-4927-4270-82f2-fecd0f108311\") " Oct 30 09:38:11.477015 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.476951 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a745f00-5456-4a63-86f9-19c03a808c05-workload-certs" pod "7a745f00-5456-4a63-86f9-19c03a808c05" (UID: "7a745f00-5456-4a63-86f9-19c03a808c05"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:38:11.477015 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.476977 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a745f00-5456-4a63-86f9-19c03a808c05-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a745f00-5456-4a63-86f9-19c03a808c05-credential-socket\") pod \"7a745f00-5456-4a63-86f9-19c03a808c05\" (UID: \"7a745f00-5456-4a63-86f9-19c03a808c05\") " Oct 30 09:38:11.477829 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.477089 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9491d089-4927-4270-82f2-fecd0f108311-credential-socket" pod "9491d089-4927-4270-82f2-fecd0f108311" (UID: "9491d089-4927-4270-82f2-fecd0f108311"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:38:11.477829 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.477193 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a745f00-5456-4a63-86f9-19c03a808c05-istiod-ca-cert" pod "7a745f00-5456-4a63-86f9-19c03a808c05" (UID: "7a745f00-5456-4a63-86f9-19c03a808c05"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:38:11.477829 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.477286 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9491d089-4927-4270-82f2-fecd0f108311-istiod-ca-cert" pod "9491d089-4927-4270-82f2-fecd0f108311" (UID: "9491d089-4927-4270-82f2-fecd0f108311"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:38:11.477829 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.477368 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a745f00-5456-4a63-86f9-19c03a808c05-workload-socket" pod "7a745f00-5456-4a63-86f9-19c03a808c05" (UID: "7a745f00-5456-4a63-86f9-19c03a808c05"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:38:11.477829 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.477731 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a745f00-5456-4a63-86f9-19c03a808c05-credential-socket" pod "7a745f00-5456-4a63-86f9-19c03a808c05" (UID: "7a745f00-5456-4a63-86f9-19c03a808c05"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:38:11.478297 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.478273 4562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21455e45-a059-4b1a-81db-c4bba40c8442-istio-podinfo\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:38:11.478430 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.478301 4562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a745f00-5456-4a63-86f9-19c03a808c05-workload-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:38:11.478430 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.478318 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj8dp\" (UniqueName: \"kubernetes.io/projected/6c176252-e0bb-44a8-93d9-4902b0afcb06-kube-api-access-cj8dp\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:38:11.478430 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.478332 4562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9491d089-4927-4270-82f2-fecd0f108311-istiod-ca-cert\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:38:11.478430 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.478370 4562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21455e45-a059-4b1a-81db-c4bba40c8442-istio-envoy\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:38:11.478430 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.478410 4562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a745f00-5456-4a63-86f9-19c03a808c05-credential-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:38:11.478430 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.478426 4562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21455e45-a059-4b1a-81db-c4bba40c8442-istio-token\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:38:11.479235 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.478440 4562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a745f00-5456-4a63-86f9-19c03a808c05-istio-data\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:38:11.479235 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.478452 4562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9491d089-4927-4270-82f2-fecd0f108311-workload-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:38:11.479235 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.478466 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h85mp\" (UniqueName: \"kubernetes.io/projected/21455e45-a059-4b1a-81db-c4bba40c8442-kube-api-access-h85mp\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:38:11.479235 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.478479 4562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9491d089-4927-4270-82f2-fecd0f108311-workload-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:38:11.479235 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.478493 4562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c176252-e0bb-44a8-93d9-4902b0afcb06-istio-token\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:38:11.479235 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.478505 4562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9491d089-4927-4270-82f2-fecd0f108311-istio-data\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:38:11.479235 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.478518 4562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a745f00-5456-4a63-86f9-19c03a808c05-istiod-ca-cert\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:38:11.479235 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.478530 4562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a745f00-5456-4a63-86f9-19c03a808c05-workload-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:38:11.479235 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.478543 4562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9491d089-4927-4270-82f2-fecd0f108311-credential-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:38:11.485017 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.484993 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9491d089-4927-4270-82f2-fecd0f108311-istio-podinfo" pod "9491d089-4927-4270-82f2-fecd0f108311" (UID: "9491d089-4927-4270-82f2-fecd0f108311"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:38:11.487332 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.487302 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a745f00-5456-4a63-86f9-19c03a808c05-istio-token" pod "7a745f00-5456-4a63-86f9-19c03a808c05" (UID: "7a745f00-5456-4a63-86f9-19c03a808c05"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:38:11.487782 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.487719 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a745f00-5456-4a63-86f9-19c03a808c05-istio-envoy" pod "7a745f00-5456-4a63-86f9-19c03a808c05" (UID: "7a745f00-5456-4a63-86f9-19c03a808c05"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:38:11.487782 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.487757 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9491d089-4927-4270-82f2-fecd0f108311-istio-envoy" pod "9491d089-4927-4270-82f2-fecd0f108311" (UID: "9491d089-4927-4270-82f2-fecd0f108311"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:38:11.487969 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.487878 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9491d089-4927-4270-82f2-fecd0f108311-kube-api-access-twwrq" pod "9491d089-4927-4270-82f2-fecd0f108311" (UID: "9491d089-4927-4270-82f2-fecd0f108311"). InnerVolumeSpecName "kube-api-access-twwrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:38:11.487969 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.487899 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a745f00-5456-4a63-86f9-19c03a808c05-kube-api-access-292fn" pod "7a745f00-5456-4a63-86f9-19c03a808c05" (UID: "7a745f00-5456-4a63-86f9-19c03a808c05"). InnerVolumeSpecName "kube-api-access-292fn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:38:11.488083 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.487990 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7a745f00-5456-4a63-86f9-19c03a808c05-istio-podinfo" pod "7a745f00-5456-4a63-86f9-19c03a808c05" (UID: "7a745f00-5456-4a63-86f9-19c03a808c05"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:38:11.489621 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.489595 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9491d089-4927-4270-82f2-fecd0f108311-istio-token" pod "9491d089-4927-4270-82f2-fecd0f108311" (UID: "9491d089-4927-4270-82f2-fecd0f108311"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:38:11.580520 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.580432 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twwrq\" (UniqueName: \"kubernetes.io/projected/9491d089-4927-4270-82f2-fecd0f108311-kube-api-access-twwrq\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:38:11.580520 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.580471 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-292fn\" (UniqueName: \"kubernetes.io/projected/7a745f00-5456-4a63-86f9-19c03a808c05-kube-api-access-292fn\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:38:11.580520 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.580484 4562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9491d089-4927-4270-82f2-fecd0f108311-istio-envoy\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:38:11.580520 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.580498 4562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9491d089-4927-4270-82f2-fecd0f108311-istio-token\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:38:11.580520 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.580509 4562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a745f00-5456-4a63-86f9-19c03a808c05-istio-token\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:38:11.580520 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.580521 4562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a745f00-5456-4a63-86f9-19c03a808c05-istio-envoy\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:38:11.580816 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.580534 4562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9491d089-4927-4270-82f2-fecd0f108311-istio-podinfo\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:38:11.580816 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.580545 4562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a745f00-5456-4a63-86f9-19c03a808c05-istio-podinfo\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:38:11.710565 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.710527 4562 generic.go:356] "Generic (PLEG): container finished" podID="9491d089-4927-4270-82f2-fecd0f108311" containerID="5ad2bc72a04755052f3f98d927116b4d65e50d9a8d9f3d6f37017c84c6f7d18e" exitCode=0 Oct 30 09:38:11.711000 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.710585 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh" Oct 30 09:38:11.711000 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.710616 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh" event={"ID":"9491d089-4927-4270-82f2-fecd0f108311","Type":"ContainerDied","Data":"5ad2bc72a04755052f3f98d927116b4d65e50d9a8d9f3d6f37017c84c6f7d18e"} Oct 30 09:38:11.711000 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.710706 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh" event={"ID":"9491d089-4927-4270-82f2-fecd0f108311","Type":"ContainerDied","Data":"6f6ea8eb248692526d9a787f5dd1c35c4bf96a6a7bf631b76ad8552cc682e27f"} Oct 30 09:38:11.711000 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.710742 4562 scope.go:122] "RemoveContainer" containerID="5ad2bc72a04755052f3f98d927116b4d65e50d9a8d9f3d6f37017c84c6f7d18e" Oct 30 09:38:11.712054 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.712009 4562 generic.go:356] "Generic (PLEG): container finished" podID="21455e45-a059-4b1a-81db-c4bba40c8442" containerID="1dc51e04524037602c25b3a193f6613599961ec7d4b01a772fb093a62f172415" exitCode=0 Oct 30 09:38:11.712145 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.712062 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl" Oct 30 09:38:11.712145 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.712095 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl" event={"ID":"21455e45-a059-4b1a-81db-c4bba40c8442","Type":"ContainerDied","Data":"1dc51e04524037602c25b3a193f6613599961ec7d4b01a772fb093a62f172415"} Oct 30 09:38:11.712267 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.712144 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl" event={"ID":"21455e45-a059-4b1a-81db-c4bba40c8442","Type":"ContainerDied","Data":"2e1d4b80b56b633a63951d5c86fc5d85fb41cbfd5a62d6fe034b46e65862d1e3"} Oct 30 09:38:11.713497 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.713476 4562 generic.go:356] "Generic (PLEG): container finished" podID="6c176252-e0bb-44a8-93d9-4902b0afcb06" containerID="9bd6b4f47ac172758390f3f1e0c060358816211371a863c9bdfe060980f85c34" exitCode=0 Oct 30 09:38:11.713589 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.713503 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7" event={"ID":"6c176252-e0bb-44a8-93d9-4902b0afcb06","Type":"ContainerDied","Data":"9bd6b4f47ac172758390f3f1e0c060358816211371a863c9bdfe060980f85c34"} Oct 30 09:38:11.713589 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.713525 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7" Oct 30 09:38:11.713589 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.713550 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7" event={"ID":"6c176252-e0bb-44a8-93d9-4902b0afcb06","Type":"ContainerDied","Data":"25b6ab0be2f6f577ea3af3998035dbdfacdffe4514a4335f91a618c00b3c4289"} Oct 30 09:38:11.715114 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.715093 4562 generic.go:356] "Generic (PLEG): container finished" podID="7a745f00-5456-4a63-86f9-19c03a808c05" containerID="c58ae56cc4d8ae95c0a4519182658ff9f2c15c0321ecb20fc7dcd0aab8ba6cad" exitCode=0 Oct 30 09:38:11.715236 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.715132 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh" Oct 30 09:38:11.715236 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.715166 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh" event={"ID":"7a745f00-5456-4a63-86f9-19c03a808c05","Type":"ContainerDied","Data":"c58ae56cc4d8ae95c0a4519182658ff9f2c15c0321ecb20fc7dcd0aab8ba6cad"} Oct 30 09:38:11.715236 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.715224 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh" event={"ID":"7a745f00-5456-4a63-86f9-19c03a808c05","Type":"ContainerDied","Data":"6f902f3daafe3b5ecf7414602636267e9daeb73a21a7da4a473987cad0317a80"} Oct 30 09:38:11.724350 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.724332 4562 scope.go:122] "RemoveContainer" containerID="5ad2bc72a04755052f3f98d927116b4d65e50d9a8d9f3d6f37017c84c6f7d18e" Oct 30 09:38:11.724723 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:38:11.724701 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ad2bc72a04755052f3f98d927116b4d65e50d9a8d9f3d6f37017c84c6f7d18e\": container with ID starting with 5ad2bc72a04755052f3f98d927116b4d65e50d9a8d9f3d6f37017c84c6f7d18e not found: ID does not exist" containerID="5ad2bc72a04755052f3f98d927116b4d65e50d9a8d9f3d6f37017c84c6f7d18e" Oct 30 09:38:11.724815 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.724731 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ad2bc72a04755052f3f98d927116b4d65e50d9a8d9f3d6f37017c84c6f7d18e"} err="failed to get container status \"5ad2bc72a04755052f3f98d927116b4d65e50d9a8d9f3d6f37017c84c6f7d18e\": rpc error: code = NotFound desc = could not find container \"5ad2bc72a04755052f3f98d927116b4d65e50d9a8d9f3d6f37017c84c6f7d18e\": container with ID starting with 5ad2bc72a04755052f3f98d927116b4d65e50d9a8d9f3d6f37017c84c6f7d18e not found: ID does not exist" Oct 30 09:38:11.724815 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.724752 4562 scope.go:122] "RemoveContainer" containerID="1dc51e04524037602c25b3a193f6613599961ec7d4b01a772fb093a62f172415" Oct 30 09:38:11.735698 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.735680 4562 scope.go:122] "RemoveContainer" containerID="1dc51e04524037602c25b3a193f6613599961ec7d4b01a772fb093a62f172415" Oct 30 09:38:11.735995 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:38:11.735979 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dc51e04524037602c25b3a193f6613599961ec7d4b01a772fb093a62f172415\": container with ID starting with 1dc51e04524037602c25b3a193f6613599961ec7d4b01a772fb093a62f172415 not found: ID does not exist" containerID="1dc51e04524037602c25b3a193f6613599961ec7d4b01a772fb093a62f172415" Oct 30 09:38:11.736039 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.736002 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dc51e04524037602c25b3a193f6613599961ec7d4b01a772fb093a62f172415"} err="failed to get container status \"1dc51e04524037602c25b3a193f6613599961ec7d4b01a772fb093a62f172415\": rpc error: code = NotFound desc = could not find container \"1dc51e04524037602c25b3a193f6613599961ec7d4b01a772fb093a62f172415\": container with ID starting with 1dc51e04524037602c25b3a193f6613599961ec7d4b01a772fb093a62f172415 not found: ID does not exist" Oct 30 09:38:11.736039 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.736020 4562 scope.go:122] "RemoveContainer" containerID="9bd6b4f47ac172758390f3f1e0c060358816211371a863c9bdfe060980f85c34" Oct 30 09:38:11.745728 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.745711 4562 scope.go:122] "RemoveContainer" containerID="9bd6b4f47ac172758390f3f1e0c060358816211371a863c9bdfe060980f85c34" Oct 30 09:38:11.745984 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:38:11.745969 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bd6b4f47ac172758390f3f1e0c060358816211371a863c9bdfe060980f85c34\": container with ID starting with 9bd6b4f47ac172758390f3f1e0c060358816211371a863c9bdfe060980f85c34 not found: ID does not exist" containerID="9bd6b4f47ac172758390f3f1e0c060358816211371a863c9bdfe060980f85c34" Oct 30 09:38:11.746046 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.745990 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bd6b4f47ac172758390f3f1e0c060358816211371a863c9bdfe060980f85c34"} err="failed to get container status \"9bd6b4f47ac172758390f3f1e0c060358816211371a863c9bdfe060980f85c34\": rpc error: code = NotFound desc = could not find container \"9bd6b4f47ac172758390f3f1e0c060358816211371a863c9bdfe060980f85c34\": container with ID starting with 9bd6b4f47ac172758390f3f1e0c060358816211371a863c9bdfe060980f85c34 not found: ID does not exist" Oct 30 09:38:11.746046 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.746006 4562 scope.go:122] "RemoveContainer" containerID="c58ae56cc4d8ae95c0a4519182658ff9f2c15c0321ecb20fc7dcd0aab8ba6cad" Oct 30 09:38:11.755670 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.755654 4562 scope.go:122] "RemoveContainer" containerID="c58ae56cc4d8ae95c0a4519182658ff9f2c15c0321ecb20fc7dcd0aab8ba6cad" Oct 30 09:38:11.755944 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:38:11.755927 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c58ae56cc4d8ae95c0a4519182658ff9f2c15c0321ecb20fc7dcd0aab8ba6cad\": container with ID starting with c58ae56cc4d8ae95c0a4519182658ff9f2c15c0321ecb20fc7dcd0aab8ba6cad not found: ID does not exist" containerID="c58ae56cc4d8ae95c0a4519182658ff9f2c15c0321ecb20fc7dcd0aab8ba6cad" Oct 30 09:38:11.756016 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.755954 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c58ae56cc4d8ae95c0a4519182658ff9f2c15c0321ecb20fc7dcd0aab8ba6cad"} err="failed to get container status \"c58ae56cc4d8ae95c0a4519182658ff9f2c15c0321ecb20fc7dcd0aab8ba6cad\": rpc error: code = NotFound desc = could not find container \"c58ae56cc4d8ae95c0a4519182658ff9f2c15c0321ecb20fc7dcd0aab8ba6cad\": container with ID starting with c58ae56cc4d8ae95c0a4519182658ff9f2c15c0321ecb20fc7dcd0aab8ba6cad not found: ID does not exist" Oct 30 09:38:11.814443 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.814411 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7"] Oct 30 09:38:11.827308 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.827279 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-2psl7"] Oct 30 09:38:11.863715 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.863659 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl"] Oct 30 09:38:11.876653 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.876630 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c176252-e0bb-44a8-93d9-4902b0afcb06" path="/var/lib/kubelet/pods/6c176252-e0bb-44a8-93d9-4902b0afcb06/volumes" Oct 30 09:38:11.876960 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.876947 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-dsxpl"] Oct 30 09:38:11.901288 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.901265 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh"] Oct 30 09:38:11.904471 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.904451 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-hrcjh"] Oct 30 09:38:11.940336 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.940310 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh"] Oct 30 09:38:11.944609 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:11.944588 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c5447d-openshift-default-5bf65d8949-9bcnh"] Oct 30 09:38:13.876179 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:13.876104 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21455e45-a059-4b1a-81db-c4bba40c8442" path="/var/lib/kubelet/pods/21455e45-a059-4b1a-81db-c4bba40c8442/volumes" Oct 30 09:38:13.876674 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:13.876658 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a745f00-5456-4a63-86f9-19c03a808c05" path="/var/lib/kubelet/pods/7a745f00-5456-4a63-86f9-19c03a808c05/volumes" Oct 30 09:38:13.877186 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:13.877172 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9491d089-4927-4270-82f2-fecd0f108311" path="/var/lib/kubelet/pods/9491d089-4927-4270-82f2-fecd0f108311/volumes" Oct 30 09:38:14.720647 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:14.720614 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-aa2fdf/auth-test-model-aa2fdf-kserve-699ff98945-zq96m"] Oct 30 09:38:16.677837 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:16.677806 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-388653-576db78cbd-t8ncz" Oct 30 09:38:16.678229 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:16.678040 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-388653-56dfb7cb47-8th58" Oct 30 09:38:16.709576 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:16.709543 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h"] Oct 30 09:38:16.710236 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:16.710217 4562 cpu_manager.go:414] "containerMap: removing container" podUID="6c176252-e0bb-44a8-93d9-4902b0afcb06" containerName="istio-proxy" Oct 30 09:38:16.710344 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:16.710237 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c176252-e0bb-44a8-93d9-4902b0afcb06" containerName="istio-proxy" Oct 30 09:38:16.710344 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:16.710271 4562 cpu_manager.go:414] "containerMap: removing container" podUID="7a745f00-5456-4a63-86f9-19c03a808c05" containerName="istio-proxy" Oct 30 09:38:16.710344 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:16.710279 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a745f00-5456-4a63-86f9-19c03a808c05" containerName="istio-proxy" Oct 30 09:38:16.710344 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:16.710291 4562 cpu_manager.go:414] "containerMap: removing container" podUID="21455e45-a059-4b1a-81db-c4bba40c8442" containerName="istio-proxy" Oct 30 09:38:16.710344 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:16.710299 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21455e45-a059-4b1a-81db-c4bba40c8442" containerName="istio-proxy" Oct 30 09:38:16.710344 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:16.710310 4562 cpu_manager.go:414] "containerMap: removing container" podUID="9491d089-4927-4270-82f2-fecd0f108311" containerName="istio-proxy" Oct 30 09:38:16.710344 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:16.710317 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9491d089-4927-4270-82f2-fecd0f108311" containerName="istio-proxy" Oct 30 09:38:16.710884 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:16.710498 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a745f00-5456-4a63-86f9-19c03a808c05" containerName="istio-proxy" Oct 30 09:38:16.710884 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:16.710514 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="21455e45-a059-4b1a-81db-c4bba40c8442" containerName="istio-proxy" Oct 30 09:38:16.710884 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:16.710528 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="9491d089-4927-4270-82f2-fecd0f108311" containerName="istio-proxy" Oct 30 09:38:16.710884 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:16.710545 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c176252-e0bb-44a8-93d9-4902b0afcb06" containerName="istio-proxy" Oct 30 09:38:16.716160 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:16.716124 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h" Oct 30 09:38:16.738837 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:16.738811 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-881f83-openshift-default-dockercfg-s64lm\"" Oct 30 09:38:16.740120 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:16.740078 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h"] Oct 30 09:38:16.865094 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:16.865054 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aea403b0-3184-48ef-8e1a-b1f91b56d703-workload-certs\") pod \"e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h\" (UID: \"aea403b0-3184-48ef-8e1a-b1f91b56d703\") " pod="openshift-ingress/e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h" Oct 30 09:38:16.865263 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:16.865094 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aea403b0-3184-48ef-8e1a-b1f91b56d703-istio-data\") pod \"e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h\" (UID: \"aea403b0-3184-48ef-8e1a-b1f91b56d703\") " pod="openshift-ingress/e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h" Oct 30 09:38:16.865305 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:16.865269 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aea403b0-3184-48ef-8e1a-b1f91b56d703-credential-socket\") pod \"e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h\" (UID: \"aea403b0-3184-48ef-8e1a-b1f91b56d703\") " pod="openshift-ingress/e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h" Oct 30 09:38:16.865305 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:16.865299 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aea403b0-3184-48ef-8e1a-b1f91b56d703-istio-token\") pod \"e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h\" (UID: \"aea403b0-3184-48ef-8e1a-b1f91b56d703\") " pod="openshift-ingress/e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h" Oct 30 09:38:16.865413 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:16.865397 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbvb2\" (UniqueName: \"kubernetes.io/projected/aea403b0-3184-48ef-8e1a-b1f91b56d703-kube-api-access-sbvb2\") pod \"e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h\" (UID: \"aea403b0-3184-48ef-8e1a-b1f91b56d703\") " pod="openshift-ingress/e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h" Oct 30 09:38:16.865514 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:16.865464 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aea403b0-3184-48ef-8e1a-b1f91b56d703-istio-envoy\") pod \"e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h\" (UID: \"aea403b0-3184-48ef-8e1a-b1f91b56d703\") " pod="openshift-ingress/e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h" Oct 30 09:38:16.865577 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:16.865554 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aea403b0-3184-48ef-8e1a-b1f91b56d703-istiod-ca-cert\") pod \"e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h\" (UID: \"aea403b0-3184-48ef-8e1a-b1f91b56d703\") " pod="openshift-ingress/e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h" Oct 30 09:38:16.865692 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:16.865665 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aea403b0-3184-48ef-8e1a-b1f91b56d703-workload-socket\") pod \"e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h\" (UID: \"aea403b0-3184-48ef-8e1a-b1f91b56d703\") " pod="openshift-ingress/e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h" Oct 30 09:38:16.865770 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:16.865750 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aea403b0-3184-48ef-8e1a-b1f91b56d703-istio-podinfo\") pod \"e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h\" (UID: \"aea403b0-3184-48ef-8e1a-b1f91b56d703\") " pod="openshift-ingress/e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h" Oct 30 09:38:16.976411 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:16.975240 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aea403b0-3184-48ef-8e1a-b1f91b56d703-istio-envoy\") pod \"e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h\" (UID: \"aea403b0-3184-48ef-8e1a-b1f91b56d703\") " pod="openshift-ingress/e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h" Oct 30 09:38:16.976411 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:16.975326 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aea403b0-3184-48ef-8e1a-b1f91b56d703-istiod-ca-cert\") pod \"e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h\" (UID: \"aea403b0-3184-48ef-8e1a-b1f91b56d703\") " pod="openshift-ingress/e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h" Oct 30 09:38:16.976411 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:16.975399 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aea403b0-3184-48ef-8e1a-b1f91b56d703-workload-socket\") pod \"e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h\" (UID: \"aea403b0-3184-48ef-8e1a-b1f91b56d703\") " pod="openshift-ingress/e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h" Oct 30 09:38:16.976411 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:16.975458 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aea403b0-3184-48ef-8e1a-b1f91b56d703-istio-podinfo\") pod \"e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h\" (UID: \"aea403b0-3184-48ef-8e1a-b1f91b56d703\") " pod="openshift-ingress/e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h" Oct 30 09:38:16.976411 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:16.975547 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aea403b0-3184-48ef-8e1a-b1f91b56d703-workload-certs\") pod \"e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h\" (UID: \"aea403b0-3184-48ef-8e1a-b1f91b56d703\") " pod="openshift-ingress/e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h" Oct 30 09:38:16.976411 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:16.975576 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aea403b0-3184-48ef-8e1a-b1f91b56d703-istio-data\") pod \"e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h\" (UID: \"aea403b0-3184-48ef-8e1a-b1f91b56d703\") " pod="openshift-ingress/e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h" Oct 30 09:38:16.976411 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:16.975713 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aea403b0-3184-48ef-8e1a-b1f91b56d703-credential-socket\") pod \"e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h\" (UID: \"aea403b0-3184-48ef-8e1a-b1f91b56d703\") " pod="openshift-ingress/e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h" Oct 30 09:38:16.976411 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:16.975740 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aea403b0-3184-48ef-8e1a-b1f91b56d703-istio-token\") pod \"e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h\" (UID: \"aea403b0-3184-48ef-8e1a-b1f91b56d703\") " pod="openshift-ingress/e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h" Oct 30 09:38:16.976411 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:16.975804 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbvb2\" (UniqueName: \"kubernetes.io/projected/aea403b0-3184-48ef-8e1a-b1f91b56d703-kube-api-access-sbvb2\") pod \"e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h\" (UID: \"aea403b0-3184-48ef-8e1a-b1f91b56d703\") " pod="openshift-ingress/e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h" Oct 30 09:38:16.977040 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:16.976939 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aea403b0-3184-48ef-8e1a-b1f91b56d703-credential-socket\") pod \"e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h\" (UID: \"aea403b0-3184-48ef-8e1a-b1f91b56d703\") " pod="openshift-ingress/e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h" Oct 30 09:38:17.001431 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:16.977192 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aea403b0-3184-48ef-8e1a-b1f91b56d703-istio-data\") pod \"e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h\" (UID: \"aea403b0-3184-48ef-8e1a-b1f91b56d703\") " pod="openshift-ingress/e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h" Oct 30 09:38:17.001431 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:16.977466 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aea403b0-3184-48ef-8e1a-b1f91b56d703-workload-socket\") pod \"e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h\" (UID: \"aea403b0-3184-48ef-8e1a-b1f91b56d703\") " pod="openshift-ingress/e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h" Oct 30 09:38:17.001431 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:16.988795 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aea403b0-3184-48ef-8e1a-b1f91b56d703-istiod-ca-cert\") pod \"e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h\" (UID: \"aea403b0-3184-48ef-8e1a-b1f91b56d703\") " pod="openshift-ingress/e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h" Oct 30 09:38:17.001431 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:16.989084 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aea403b0-3184-48ef-8e1a-b1f91b56d703-workload-certs\") pod \"e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h\" (UID: \"aea403b0-3184-48ef-8e1a-b1f91b56d703\") " pod="openshift-ingress/e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h" Oct 30 09:38:17.001431 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:16.991061 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aea403b0-3184-48ef-8e1a-b1f91b56d703-istio-envoy\") pod \"e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h\" (UID: \"aea403b0-3184-48ef-8e1a-b1f91b56d703\") " pod="openshift-ingress/e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h" Oct 30 09:38:17.022436 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:17.018403 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aea403b0-3184-48ef-8e1a-b1f91b56d703-istio-podinfo\") pod \"e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h\" (UID: \"aea403b0-3184-48ef-8e1a-b1f91b56d703\") " pod="openshift-ingress/e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h" Oct 30 09:38:17.048354 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:17.043175 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbvb2\" (UniqueName: \"kubernetes.io/projected/aea403b0-3184-48ef-8e1a-b1f91b56d703-kube-api-access-sbvb2\") pod \"e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h\" (UID: \"aea403b0-3184-48ef-8e1a-b1f91b56d703\") " pod="openshift-ingress/e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h" Oct 30 09:38:17.048354 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:17.046320 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aea403b0-3184-48ef-8e1a-b1f91b56d703-istio-token\") pod \"e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h\" (UID: \"aea403b0-3184-48ef-8e1a-b1f91b56d703\") " pod="openshift-ingress/e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h" Oct 30 09:38:17.058201 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:17.057104 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h" Oct 30 09:38:17.475409 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:17.471877 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h"] Oct 30 09:38:17.476453 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:38:17.476413 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaea403b0_3184_48ef_8e1a_b1f91b56d703.slice/crio-fe53ffb9bc03c96b43aadc258ba56121f56ccd776bf9bcbdb3191c5ace67f13e WatchSource:0}: Error finding container fe53ffb9bc03c96b43aadc258ba56121f56ccd776bf9bcbdb3191c5ace67f13e: Status 404 returned error can't find the container with id fe53ffb9bc03c96b43aadc258ba56121f56ccd776bf9bcbdb3191c5ace67f13e Oct 30 09:38:17.480170 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:17.480136 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:38:17.480255 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:17.480208 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:38:17.480255 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:17.480238 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:38:17.765528 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:17.765435 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h" event={"ID":"aea403b0-3184-48ef-8e1a-b1f91b56d703","Type":"ContainerStarted","Data":"2b2612ef43ed39a643510f2582516db4e90bb96a434e3624eaaf66443eb33d37"} Oct 30 09:38:17.765528 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:17.765521 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h" event={"ID":"aea403b0-3184-48ef-8e1a-b1f91b56d703","Type":"ContainerStarted","Data":"fe53ffb9bc03c96b43aadc258ba56121f56ccd776bf9bcbdb3191c5ace67f13e"} Oct 30 09:38:17.811299 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:17.811220 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h" podStartSLOduration=1.811201337 podStartE2EDuration="1.811201337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:38:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:38:17.808456933 +0000 UTC m=+2658.070017921" watchObservedRunningTime="2026-10-30 09:38:17.811201337 +0000 UTC m=+2658.072762326" Oct 30 09:38:18.058372 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:18.058292 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h" Oct 30 09:38:18.064356 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:18.064322 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h" Oct 30 09:38:18.772010 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:18.771979 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h" Oct 30 09:38:18.773066 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:18.773047 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h" Oct 30 09:38:22.020541 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:22.020511 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-881f83-c579c5979-vj4zx"] Oct 30 09:38:22.024825 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:22.024805 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-881f83-c579c5979-vj4zx" Oct 30 09:38:22.054277 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:22.052595 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-881f83\"" Oct 30 09:38:22.143364 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:22.115506 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-881f83-c579c5979-vj4zx"] Oct 30 09:38:22.172441 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:22.165835 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjpb2\" (UniqueName: \"kubernetes.io/projected/59e94911-c9dc-4fbd-b813-e22c0bc51fed-kube-api-access-qjpb2\") pod \"payload-pre-processing-e2e-extmod-a-881f83-c579c5979-vj4zx\" (UID: \"59e94911-c9dc-4fbd-b813-e22c0bc51fed\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-881f83-c579c5979-vj4zx" Oct 30 09:38:22.172441 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:22.165942 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59e94911-c9dc-4fbd-b813-e22c0bc51fed-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-881f83-c579c5979-vj4zx\" (UID: \"59e94911-c9dc-4fbd-b813-e22c0bc51fed\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-881f83-c579c5979-vj4zx" Oct 30 09:38:22.257405 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:22.256983 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-881f83-dockercfg-pjp5h\"" Oct 30 09:38:22.291535 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:22.291443 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjpb2\" (UniqueName: \"kubernetes.io/projected/59e94911-c9dc-4fbd-b813-e22c0bc51fed-kube-api-access-qjpb2\") pod \"payload-pre-processing-e2e-extmod-a-881f83-c579c5979-vj4zx\" (UID: \"59e94911-c9dc-4fbd-b813-e22c0bc51fed\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-881f83-c579c5979-vj4zx" Oct 30 09:38:22.294478 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:22.294436 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59e94911-c9dc-4fbd-b813-e22c0bc51fed-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-881f83-c579c5979-vj4zx\" (UID: \"59e94911-c9dc-4fbd-b813-e22c0bc51fed\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-881f83-c579c5979-vj4zx" Oct 30 09:38:22.297621 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:22.296707 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59e94911-c9dc-4fbd-b813-e22c0bc51fed-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-881f83-c579c5979-vj4zx\" (UID: \"59e94911-c9dc-4fbd-b813-e22c0bc51fed\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-881f83-c579c5979-vj4zx" Oct 30 09:38:22.299470 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:22.299439 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-881f83-6b6777d984-fh7q9"] Oct 30 09:38:22.309765 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:22.305269 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-881f83-6b6777d984-fh7q9"] Oct 30 09:38:22.309765 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:22.305376 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-881f83-6b6777d984-fh7q9" Oct 30 09:38:22.316065 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:22.316033 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjpb2\" (UniqueName: \"kubernetes.io/projected/59e94911-c9dc-4fbd-b813-e22c0bc51fed-kube-api-access-qjpb2\") pod \"payload-pre-processing-e2e-extmod-a-881f83-c579c5979-vj4zx\" (UID: \"59e94911-c9dc-4fbd-b813-e22c0bc51fed\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-881f83-c579c5979-vj4zx" Oct 30 09:38:22.339085 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:22.339065 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-881f83-c579c5979-vj4zx" Oct 30 09:38:22.398045 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:22.397908 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtp99\" (UniqueName: \"kubernetes.io/projected/169915d0-2153-46a4-a1af-a9d1670af675-kube-api-access-qtp99\") pod \"payload-processing-e2e-extmod-a-881f83-6b6777d984-fh7q9\" (UID: \"169915d0-2153-46a4-a1af-a9d1670af675\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-881f83-6b6777d984-fh7q9" Oct 30 09:38:22.398127 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:22.398100 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/169915d0-2153-46a4-a1af-a9d1670af675-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-881f83-6b6777d984-fh7q9\" (UID: \"169915d0-2153-46a4-a1af-a9d1670af675\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-881f83-6b6777d984-fh7q9" Oct 30 09:38:22.502115 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:22.500908 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/169915d0-2153-46a4-a1af-a9d1670af675-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-881f83-6b6777d984-fh7q9\" (UID: \"169915d0-2153-46a4-a1af-a9d1670af675\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-881f83-6b6777d984-fh7q9" Oct 30 09:38:22.502115 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:22.501117 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtp99\" (UniqueName: \"kubernetes.io/projected/169915d0-2153-46a4-a1af-a9d1670af675-kube-api-access-qtp99\") pod \"payload-processing-e2e-extmod-a-881f83-6b6777d984-fh7q9\" (UID: \"169915d0-2153-46a4-a1af-a9d1670af675\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-881f83-6b6777d984-fh7q9" Oct 30 09:38:22.509092 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:22.503605 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/169915d0-2153-46a4-a1af-a9d1670af675-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-881f83-6b6777d984-fh7q9\" (UID: \"169915d0-2153-46a4-a1af-a9d1670af675\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-881f83-6b6777d984-fh7q9" Oct 30 09:38:22.547453 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:22.545261 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtp99\" (UniqueName: \"kubernetes.io/projected/169915d0-2153-46a4-a1af-a9d1670af675-kube-api-access-qtp99\") pod \"payload-processing-e2e-extmod-a-881f83-6b6777d984-fh7q9\" (UID: \"169915d0-2153-46a4-a1af-a9d1670af675\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-881f83-6b6777d984-fh7q9" Oct 30 09:38:22.635141 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:38:22.635109 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59e94911_c9dc_4fbd_b813_e22c0bc51fed.slice/crio-a88d51354e50f37782c23d614fd20ec0dd9b6c0d4071ed684ca2d34e078f49ad WatchSource:0}: Error finding container a88d51354e50f37782c23d614fd20ec0dd9b6c0d4071ed684ca2d34e078f49ad: Status 404 returned error can't find the container with id a88d51354e50f37782c23d614fd20ec0dd9b6c0d4071ed684ca2d34e078f49ad Oct 30 09:38:22.636257 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:22.636227 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-881f83-6b6777d984-fh7q9" Oct 30 09:38:22.638800 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:22.638774 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-881f83-c579c5979-vj4zx"] Oct 30 09:38:22.800497 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:22.800337 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-881f83-c579c5979-vj4zx" event={"ID":"59e94911-c9dc-4fbd-b813-e22c0bc51fed","Type":"ContainerStarted","Data":"9ce672cce42e4e9ff3d8e8979f2a7e3dc14320e27c07728e245401d27e1efa76"} Oct 30 09:38:22.800497 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:22.800491 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-881f83-c579c5979-vj4zx" Oct 30 09:38:22.800713 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:22.800520 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-881f83-c579c5979-vj4zx" event={"ID":"59e94911-c9dc-4fbd-b813-e22c0bc51fed","Type":"ContainerStarted","Data":"a88d51354e50f37782c23d614fd20ec0dd9b6c0d4071ed684ca2d34e078f49ad"} Oct 30 09:38:22.842107 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:22.842048 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-881f83-c579c5979-vj4zx" podStartSLOduration=1.84202928 podStartE2EDuration="1.84202928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:38:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:38:22.838638978 +0000 UTC m=+2663.100199966" watchObservedRunningTime="2026-10-30 09:38:22.84202928 +0000 UTC m=+2663.103590269" Oct 30 09:38:22.943349 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:38:22.943322 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod169915d0_2153_46a4_a1af_a9d1670af675.slice/crio-177af5e069cacc05a7eca6e290c570491f42cb69acae997e1c8d800a8e5bee99 WatchSource:0}: Error finding container 177af5e069cacc05a7eca6e290c570491f42cb69acae997e1c8d800a8e5bee99: Status 404 returned error can't find the container with id 177af5e069cacc05a7eca6e290c570491f42cb69acae997e1c8d800a8e5bee99 Oct 30 09:38:22.945141 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:22.945117 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-881f83-6b6777d984-fh7q9"] Oct 30 09:38:23.810367 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:23.810329 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-881f83-6b6777d984-fh7q9" event={"ID":"169915d0-2153-46a4-a1af-a9d1670af675","Type":"ContainerStarted","Data":"b1ca48ea1c2ce89477c54e3598289c25838533a471f2196df231db129e4caef4"} Oct 30 09:38:23.810832 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:23.810405 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-881f83-6b6777d984-fh7q9" event={"ID":"169915d0-2153-46a4-a1af-a9d1670af675","Type":"ContainerStarted","Data":"177af5e069cacc05a7eca6e290c570491f42cb69acae997e1c8d800a8e5bee99"} Oct 30 09:38:23.810832 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:23.810548 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-881f83-6b6777d984-fh7q9" Oct 30 09:38:23.852479 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:23.852412 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-881f83-6b6777d984-fh7q9" podStartSLOduration=1.852393441 podStartE2EDuration="1.852393441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:38:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:38:23.850527959 +0000 UTC m=+2664.112088986" watchObservedRunningTime="2026-10-30 09:38:23.852393441 +0000 UTC m=+2664.113954420" Oct 30 09:38:24.308530 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:24.308493 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd"] Oct 30 09:38:24.308735 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:24.308687 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd" podUID="802df8bf-9c24-428b-9d87-96f58054ad68" containerName="istio-proxy" containerID="cri-o://599ca44ae43f5ca43d5317ff007cafd1348723aedf14a528035d83a8badbd353" gracePeriod=30 Oct 30 09:38:28.597542 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:38:28.597504 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:38:29.068071 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:29.068035 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-388653-6f7cf8fd57-686xr"] Oct 30 09:38:29.072919 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:29.072891 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-388653-6f7cf8fd57-686xr" Oct 30 09:38:29.091251 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:29.091223 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-388653-6f7cf8fd57-686xr"] Oct 30 09:38:29.112292 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:29.107779 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-388653-serving-cert\"" Oct 30 09:38:29.271908 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:29.271875 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14eb89e0-861c-41a9-a245-e97adfa39f25-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-388653-6f7cf8fd57-686xr\" (UID: \"14eb89e0-861c-41a9-a245-e97adfa39f25\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-388653-6f7cf8fd57-686xr" Oct 30 09:38:29.272084 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:29.272037 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14eb89e0-861c-41a9-a245-e97adfa39f25-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-388653-6f7cf8fd57-686xr\" (UID: \"14eb89e0-861c-41a9-a245-e97adfa39f25\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-388653-6f7cf8fd57-686xr" Oct 30 09:38:29.272173 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:29.272158 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvz7g\" (UniqueName: \"kubernetes.io/projected/14eb89e0-861c-41a9-a245-e97adfa39f25-kube-api-access-zvz7g\") pod \"maas-api-e2e-shared-b-w0-388653-6f7cf8fd57-686xr\" (UID: \"14eb89e0-861c-41a9-a245-e97adfa39f25\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-388653-6f7cf8fd57-686xr" Oct 30 09:38:29.272239 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:29.272225 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14eb89e0-861c-41a9-a245-e97adfa39f25-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-388653-6f7cf8fd57-686xr\" (UID: \"14eb89e0-861c-41a9-a245-e97adfa39f25\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-388653-6f7cf8fd57-686xr" Oct 30 09:38:29.373329 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:29.373298 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvz7g\" (UniqueName: \"kubernetes.io/projected/14eb89e0-861c-41a9-a245-e97adfa39f25-kube-api-access-zvz7g\") pod \"maas-api-e2e-shared-b-w0-388653-6f7cf8fd57-686xr\" (UID: \"14eb89e0-861c-41a9-a245-e97adfa39f25\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-388653-6f7cf8fd57-686xr" Oct 30 09:38:29.373483 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:29.373364 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14eb89e0-861c-41a9-a245-e97adfa39f25-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-388653-6f7cf8fd57-686xr\" (UID: \"14eb89e0-861c-41a9-a245-e97adfa39f25\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-388653-6f7cf8fd57-686xr" Oct 30 09:38:29.373530 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:29.373498 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14eb89e0-861c-41a9-a245-e97adfa39f25-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-388653-6f7cf8fd57-686xr\" (UID: \"14eb89e0-861c-41a9-a245-e97adfa39f25\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-388653-6f7cf8fd57-686xr" Oct 30 09:38:29.373637 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:29.373618 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14eb89e0-861c-41a9-a245-e97adfa39f25-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-388653-6f7cf8fd57-686xr\" (UID: \"14eb89e0-861c-41a9-a245-e97adfa39f25\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-388653-6f7cf8fd57-686xr" Oct 30 09:38:29.374177 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:29.374140 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14eb89e0-861c-41a9-a245-e97adfa39f25-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-388653-6f7cf8fd57-686xr\" (UID: \"14eb89e0-861c-41a9-a245-e97adfa39f25\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-388653-6f7cf8fd57-686xr" Oct 30 09:38:29.375853 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:29.375832 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14eb89e0-861c-41a9-a245-e97adfa39f25-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-388653-6f7cf8fd57-686xr\" (UID: \"14eb89e0-861c-41a9-a245-e97adfa39f25\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-388653-6f7cf8fd57-686xr" Oct 30 09:38:29.375943 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:29.375871 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14eb89e0-861c-41a9-a245-e97adfa39f25-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-388653-6f7cf8fd57-686xr\" (UID: \"14eb89e0-861c-41a9-a245-e97adfa39f25\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-388653-6f7cf8fd57-686xr" Oct 30 09:38:29.393077 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:29.393047 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvz7g\" (UniqueName: \"kubernetes.io/projected/14eb89e0-861c-41a9-a245-e97adfa39f25-kube-api-access-zvz7g\") pod \"maas-api-e2e-shared-b-w0-388653-6f7cf8fd57-686xr\" (UID: \"14eb89e0-861c-41a9-a245-e97adfa39f25\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-388653-6f7cf8fd57-686xr" Oct 30 09:38:29.423274 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:29.414136 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-388653-6f7cf8fd57-686xr" Oct 30 09:38:29.715451 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:29.714962 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk"] Oct 30 09:38:29.720959 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:29.720938 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk" Oct 30 09:38:29.743337 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:29.743302 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-aa2fdf/auth-test-model-aa2fdf-kserve-699ff98945-zq96m" podUID="00d5c9cf-9dae-425f-8189-870602061926" containerName="main" containerID="cri-o://d80fd8b11c285fd2e2dd627d81427dc3443373a63b90b31cdca5ae043c917ff5" gracePeriod=45 Oct 30 09:38:29.751218 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:29.751184 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk"] Oct 30 09:38:29.752054 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:29.752025 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-0f6f96-openshift-default-dockercfg-n7dh9\"" Oct 30 09:38:29.784072 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:29.783766 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-aa2fdf/auth-test-model-aa2fdf-kserve-699ff98945-zq96m" podUID="00d5c9cf-9dae-425f-8189-870602061926" containerName="main" probeResult="failure" output="Get \"https://10.133.0.190:8000/ready\": dial tcp 10.133.0.190:8000: connect: connection refused" Oct 30 09:38:29.787495 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:29.786855 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54cc6629-e1ca-4af7-9455-80e3b4d097f6-istiod-ca-cert\") pod \"e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk\" (UID: \"54cc6629-e1ca-4af7-9455-80e3b4d097f6\") " pod="openshift-ingress/e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk" Oct 30 09:38:29.787495 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:29.786927 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54cc6629-e1ca-4af7-9455-80e3b4d097f6-workload-certs\") pod \"e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk\" (UID: \"54cc6629-e1ca-4af7-9455-80e3b4d097f6\") " pod="openshift-ingress/e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk" Oct 30 09:38:29.787495 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:29.786969 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54cc6629-e1ca-4af7-9455-80e3b4d097f6-istio-envoy\") pod \"e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk\" (UID: \"54cc6629-e1ca-4af7-9455-80e3b4d097f6\") " pod="openshift-ingress/e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk" Oct 30 09:38:29.787495 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:29.787083 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54cc6629-e1ca-4af7-9455-80e3b4d097f6-istio-token\") pod \"e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk\" (UID: \"54cc6629-e1ca-4af7-9455-80e3b4d097f6\") " pod="openshift-ingress/e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk" Oct 30 09:38:29.787495 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:29.787154 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54cc6629-e1ca-4af7-9455-80e3b4d097f6-workload-socket\") pod \"e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk\" (UID: \"54cc6629-e1ca-4af7-9455-80e3b4d097f6\") " pod="openshift-ingress/e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk" Oct 30 09:38:29.787495 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:29.787191 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54cc6629-e1ca-4af7-9455-80e3b4d097f6-istio-data\") pod \"e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk\" (UID: \"54cc6629-e1ca-4af7-9455-80e3b4d097f6\") " pod="openshift-ingress/e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk" Oct 30 09:38:29.787495 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:29.787264 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54cc6629-e1ca-4af7-9455-80e3b4d097f6-istio-podinfo\") pod \"e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk\" (UID: \"54cc6629-e1ca-4af7-9455-80e3b4d097f6\") " pod="openshift-ingress/e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk" Oct 30 09:38:29.787495 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:29.787297 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg5sd\" (UniqueName: \"kubernetes.io/projected/54cc6629-e1ca-4af7-9455-80e3b4d097f6-kube-api-access-kg5sd\") pod \"e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk\" (UID: \"54cc6629-e1ca-4af7-9455-80e3b4d097f6\") " pod="openshift-ingress/e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk" Oct 30 09:38:29.787495 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:29.787328 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54cc6629-e1ca-4af7-9455-80e3b4d097f6-credential-socket\") pod \"e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk\" (UID: \"54cc6629-e1ca-4af7-9455-80e3b4d097f6\") " pod="openshift-ingress/e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk" Oct 30 09:38:29.903231 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:29.903183 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54cc6629-e1ca-4af7-9455-80e3b4d097f6-istiod-ca-cert\") pod \"e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk\" (UID: \"54cc6629-e1ca-4af7-9455-80e3b4d097f6\") " pod="openshift-ingress/e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk" Oct 30 09:38:29.903413 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:29.903277 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54cc6629-e1ca-4af7-9455-80e3b4d097f6-workload-certs\") pod \"e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk\" (UID: \"54cc6629-e1ca-4af7-9455-80e3b4d097f6\") " pod="openshift-ingress/e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk" Oct 30 09:38:29.903413 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:29.903322 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54cc6629-e1ca-4af7-9455-80e3b4d097f6-istio-envoy\") pod \"e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk\" (UID: \"54cc6629-e1ca-4af7-9455-80e3b4d097f6\") " pod="openshift-ingress/e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk" Oct 30 09:38:29.903585 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:29.903461 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54cc6629-e1ca-4af7-9455-80e3b4d097f6-istio-token\") pod \"e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk\" (UID: \"54cc6629-e1ca-4af7-9455-80e3b4d097f6\") " pod="openshift-ingress/e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk" Oct 30 09:38:29.903585 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:29.903534 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54cc6629-e1ca-4af7-9455-80e3b4d097f6-workload-socket\") pod \"e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk\" (UID: \"54cc6629-e1ca-4af7-9455-80e3b4d097f6\") " pod="openshift-ingress/e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk" Oct 30 09:38:29.903585 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:29.903568 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54cc6629-e1ca-4af7-9455-80e3b4d097f6-istio-data\") pod \"e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk\" (UID: \"54cc6629-e1ca-4af7-9455-80e3b4d097f6\") " pod="openshift-ingress/e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk" Oct 30 09:38:29.903813 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:29.903678 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54cc6629-e1ca-4af7-9455-80e3b4d097f6-istio-podinfo\") pod \"e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk\" (UID: \"54cc6629-e1ca-4af7-9455-80e3b4d097f6\") " pod="openshift-ingress/e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk" Oct 30 09:38:29.903813 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:29.903736 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg5sd\" (UniqueName: \"kubernetes.io/projected/54cc6629-e1ca-4af7-9455-80e3b4d097f6-kube-api-access-kg5sd\") pod \"e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk\" (UID: \"54cc6629-e1ca-4af7-9455-80e3b4d097f6\") " pod="openshift-ingress/e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk" Oct 30 09:38:29.903813 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:29.903780 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54cc6629-e1ca-4af7-9455-80e3b4d097f6-credential-socket\") pod \"e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk\" (UID: \"54cc6629-e1ca-4af7-9455-80e3b4d097f6\") " pod="openshift-ingress/e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk" Oct 30 09:38:29.904278 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:29.904251 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54cc6629-e1ca-4af7-9455-80e3b4d097f6-credential-socket\") pod \"e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk\" (UID: \"54cc6629-e1ca-4af7-9455-80e3b4d097f6\") " pod="openshift-ingress/e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk" Oct 30 09:38:29.911646 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:29.911612 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54cc6629-e1ca-4af7-9455-80e3b4d097f6-workload-certs\") pod \"e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk\" (UID: \"54cc6629-e1ca-4af7-9455-80e3b4d097f6\") " pod="openshift-ingress/e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk" Oct 30 09:38:29.918152 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:29.918119 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54cc6629-e1ca-4af7-9455-80e3b4d097f6-istio-data\") pod \"e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk\" (UID: \"54cc6629-e1ca-4af7-9455-80e3b4d097f6\") " pod="openshift-ingress/e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk" Oct 30 09:38:29.920457 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:29.919488 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54cc6629-e1ca-4af7-9455-80e3b4d097f6-istio-envoy\") pod \"e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk\" (UID: \"54cc6629-e1ca-4af7-9455-80e3b4d097f6\") " pod="openshift-ingress/e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk" Oct 30 09:38:29.920457 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:29.919764 4562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 09:38:29.925321 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:29.925266 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54cc6629-e1ca-4af7-9455-80e3b4d097f6-istiod-ca-cert\") pod \"e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk\" (UID: \"54cc6629-e1ca-4af7-9455-80e3b4d097f6\") " pod="openshift-ingress/e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk" Oct 30 09:38:29.926471 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:29.926447 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54cc6629-e1ca-4af7-9455-80e3b4d097f6-workload-socket\") pod \"e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk\" (UID: \"54cc6629-e1ca-4af7-9455-80e3b4d097f6\") " pod="openshift-ingress/e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk" Oct 30 09:38:29.953319 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:29.953179 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg5sd\" (UniqueName: \"kubernetes.io/projected/54cc6629-e1ca-4af7-9455-80e3b4d097f6-kube-api-access-kg5sd\") pod \"e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk\" (UID: \"54cc6629-e1ca-4af7-9455-80e3b4d097f6\") " pod="openshift-ingress/e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk" Oct 30 09:38:29.982722 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:29.982691 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd" Oct 30 09:38:29.989260 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:29.989217 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54cc6629-e1ca-4af7-9455-80e3b4d097f6-istio-token\") pod \"e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk\" (UID: \"54cc6629-e1ca-4af7-9455-80e3b4d097f6\") " pod="openshift-ingress/e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk" Oct 30 09:38:29.990445 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:29.990374 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54cc6629-e1ca-4af7-9455-80e3b4d097f6-istio-podinfo\") pod \"e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk\" (UID: \"54cc6629-e1ca-4af7-9455-80e3b4d097f6\") " pod="openshift-ingress/e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk" Oct 30 09:38:29.992796 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:29.992774 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-388653-6f7cf8fd57-686xr"] Oct 30 09:38:29.993414 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:29.993377 4562 generic.go:356] "Generic (PLEG): container finished" podID="00d5c9cf-9dae-425f-8189-870602061926" containerID="d80fd8b11c285fd2e2dd627d81427dc3443373a63b90b31cdca5ae043c917ff5" exitCode=0 Oct 30 09:38:29.993476 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:29.993424 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-aa2fdf/auth-test-model-aa2fdf-kserve-699ff98945-zq96m" event={"ID":"00d5c9cf-9dae-425f-8189-870602061926","Type":"ContainerDied","Data":"d80fd8b11c285fd2e2dd627d81427dc3443373a63b90b31cdca5ae043c917ff5"} Oct 30 09:38:29.994762 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:29.994736 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-388653-6f7cf8fd57-686xr" event={"ID":"14eb89e0-861c-41a9-a245-e97adfa39f25","Type":"ContainerStarted","Data":"5e01a245c66f377d95cea92de964f5d17b6b51e4a7926034353c1588de0a9369"} Oct 30 09:38:29.996141 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:29.996123 4562 generic.go:356] "Generic (PLEG): container finished" podID="802df8bf-9c24-428b-9d87-96f58054ad68" containerID="599ca44ae43f5ca43d5317ff007cafd1348723aedf14a528035d83a8badbd353" exitCode=0 Oct 30 09:38:29.996208 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:29.996174 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd" Oct 30 09:38:29.996289 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:29.996272 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd" event={"ID":"802df8bf-9c24-428b-9d87-96f58054ad68","Type":"ContainerDied","Data":"599ca44ae43f5ca43d5317ff007cafd1348723aedf14a528035d83a8badbd353"} Oct 30 09:38:29.996393 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:29.996369 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd" event={"ID":"802df8bf-9c24-428b-9d87-96f58054ad68","Type":"ContainerDied","Data":"5f12043fee692da94e520a04b806d89d9791667669f61f47bd5c6742a6ade7c4"} Oct 30 09:38:29.996480 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:29.996439 4562 scope.go:122] "RemoveContainer" containerID="599ca44ae43f5ca43d5317ff007cafd1348723aedf14a528035d83a8badbd353" Oct 30 09:38:30.008025 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:30.008008 4562 scope.go:122] "RemoveContainer" containerID="599ca44ae43f5ca43d5317ff007cafd1348723aedf14a528035d83a8badbd353" Oct 30 09:38:30.008317 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:38:30.008292 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"599ca44ae43f5ca43d5317ff007cafd1348723aedf14a528035d83a8badbd353\": container with ID starting with 599ca44ae43f5ca43d5317ff007cafd1348723aedf14a528035d83a8badbd353 not found: ID does not exist" containerID="599ca44ae43f5ca43d5317ff007cafd1348723aedf14a528035d83a8badbd353" Oct 30 09:38:30.008429 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:30.008322 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"599ca44ae43f5ca43d5317ff007cafd1348723aedf14a528035d83a8badbd353"} err="failed to get container status \"599ca44ae43f5ca43d5317ff007cafd1348723aedf14a528035d83a8badbd353\": rpc error: code = NotFound desc = could not find container \"599ca44ae43f5ca43d5317ff007cafd1348723aedf14a528035d83a8badbd353\": container with ID starting with 599ca44ae43f5ca43d5317ff007cafd1348723aedf14a528035d83a8badbd353 not found: ID does not exist" Oct 30 09:38:30.057735 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:30.057698 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk" Oct 30 09:38:30.109931 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:30.108529 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/802df8bf-9c24-428b-9d87-96f58054ad68-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/802df8bf-9c24-428b-9d87-96f58054ad68-credential-socket\") pod \"802df8bf-9c24-428b-9d87-96f58054ad68\" (UID: \"802df8bf-9c24-428b-9d87-96f58054ad68\") " Oct 30 09:38:30.109931 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:30.108595 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/802df8bf-9c24-428b-9d87-96f58054ad68-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/802df8bf-9c24-428b-9d87-96f58054ad68-istio-envoy\") pod \"802df8bf-9c24-428b-9d87-96f58054ad68\" (UID: \"802df8bf-9c24-428b-9d87-96f58054ad68\") " Oct 30 09:38:30.109931 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:30.108703 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/802df8bf-9c24-428b-9d87-96f58054ad68-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/802df8bf-9c24-428b-9d87-96f58054ad68-istiod-ca-cert\") pod \"802df8bf-9c24-428b-9d87-96f58054ad68\" (UID: \"802df8bf-9c24-428b-9d87-96f58054ad68\") " Oct 30 09:38:30.109931 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:30.108743 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/802df8bf-9c24-428b-9d87-96f58054ad68-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/802df8bf-9c24-428b-9d87-96f58054ad68-istio-podinfo\") pod \"802df8bf-9c24-428b-9d87-96f58054ad68\" (UID: \"802df8bf-9c24-428b-9d87-96f58054ad68\") " Oct 30 09:38:30.109931 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:30.108768 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/802df8bf-9c24-428b-9d87-96f58054ad68-istio-token\" (UniqueName: \"kubernetes.io/projected/802df8bf-9c24-428b-9d87-96f58054ad68-istio-token\") pod \"802df8bf-9c24-428b-9d87-96f58054ad68\" (UID: \"802df8bf-9c24-428b-9d87-96f58054ad68\") " Oct 30 09:38:30.109931 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:30.108824 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/802df8bf-9c24-428b-9d87-96f58054ad68-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/802df8bf-9c24-428b-9d87-96f58054ad68-workload-certs\") pod \"802df8bf-9c24-428b-9d87-96f58054ad68\" (UID: \"802df8bf-9c24-428b-9d87-96f58054ad68\") " Oct 30 09:38:30.109931 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:30.108910 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/802df8bf-9c24-428b-9d87-96f58054ad68-kube-api-access-htb2w\" (UniqueName: \"kubernetes.io/projected/802df8bf-9c24-428b-9d87-96f58054ad68-kube-api-access-htb2w\") pod \"802df8bf-9c24-428b-9d87-96f58054ad68\" (UID: \"802df8bf-9c24-428b-9d87-96f58054ad68\") " Oct 30 09:38:30.109931 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:30.108951 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/802df8bf-9c24-428b-9d87-96f58054ad68-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/802df8bf-9c24-428b-9d87-96f58054ad68-istio-data\") pod \"802df8bf-9c24-428b-9d87-96f58054ad68\" (UID: \"802df8bf-9c24-428b-9d87-96f58054ad68\") " Oct 30 09:38:30.109931 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:30.108978 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/802df8bf-9c24-428b-9d87-96f58054ad68-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/802df8bf-9c24-428b-9d87-96f58054ad68-workload-socket\") pod \"802df8bf-9c24-428b-9d87-96f58054ad68\" (UID: \"802df8bf-9c24-428b-9d87-96f58054ad68\") " Oct 30 09:38:30.112935 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:30.111606 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/802df8bf-9c24-428b-9d87-96f58054ad68-workload-certs" pod "802df8bf-9c24-428b-9d87-96f58054ad68" (UID: "802df8bf-9c24-428b-9d87-96f58054ad68"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:38:30.112935 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:30.112715 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/802df8bf-9c24-428b-9d87-96f58054ad68-istiod-ca-cert" pod "802df8bf-9c24-428b-9d87-96f58054ad68" (UID: "802df8bf-9c24-428b-9d87-96f58054ad68"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:38:30.112935 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:30.112755 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/802df8bf-9c24-428b-9d87-96f58054ad68-istio-data" pod "802df8bf-9c24-428b-9d87-96f58054ad68" (UID: "802df8bf-9c24-428b-9d87-96f58054ad68"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:38:30.112935 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:30.112794 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/802df8bf-9c24-428b-9d87-96f58054ad68-workload-socket" pod "802df8bf-9c24-428b-9d87-96f58054ad68" (UID: "802df8bf-9c24-428b-9d87-96f58054ad68"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:38:30.114314 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:30.113197 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/802df8bf-9c24-428b-9d87-96f58054ad68-credential-socket" pod "802df8bf-9c24-428b-9d87-96f58054ad68" (UID: "802df8bf-9c24-428b-9d87-96f58054ad68"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:38:30.120715 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:30.120686 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/802df8bf-9c24-428b-9d87-96f58054ad68-kube-api-access-htb2w" pod "802df8bf-9c24-428b-9d87-96f58054ad68" (UID: "802df8bf-9c24-428b-9d87-96f58054ad68"). InnerVolumeSpecName "kube-api-access-htb2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:38:30.120834 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:30.120755 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/802df8bf-9c24-428b-9d87-96f58054ad68-istio-podinfo" pod "802df8bf-9c24-428b-9d87-96f58054ad68" (UID: "802df8bf-9c24-428b-9d87-96f58054ad68"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:38:30.121307 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:30.121274 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/802df8bf-9c24-428b-9d87-96f58054ad68-istio-token" pod "802df8bf-9c24-428b-9d87-96f58054ad68" (UID: "802df8bf-9c24-428b-9d87-96f58054ad68"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:38:30.122829 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:30.122803 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/802df8bf-9c24-428b-9d87-96f58054ad68-istio-envoy" pod "802df8bf-9c24-428b-9d87-96f58054ad68" (UID: "802df8bf-9c24-428b-9d87-96f58054ad68"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:38:30.211442 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:30.210812 4562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/802df8bf-9c24-428b-9d87-96f58054ad68-istiod-ca-cert\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:38:30.211442 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:30.210856 4562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/802df8bf-9c24-428b-9d87-96f58054ad68-istio-podinfo\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:38:30.211442 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:30.210872 4562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/802df8bf-9c24-428b-9d87-96f58054ad68-istio-token\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:38:30.211442 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:30.210888 4562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/802df8bf-9c24-428b-9d87-96f58054ad68-workload-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:38:30.211442 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:30.210902 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htb2w\" (UniqueName: \"kubernetes.io/projected/802df8bf-9c24-428b-9d87-96f58054ad68-kube-api-access-htb2w\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:38:30.211442 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:30.210921 4562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/802df8bf-9c24-428b-9d87-96f58054ad68-istio-data\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:38:30.211442 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:30.210935 4562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/802df8bf-9c24-428b-9d87-96f58054ad68-workload-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:38:30.211442 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:30.210949 4562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/802df8bf-9c24-428b-9d87-96f58054ad68-credential-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:38:30.211442 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:30.210962 4562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/802df8bf-9c24-428b-9d87-96f58054ad68-istio-envoy\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:38:30.226900 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:30.226876 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-aa2fdf/auth-test-model-aa2fdf-kserve-699ff98945-zq96m" Oct 30 09:38:30.350086 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:30.350055 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd"] Oct 30 09:38:30.355676 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:30.355649 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-zlmzd"] Oct 30 09:38:30.430762 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:30.430716 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00d5c9cf-9dae-425f-8189-870602061926-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00d5c9cf-9dae-425f-8189-870602061926-tmp-dir\") pod \"00d5c9cf-9dae-425f-8189-870602061926\" (UID: \"00d5c9cf-9dae-425f-8189-870602061926\") " Oct 30 09:38:30.483184 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:30.483155 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00d5c9cf-9dae-425f-8189-870602061926-tls-certs\" (UniqueName: \"kubernetes.io/secret/00d5c9cf-9dae-425f-8189-870602061926-tls-certs\") pod \"00d5c9cf-9dae-425f-8189-870602061926\" (UID: \"00d5c9cf-9dae-425f-8189-870602061926\") " Oct 30 09:38:30.483456 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:30.483266 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00d5c9cf-9dae-425f-8189-870602061926-kube-api-access-8zvpf\" (UniqueName: \"kubernetes.io/projected/00d5c9cf-9dae-425f-8189-870602061926-kube-api-access-8zvpf\") pod \"00d5c9cf-9dae-425f-8189-870602061926\" (UID: \"00d5c9cf-9dae-425f-8189-870602061926\") " Oct 30 09:38:30.483456 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:30.483310 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00d5c9cf-9dae-425f-8189-870602061926-home\" (UniqueName: \"kubernetes.io/empty-dir/00d5c9cf-9dae-425f-8189-870602061926-home\") pod \"00d5c9cf-9dae-425f-8189-870602061926\" (UID: \"00d5c9cf-9dae-425f-8189-870602061926\") " Oct 30 09:38:30.483456 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:30.483441 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00d5c9cf-9dae-425f-8189-870602061926-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00d5c9cf-9dae-425f-8189-870602061926-model-cache\") pod \"00d5c9cf-9dae-425f-8189-870602061926\" (UID: \"00d5c9cf-9dae-425f-8189-870602061926\") " Oct 30 09:38:30.484125 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:30.483470 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00d5c9cf-9dae-425f-8189-870602061926-dshm\" (UniqueName: \"kubernetes.io/empty-dir/00d5c9cf-9dae-425f-8189-870602061926-dshm\") pod \"00d5c9cf-9dae-425f-8189-870602061926\" (UID: \"00d5c9cf-9dae-425f-8189-870602061926\") " Oct 30 09:38:30.487416 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:30.486562 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00d5c9cf-9dae-425f-8189-870602061926-tmp-dir" pod "00d5c9cf-9dae-425f-8189-870602061926" (UID: "00d5c9cf-9dae-425f-8189-870602061926"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:38:30.487893 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:30.487861 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00d5c9cf-9dae-425f-8189-870602061926-model-cache" pod "00d5c9cf-9dae-425f-8189-870602061926" (UID: "00d5c9cf-9dae-425f-8189-870602061926"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:38:30.490635 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:30.490597 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00d5c9cf-9dae-425f-8189-870602061926-kube-api-access-8zvpf" pod "00d5c9cf-9dae-425f-8189-870602061926" (UID: "00d5c9cf-9dae-425f-8189-870602061926"). InnerVolumeSpecName "kube-api-access-8zvpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:38:30.490908 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:30.490779 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk"] Oct 30 09:38:30.490908 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:30.490886 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00d5c9cf-9dae-425f-8189-870602061926-dshm" pod "00d5c9cf-9dae-425f-8189-870602061926" (UID: "00d5c9cf-9dae-425f-8189-870602061926"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:38:30.491322 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:30.491290 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00d5c9cf-9dae-425f-8189-870602061926-home" pod "00d5c9cf-9dae-425f-8189-870602061926" (UID: "00d5c9cf-9dae-425f-8189-870602061926"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:38:30.494053 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:30.493906 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00d5c9cf-9dae-425f-8189-870602061926-tls-certs" pod "00d5c9cf-9dae-425f-8189-870602061926" (UID: "00d5c9cf-9dae-425f-8189-870602061926"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:38:30.494654 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:30.494617 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:38:30.494750 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:30.494715 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:38:30.494849 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:30.494768 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:38:30.585093 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:30.585072 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zvpf\" (UniqueName: \"kubernetes.io/projected/00d5c9cf-9dae-425f-8189-870602061926-kube-api-access-8zvpf\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:38:30.585212 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:30.585096 4562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00d5c9cf-9dae-425f-8189-870602061926-home\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:38:30.585212 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:30.585105 4562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00d5c9cf-9dae-425f-8189-870602061926-model-cache\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:38:30.585212 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:30.585115 4562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00d5c9cf-9dae-425f-8189-870602061926-dshm\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:38:30.585212 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:30.585123 4562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00d5c9cf-9dae-425f-8189-870602061926-tmp-dir\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:38:30.585212 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:30.585130 4562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00d5c9cf-9dae-425f-8189-870602061926-tls-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:38:31.026649 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:31.026611 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-aa2fdf/auth-test-model-aa2fdf-kserve-699ff98945-zq96m" Oct 30 09:38:31.027062 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:31.026615 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-aa2fdf/auth-test-model-aa2fdf-kserve-699ff98945-zq96m" event={"ID":"00d5c9cf-9dae-425f-8189-870602061926","Type":"ContainerDied","Data":"c93c0918a0b3df1f52cfeb7fe56c18b72596ac22380472bd31b76fe04749648f"} Oct 30 09:38:31.027062 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:31.026778 4562 scope.go:122] "RemoveContainer" containerID="d80fd8b11c285fd2e2dd627d81427dc3443373a63b90b31cdca5ae043c917ff5" Oct 30 09:38:31.029224 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:31.029195 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-388653-6f7cf8fd57-686xr" event={"ID":"14eb89e0-861c-41a9-a245-e97adfa39f25","Type":"ContainerStarted","Data":"d9df37c272b79b292f23c7b3a4d2873d358422e4f8bc84e32245c65a941709a4"} Oct 30 09:38:31.029933 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:31.029914 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-388653-6f7cf8fd57-686xr" Oct 30 09:38:31.036698 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:31.036673 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk" event={"ID":"54cc6629-e1ca-4af7-9455-80e3b4d097f6","Type":"ContainerStarted","Data":"21fc9cf91ea050cde727c5f8eb885da8d61174f5d4c3cd0b9bf497ac5f65a86a"} Oct 30 09:38:31.036806 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:31.036722 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk" event={"ID":"54cc6629-e1ca-4af7-9455-80e3b4d097f6","Type":"ContainerStarted","Data":"928787ac3efa4bfe58572ea2eb2d41a8ce49601a8358a6bad030a3f7f0290339"} Oct 30 09:38:31.060691 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:31.060662 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk" Oct 30 09:38:31.062333 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:31.062309 4562 patch_prober.go:29] interesting pod/e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Oct 30 09:38:31.062456 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:31.062361 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk" podUID="54cc6629-e1ca-4af7-9455-80e3b4d097f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Oct 30 09:38:31.083801 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:31.083778 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-aa2fdf/auth-test-model-aa2fdf-kserve-699ff98945-zq96m"] Oct 30 09:38:31.090546 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:31.090524 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-aa2fdf/auth-test-model-aa2fdf-kserve-699ff98945-zq96m"] Oct 30 09:38:31.134443 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:31.134395 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-388653-6f7cf8fd57-686xr" podStartSLOduration=1.757080363 podStartE2EDuration="2.13436675s" totalImagesPullingTime="377.286387ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:38:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:38:31.131598947 +0000 UTC m=+2671.393159954" watchObservedRunningTime="2026-10-30 09:38:31.13436675 +0000 UTC m=+2671.395927738" Oct 30 09:38:31.175419 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:31.175354 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk" podStartSLOduration=2.175342091 podStartE2EDuration="2.175342091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:38:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:38:31.173272468 +0000 UTC m=+2671.434833493" watchObservedRunningTime="2026-10-30 09:38:31.175342091 +0000 UTC m=+2671.436903078" Oct 30 09:38:31.876350 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:31.876317 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00d5c9cf-9dae-425f-8189-870602061926" path="/var/lib/kubelet/pods/00d5c9cf-9dae-425f-8189-870602061926/volumes" Oct 30 09:38:31.876882 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:31.876864 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="802df8bf-9c24-428b-9d87-96f58054ad68" path="/var/lib/kubelet/pods/802df8bf-9c24-428b-9d87-96f58054ad68/volumes" Oct 30 09:38:32.064896 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:32.064866 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk" Oct 30 09:38:32.572402 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:32.563465 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d60c68-768c8ccddc-lwqlw"] Oct 30 09:38:32.572402 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:32.565160 4562 cpu_manager.go:414] "containerMap: removing container" podUID="802df8bf-9c24-428b-9d87-96f58054ad68" containerName="istio-proxy" Oct 30 09:38:32.572402 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:32.565181 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="802df8bf-9c24-428b-9d87-96f58054ad68" containerName="istio-proxy" Oct 30 09:38:32.572402 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:32.565334 4562 cpu_manager.go:414] "containerMap: removing container" podUID="00d5c9cf-9dae-425f-8189-870602061926" containerName="main" Oct 30 09:38:32.572402 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:32.565346 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00d5c9cf-9dae-425f-8189-870602061926" containerName="main" Oct 30 09:38:32.572402 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:32.565739 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="802df8bf-9c24-428b-9d87-96f58054ad68" containerName="istio-proxy" Oct 30 09:38:32.572402 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:32.565775 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="00d5c9cf-9dae-425f-8189-870602061926" containerName="main" Oct 30 09:38:32.573789 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:32.573763 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d60c68-768c8ccddc-lwqlw" Oct 30 09:38:32.585878 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:32.585854 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d60c68-768c8ccddc-lwqlw"] Oct 30 09:38:32.589090 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:32.589069 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-d60c68-dockercfg-xz7zw\"" Oct 30 09:38:32.589090 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:32.589074 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-d60c68\"" Oct 30 09:38:32.689418 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:32.672313 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtq8l\" (UniqueName: \"kubernetes.io/projected/c15af94f-dcf8-47ab-a39a-34de624afe00-kube-api-access-xtq8l\") pod \"payload-pre-processing-e2e-extmod-a-d60c68-768c8ccddc-lwqlw\" (UID: \"c15af94f-dcf8-47ab-a39a-34de624afe00\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d60c68-768c8ccddc-lwqlw" Oct 30 09:38:32.728579 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:32.689822 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c15af94f-dcf8-47ab-a39a-34de624afe00-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-d60c68-768c8ccddc-lwqlw\" (UID: \"c15af94f-dcf8-47ab-a39a-34de624afe00\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d60c68-768c8ccddc-lwqlw" Oct 30 09:38:32.779402 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:32.778476 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d60c68-865c467cd-zgnz8"] Oct 30 09:38:32.794566 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:32.784760 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d60c68-865c467cd-zgnz8" Oct 30 09:38:32.794566 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:32.784951 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d60c68-865c467cd-zgnz8"] Oct 30 09:38:32.874902 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:32.874824 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bsxn\" (UniqueName: \"kubernetes.io/projected/7f4a9934-3725-4b3c-82e4-900bdcd57e18-kube-api-access-6bsxn\") pod \"payload-processing-e2e-extmod-a-d60c68-865c467cd-zgnz8\" (UID: \"7f4a9934-3725-4b3c-82e4-900bdcd57e18\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d60c68-865c467cd-zgnz8" Oct 30 09:38:32.874902 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:32.874866 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtq8l\" (UniqueName: \"kubernetes.io/projected/c15af94f-dcf8-47ab-a39a-34de624afe00-kube-api-access-xtq8l\") pod \"payload-pre-processing-e2e-extmod-a-d60c68-768c8ccddc-lwqlw\" (UID: \"c15af94f-dcf8-47ab-a39a-34de624afe00\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d60c68-768c8ccddc-lwqlw" Oct 30 09:38:32.875081 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:32.874934 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f4a9934-3725-4b3c-82e4-900bdcd57e18-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-d60c68-865c467cd-zgnz8\" (UID: \"7f4a9934-3725-4b3c-82e4-900bdcd57e18\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d60c68-865c467cd-zgnz8" Oct 30 09:38:32.875081 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:32.875058 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c15af94f-dcf8-47ab-a39a-34de624afe00-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-d60c68-768c8ccddc-lwqlw\" (UID: \"c15af94f-dcf8-47ab-a39a-34de624afe00\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d60c68-768c8ccddc-lwqlw" Oct 30 09:38:32.875746 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:32.875729 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c15af94f-dcf8-47ab-a39a-34de624afe00-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-d60c68-768c8ccddc-lwqlw\" (UID: \"c15af94f-dcf8-47ab-a39a-34de624afe00\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d60c68-768c8ccddc-lwqlw" Oct 30 09:38:32.895832 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:32.895811 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtq8l\" (UniqueName: \"kubernetes.io/projected/c15af94f-dcf8-47ab-a39a-34de624afe00-kube-api-access-xtq8l\") pod \"payload-pre-processing-e2e-extmod-a-d60c68-768c8ccddc-lwqlw\" (UID: \"c15af94f-dcf8-47ab-a39a-34de624afe00\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d60c68-768c8ccddc-lwqlw" Oct 30 09:38:32.976093 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:32.976056 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bsxn\" (UniqueName: \"kubernetes.io/projected/7f4a9934-3725-4b3c-82e4-900bdcd57e18-kube-api-access-6bsxn\") pod \"payload-processing-e2e-extmod-a-d60c68-865c467cd-zgnz8\" (UID: \"7f4a9934-3725-4b3c-82e4-900bdcd57e18\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d60c68-865c467cd-zgnz8" Oct 30 09:38:32.976254 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:32.976115 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f4a9934-3725-4b3c-82e4-900bdcd57e18-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-d60c68-865c467cd-zgnz8\" (UID: \"7f4a9934-3725-4b3c-82e4-900bdcd57e18\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d60c68-865c467cd-zgnz8" Oct 30 09:38:32.976877 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:32.976859 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f4a9934-3725-4b3c-82e4-900bdcd57e18-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-d60c68-865c467cd-zgnz8\" (UID: \"7f4a9934-3725-4b3c-82e4-900bdcd57e18\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d60c68-865c467cd-zgnz8" Oct 30 09:38:32.997422 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:32.997372 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bsxn\" (UniqueName: \"kubernetes.io/projected/7f4a9934-3725-4b3c-82e4-900bdcd57e18-kube-api-access-6bsxn\") pod \"payload-processing-e2e-extmod-a-d60c68-865c467cd-zgnz8\" (UID: \"7f4a9934-3725-4b3c-82e4-900bdcd57e18\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d60c68-865c467cd-zgnz8" Oct 30 09:38:33.054182 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:33.054147 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk" Oct 30 09:38:33.055008 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:33.054977 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk" Oct 30 09:38:33.112852 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:33.112805 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d60c68-865c467cd-zgnz8" Oct 30 09:38:33.191702 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:33.191669 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d60c68-768c8ccddc-lwqlw" Oct 30 09:38:33.433480 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:38:33.433441 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f4a9934_3725_4b3c_82e4_900bdcd57e18.slice/crio-ccd92cc8595dd180a67bb62cc41ec345eaa754864dcdc2f9e83cb407d2c63749 WatchSource:0}: Error finding container ccd92cc8595dd180a67bb62cc41ec345eaa754864dcdc2f9e83cb407d2c63749: Status 404 returned error can't find the container with id ccd92cc8595dd180a67bb62cc41ec345eaa754864dcdc2f9e83cb407d2c63749 Oct 30 09:38:33.435411 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:33.435368 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d60c68-865c467cd-zgnz8"] Oct 30 09:38:33.524734 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:38:33.524703 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc15af94f_dcf8_47ab_a39a_34de624afe00.slice/crio-1d256fff24b4039178ce1cc9ce62ebe814047ee3e424950db6800a8a12220dfc WatchSource:0}: Error finding container 1d256fff24b4039178ce1cc9ce62ebe814047ee3e424950db6800a8a12220dfc: Status 404 returned error can't find the container with id 1d256fff24b4039178ce1cc9ce62ebe814047ee3e424950db6800a8a12220dfc Oct 30 09:38:33.526334 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:33.526304 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d60c68-768c8ccddc-lwqlw"] Oct 30 09:38:33.814108 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:33.814078 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-881f83-c579c5979-vj4zx" Oct 30 09:38:34.063629 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:34.063593 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d60c68-865c467cd-zgnz8" event={"ID":"7f4a9934-3725-4b3c-82e4-900bdcd57e18","Type":"ContainerStarted","Data":"41265d488eca01c4c12a6c7584c2a4a2407e36ffeb1c488881db0dc2d7dcb43e"} Oct 30 09:38:34.063832 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:34.063651 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d60c68-865c467cd-zgnz8" event={"ID":"7f4a9934-3725-4b3c-82e4-900bdcd57e18","Type":"ContainerStarted","Data":"ccd92cc8595dd180a67bb62cc41ec345eaa754864dcdc2f9e83cb407d2c63749"} Oct 30 09:38:34.063832 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:34.063678 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-d60c68-865c467cd-zgnz8" Oct 30 09:38:34.065563 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:34.065494 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d60c68-768c8ccddc-lwqlw" event={"ID":"c15af94f-dcf8-47ab-a39a-34de624afe00","Type":"ContainerStarted","Data":"6295fec76ca889f375a96f5a82295815b4376dd43fa7eff3948a57e9c69f0464"} Oct 30 09:38:34.065563 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:34.065552 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d60c68-768c8ccddc-lwqlw" event={"ID":"c15af94f-dcf8-47ab-a39a-34de624afe00","Type":"ContainerStarted","Data":"1d256fff24b4039178ce1cc9ce62ebe814047ee3e424950db6800a8a12220dfc"} Oct 30 09:38:34.065704 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:34.065667 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d60c68-768c8ccddc-lwqlw" Oct 30 09:38:34.105455 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:34.105412 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-d60c68-865c467cd-zgnz8" podStartSLOduration=2.10539946 podStartE2EDuration="2.10539946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:38:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:38:34.103960368 +0000 UTC m=+2674.365521356" watchObservedRunningTime="2026-10-30 09:38:34.10539946 +0000 UTC m=+2674.366960445" Oct 30 09:38:34.147049 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:34.147006 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d60c68-768c8ccddc-lwqlw" podStartSLOduration=2.146992767 podStartE2EDuration="2.146992767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:38:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:38:34.143852483 +0000 UTC m=+2674.405413485" watchObservedRunningTime="2026-10-30 09:38:34.146992767 +0000 UTC m=+2674.408553774" Oct 30 09:38:34.896150 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:34.896119 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-881f83-6b6777d984-fh7q9" Oct 30 09:38:35.512825 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:35.512793 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0f6f96-5d59889f9f-c8xpd"] Oct 30 09:38:35.517357 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:35.517335 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0f6f96-5d59889f9f-c8xpd" Oct 30 09:38:35.535442 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:35.533755 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-0f6f96\"" Oct 30 09:38:35.535442 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:35.534171 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-0f6f96-dockercfg-nsng5\"" Oct 30 09:38:35.550957 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:35.550927 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0f6f96-5d59889f9f-c8xpd"] Oct 30 09:38:35.625364 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:35.621923 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35bb8805-b6e0-4225-9345-094c08d33935-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-0f6f96-5d59889f9f-c8xpd\" (UID: \"35bb8805-b6e0-4225-9345-094c08d33935\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0f6f96-5d59889f9f-c8xpd" Oct 30 09:38:35.625364 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:35.622000 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkqxl\" (UniqueName: \"kubernetes.io/projected/35bb8805-b6e0-4225-9345-094c08d33935-kube-api-access-zkqxl\") pod \"payload-pre-processing-e2e-extmod-b-0f6f96-5d59889f9f-c8xpd\" (UID: \"35bb8805-b6e0-4225-9345-094c08d33935\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0f6f96-5d59889f9f-c8xpd" Oct 30 09:38:35.745865 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:35.743047 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35bb8805-b6e0-4225-9345-094c08d33935-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-0f6f96-5d59889f9f-c8xpd\" (UID: \"35bb8805-b6e0-4225-9345-094c08d33935\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0f6f96-5d59889f9f-c8xpd" Oct 30 09:38:35.745865 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:35.743146 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkqxl\" (UniqueName: \"kubernetes.io/projected/35bb8805-b6e0-4225-9345-094c08d33935-kube-api-access-zkqxl\") pod \"payload-pre-processing-e2e-extmod-b-0f6f96-5d59889f9f-c8xpd\" (UID: \"35bb8805-b6e0-4225-9345-094c08d33935\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0f6f96-5d59889f9f-c8xpd" Oct 30 09:38:35.748309 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:35.748271 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35bb8805-b6e0-4225-9345-094c08d33935-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-0f6f96-5d59889f9f-c8xpd\" (UID: \"35bb8805-b6e0-4225-9345-094c08d33935\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0f6f96-5d59889f9f-c8xpd" Oct 30 09:38:35.763556 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:35.763488 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkqxl\" (UniqueName: \"kubernetes.io/projected/35bb8805-b6e0-4225-9345-094c08d33935-kube-api-access-zkqxl\") pod \"payload-pre-processing-e2e-extmod-b-0f6f96-5d59889f9f-c8xpd\" (UID: \"35bb8805-b6e0-4225-9345-094c08d33935\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0f6f96-5d59889f9f-c8xpd" Oct 30 09:38:35.832091 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:35.832055 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0f6f96-5d59889f9f-c8xpd" Oct 30 09:38:36.142550 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:38:36.142521 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35bb8805_b6e0_4225_9345_094c08d33935.slice/crio-f582b735645e31f2315065d58485b9b649424934bc97f1fbdca717b56bd23511 WatchSource:0}: Error finding container f582b735645e31f2315065d58485b9b649424934bc97f1fbdca717b56bd23511: Status 404 returned error can't find the container with id f582b735645e31f2315065d58485b9b649424934bc97f1fbdca717b56bd23511 Oct 30 09:38:36.144848 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:36.144791 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0f6f96-5d59889f9f-c8xpd"] Oct 30 09:38:37.091027 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:37.090990 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0f6f96-5d59889f9f-c8xpd" event={"ID":"35bb8805-b6e0-4225-9345-094c08d33935","Type":"ContainerStarted","Data":"bc70985b974bf7ffc1367ee73adf15dcbaec421ca44f004b3b93da529c5f29be"} Oct 30 09:38:37.091419 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:37.091051 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0f6f96-5d59889f9f-c8xpd" event={"ID":"35bb8805-b6e0-4225-9345-094c08d33935","Type":"ContainerStarted","Data":"f582b735645e31f2315065d58485b9b649424934bc97f1fbdca717b56bd23511"} Oct 30 09:38:37.091419 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:37.091198 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0f6f96-5d59889f9f-c8xpd" Oct 30 09:38:37.131512 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:37.131457 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0f6f96-5d59889f9f-c8xpd" podStartSLOduration=2.131443501 podStartE2EDuration="2.131443501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:38:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:38:37.129456752 +0000 UTC m=+2677.391017773" watchObservedRunningTime="2026-10-30 09:38:37.131443501 +0000 UTC m=+2677.393004488" Oct 30 09:38:38.059710 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:38.059676 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-388653-6f7cf8fd57-686xr" Oct 30 09:38:45.091921 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:45.090822 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-d60c68-865c467cd-zgnz8" Oct 30 09:38:45.096101 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:45.096077 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d60c68-768c8ccddc-lwqlw" Oct 30 09:38:48.101309 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:48.101273 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0f6f96-5d59889f9f-c8xpd" Oct 30 09:38:49.777418 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:49.777368 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr"] Oct 30 09:38:49.781871 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:49.781853 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr" Oct 30 09:38:49.798247 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:49.798224 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-fb1a2a-openshift-default-dockercfg-cn24l\"" Oct 30 09:38:49.802279 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:49.802242 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr"] Oct 30 09:38:49.896719 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:49.896684 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/310de27f-d65f-4c73-840a-df4b5f6aed8b-istio-token\") pod \"e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr\" (UID: \"310de27f-d65f-4c73-840a-df4b5f6aed8b\") " pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr" Oct 30 09:38:49.896897 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:49.896833 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/310de27f-d65f-4c73-840a-df4b5f6aed8b-istiod-ca-cert\") pod \"e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr\" (UID: \"310de27f-d65f-4c73-840a-df4b5f6aed8b\") " pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr" Oct 30 09:38:49.896897 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:49.896872 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/310de27f-d65f-4c73-840a-df4b5f6aed8b-workload-certs\") pod \"e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr\" (UID: \"310de27f-d65f-4c73-840a-df4b5f6aed8b\") " pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr" Oct 30 09:38:49.896974 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:49.896927 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/310de27f-d65f-4c73-840a-df4b5f6aed8b-workload-socket\") pod \"e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr\" (UID: \"310de27f-d65f-4c73-840a-df4b5f6aed8b\") " pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr" Oct 30 09:38:49.897015 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:49.896995 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/310de27f-d65f-4c73-840a-df4b5f6aed8b-istio-podinfo\") pod \"e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr\" (UID: \"310de27f-d65f-4c73-840a-df4b5f6aed8b\") " pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr" Oct 30 09:38:49.897051 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:49.897035 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/310de27f-d65f-4c73-840a-df4b5f6aed8b-istio-data\") pod \"e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr\" (UID: \"310de27f-d65f-4c73-840a-df4b5f6aed8b\") " pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr" Oct 30 09:38:49.897113 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:49.897099 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/310de27f-d65f-4c73-840a-df4b5f6aed8b-credential-socket\") pod \"e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr\" (UID: \"310de27f-d65f-4c73-840a-df4b5f6aed8b\") " pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr" Oct 30 09:38:49.897150 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:49.897136 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/310de27f-d65f-4c73-840a-df4b5f6aed8b-istio-envoy\") pod \"e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr\" (UID: \"310de27f-d65f-4c73-840a-df4b5f6aed8b\") " pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr" Oct 30 09:38:49.897185 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:49.897159 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gttjf\" (UniqueName: \"kubernetes.io/projected/310de27f-d65f-4c73-840a-df4b5f6aed8b-kube-api-access-gttjf\") pod \"e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr\" (UID: \"310de27f-d65f-4c73-840a-df4b5f6aed8b\") " pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr" Oct 30 09:38:49.998609 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:49.998578 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/310de27f-d65f-4c73-840a-df4b5f6aed8b-istiod-ca-cert\") pod \"e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr\" (UID: \"310de27f-d65f-4c73-840a-df4b5f6aed8b\") " pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr" Oct 30 09:38:49.998844 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:49.998726 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/310de27f-d65f-4c73-840a-df4b5f6aed8b-workload-certs\") pod \"e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr\" (UID: \"310de27f-d65f-4c73-840a-df4b5f6aed8b\") " pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr" Oct 30 09:38:49.998844 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:49.998782 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/310de27f-d65f-4c73-840a-df4b5f6aed8b-workload-socket\") pod \"e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr\" (UID: \"310de27f-d65f-4c73-840a-df4b5f6aed8b\") " pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr" Oct 30 09:38:49.998987 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:49.998888 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/310de27f-d65f-4c73-840a-df4b5f6aed8b-istio-podinfo\") pod \"e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr\" (UID: \"310de27f-d65f-4c73-840a-df4b5f6aed8b\") " pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr" Oct 30 09:38:49.998987 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:49.998921 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/310de27f-d65f-4c73-840a-df4b5f6aed8b-istio-data\") pod \"e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr\" (UID: \"310de27f-d65f-4c73-840a-df4b5f6aed8b\") " pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr" Oct 30 09:38:49.999102 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:49.999065 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/310de27f-d65f-4c73-840a-df4b5f6aed8b-credential-socket\") pod \"e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr\" (UID: \"310de27f-d65f-4c73-840a-df4b5f6aed8b\") " pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr" Oct 30 09:38:49.999212 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:49.999186 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/310de27f-d65f-4c73-840a-df4b5f6aed8b-istio-envoy\") pod \"e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr\" (UID: \"310de27f-d65f-4c73-840a-df4b5f6aed8b\") " pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr" Oct 30 09:38:49.999452 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:49.999423 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gttjf\" (UniqueName: \"kubernetes.io/projected/310de27f-d65f-4c73-840a-df4b5f6aed8b-kube-api-access-gttjf\") pod \"e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr\" (UID: \"310de27f-d65f-4c73-840a-df4b5f6aed8b\") " pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr" Oct 30 09:38:49.999549 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:49.999464 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/310de27f-d65f-4c73-840a-df4b5f6aed8b-credential-socket\") pod \"e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr\" (UID: \"310de27f-d65f-4c73-840a-df4b5f6aed8b\") " pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr" Oct 30 09:38:49.999549 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:49.999492 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/310de27f-d65f-4c73-840a-df4b5f6aed8b-istio-token\") pod \"e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr\" (UID: \"310de27f-d65f-4c73-840a-df4b5f6aed8b\") " pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr" Oct 30 09:38:49.999549 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:49.999311 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/310de27f-d65f-4c73-840a-df4b5f6aed8b-workload-socket\") pod \"e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr\" (UID: \"310de27f-d65f-4c73-840a-df4b5f6aed8b\") " pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr" Oct 30 09:38:49.999549 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:49.999493 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/310de27f-d65f-4c73-840a-df4b5f6aed8b-istio-data\") pod \"e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr\" (UID: \"310de27f-d65f-4c73-840a-df4b5f6aed8b\") " pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr" Oct 30 09:38:49.999761 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:49.999572 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/310de27f-d65f-4c73-840a-df4b5f6aed8b-istiod-ca-cert\") pod \"e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr\" (UID: \"310de27f-d65f-4c73-840a-df4b5f6aed8b\") " pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr" Oct 30 09:38:49.999817 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:49.999795 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/310de27f-d65f-4c73-840a-df4b5f6aed8b-workload-certs\") pod \"e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr\" (UID: \"310de27f-d65f-4c73-840a-df4b5f6aed8b\") " pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr" Oct 30 09:38:50.001643 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:50.001623 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/310de27f-d65f-4c73-840a-df4b5f6aed8b-istio-podinfo\") pod \"e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr\" (UID: \"310de27f-d65f-4c73-840a-df4b5f6aed8b\") " pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr" Oct 30 09:38:50.001767 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:50.001750 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/310de27f-d65f-4c73-840a-df4b5f6aed8b-istio-envoy\") pod \"e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr\" (UID: \"310de27f-d65f-4c73-840a-df4b5f6aed8b\") " pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr" Oct 30 09:38:50.019217 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:50.019191 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gttjf\" (UniqueName: \"kubernetes.io/projected/310de27f-d65f-4c73-840a-df4b5f6aed8b-kube-api-access-gttjf\") pod \"e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr\" (UID: \"310de27f-d65f-4c73-840a-df4b5f6aed8b\") " pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr" Oct 30 09:38:50.019375 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:50.019354 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/310de27f-d65f-4c73-840a-df4b5f6aed8b-istio-token\") pod \"e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr\" (UID: \"310de27f-d65f-4c73-840a-df4b5f6aed8b\") " pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr" Oct 30 09:38:50.096931 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:50.096869 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr" Oct 30 09:38:50.425088 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:38:50.425057 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod310de27f_d65f_4c73_840a_df4b5f6aed8b.slice/crio-24111d419a0b7c991fa69badd17d365a23bda3446a53c90f013d4b7774d04215 WatchSource:0}: Error finding container 24111d419a0b7c991fa69badd17d365a23bda3446a53c90f013d4b7774d04215: Status 404 returned error can't find the container with id 24111d419a0b7c991fa69badd17d365a23bda3446a53c90f013d4b7774d04215 Oct 30 09:38:50.426068 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:50.426041 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr"] Oct 30 09:38:50.428051 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:50.428019 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:38:50.428138 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:50.428084 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:38:50.428138 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:50.428116 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:38:51.200579 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:51.200538 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr" event={"ID":"310de27f-d65f-4c73-840a-df4b5f6aed8b","Type":"ContainerStarted","Data":"d8892b8b3bcc9a80af360f1d79970a7a39fd4141ac28bba34b334f125530bbc6"} Oct 30 09:38:51.201181 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:51.200607 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr" event={"ID":"310de27f-d65f-4c73-840a-df4b5f6aed8b","Type":"ContainerStarted","Data":"24111d419a0b7c991fa69badd17d365a23bda3446a53c90f013d4b7774d04215"} Oct 30 09:38:51.276731 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:51.276673 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr" podStartSLOduration=2.276658785 podStartE2EDuration="2.276658785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:38:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:38:51.263427227 +0000 UTC m=+2691.524988212" watchObservedRunningTime="2026-10-30 09:38:51.276658785 +0000 UTC m=+2691.538219773" Oct 30 09:38:52.097334 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:52.097295 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr" Oct 30 09:38:52.098995 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:52.098966 4562 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body= Oct 30 09:38:52.099129 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:52.099046 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr" podUID="310de27f-d65f-4c73-840a-df4b5f6aed8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" Oct 30 09:38:53.097736 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:53.097703 4562 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body= Oct 30 09:38:53.098154 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:53.097760 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr" podUID="310de27f-d65f-4c73-840a-df4b5f6aed8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" Oct 30 09:38:54.097277 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:54.097238 4562 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body= Oct 30 09:38:54.097512 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:54.097309 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr" podUID="310de27f-d65f-4c73-840a-df4b5f6aed8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" Oct 30 09:38:55.097986 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:55.097953 4562 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body= Oct 30 09:38:55.098458 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:55.098021 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr" podUID="310de27f-d65f-4c73-840a-df4b5f6aed8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" Oct 30 09:38:56.097642 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:56.097609 4562 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body= Oct 30 09:38:56.097819 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:56.097662 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr" podUID="310de27f-d65f-4c73-840a-df4b5f6aed8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" Oct 30 09:38:57.097582 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:57.097549 4562 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body= Oct 30 09:38:57.098009 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:57.097598 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr" podUID="310de27f-d65f-4c73-840a-df4b5f6aed8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" Oct 30 09:38:58.102335 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:58.102306 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr" Oct 30 09:38:58.102857 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:58.102703 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr" Oct 30 09:38:58.103508 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:38:58.103486 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr" Oct 30 09:38:58.596338 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:38:58.596304 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:39:00.255488 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:00.255461 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6q7n_1f51db5a-5720-4dea-ad02-f336b35d43b1/ovn-acl-logging/0.log" Oct 30 09:39:00.264207 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:00.264184 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6q7n_1f51db5a-5720-4dea-ad02-f336b35d43b1/ovn-acl-logging/0.log" Oct 30 09:39:06.283498 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:06.283459 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24"] Oct 30 09:39:06.299205 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:06.299166 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24" Oct 30 09:39:06.318623 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:06.318532 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18686e38-6651-454d-b9ad-996ffebe2b91-istio-podinfo\") pod \"e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24\" (UID: \"18686e38-6651-454d-b9ad-996ffebe2b91\") " pod="openshift-ingress/e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24" Oct 30 09:39:06.318623 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:06.318608 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18686e38-6651-454d-b9ad-996ffebe2b91-istio-token\") pod \"e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24\" (UID: \"18686e38-6651-454d-b9ad-996ffebe2b91\") " pod="openshift-ingress/e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24" Oct 30 09:39:06.318970 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:06.318736 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18686e38-6651-454d-b9ad-996ffebe2b91-istiod-ca-cert\") pod \"e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24\" (UID: \"18686e38-6651-454d-b9ad-996ffebe2b91\") " pod="openshift-ingress/e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24" Oct 30 09:39:06.318970 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:06.318816 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18686e38-6651-454d-b9ad-996ffebe2b91-credential-socket\") pod \"e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24\" (UID: \"18686e38-6651-454d-b9ad-996ffebe2b91\") " pod="openshift-ingress/e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24" Oct 30 09:39:06.318970 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:06.318894 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18686e38-6651-454d-b9ad-996ffebe2b91-istio-envoy\") pod \"e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24\" (UID: \"18686e38-6651-454d-b9ad-996ffebe2b91\") " pod="openshift-ingress/e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24" Oct 30 09:39:06.318970 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:06.318942 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18686e38-6651-454d-b9ad-996ffebe2b91-workload-certs\") pod \"e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24\" (UID: \"18686e38-6651-454d-b9ad-996ffebe2b91\") " pod="openshift-ingress/e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24" Oct 30 09:39:06.319680 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:06.319001 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxdzg\" (UniqueName: \"kubernetes.io/projected/18686e38-6651-454d-b9ad-996ffebe2b91-kube-api-access-lxdzg\") pod \"e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24\" (UID: \"18686e38-6651-454d-b9ad-996ffebe2b91\") " pod="openshift-ingress/e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24" Oct 30 09:39:06.319680 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:06.319043 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18686e38-6651-454d-b9ad-996ffebe2b91-istio-data\") pod \"e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24\" (UID: \"18686e38-6651-454d-b9ad-996ffebe2b91\") " pod="openshift-ingress/e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24" Oct 30 09:39:06.319680 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:06.319084 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18686e38-6651-454d-b9ad-996ffebe2b91-workload-socket\") pod \"e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24\" (UID: \"18686e38-6651-454d-b9ad-996ffebe2b91\") " pod="openshift-ingress/e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24" Oct 30 09:39:06.326128 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:06.324438 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24"] Oct 30 09:39:06.326128 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:06.325302 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-a525bd-openshift-default-dockercfg-9wd2d\"" Oct 30 09:39:06.429697 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:06.429656 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18686e38-6651-454d-b9ad-996ffebe2b91-istio-podinfo\") pod \"e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24\" (UID: \"18686e38-6651-454d-b9ad-996ffebe2b91\") " pod="openshift-ingress/e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24" Oct 30 09:39:06.429996 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:06.429967 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18686e38-6651-454d-b9ad-996ffebe2b91-istio-token\") pod \"e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24\" (UID: \"18686e38-6651-454d-b9ad-996ffebe2b91\") " pod="openshift-ingress/e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24" Oct 30 09:39:06.430304 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:06.430285 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18686e38-6651-454d-b9ad-996ffebe2b91-istiod-ca-cert\") pod \"e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24\" (UID: \"18686e38-6651-454d-b9ad-996ffebe2b91\") " pod="openshift-ingress/e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24" Oct 30 09:39:06.430476 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:06.430460 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18686e38-6651-454d-b9ad-996ffebe2b91-credential-socket\") pod \"e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24\" (UID: \"18686e38-6651-454d-b9ad-996ffebe2b91\") " pod="openshift-ingress/e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24" Oct 30 09:39:06.430672 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:06.430645 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18686e38-6651-454d-b9ad-996ffebe2b91-istio-envoy\") pod \"e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24\" (UID: \"18686e38-6651-454d-b9ad-996ffebe2b91\") " pod="openshift-ingress/e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24" Oct 30 09:39:06.430858 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:06.430839 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18686e38-6651-454d-b9ad-996ffebe2b91-workload-certs\") pod \"e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24\" (UID: \"18686e38-6651-454d-b9ad-996ffebe2b91\") " pod="openshift-ingress/e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24" Oct 30 09:39:06.431085 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:06.431027 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxdzg\" (UniqueName: \"kubernetes.io/projected/18686e38-6651-454d-b9ad-996ffebe2b91-kube-api-access-lxdzg\") pod \"e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24\" (UID: \"18686e38-6651-454d-b9ad-996ffebe2b91\") " pod="openshift-ingress/e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24" Oct 30 09:39:06.431252 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:06.431232 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18686e38-6651-454d-b9ad-996ffebe2b91-istio-data\") pod \"e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24\" (UID: \"18686e38-6651-454d-b9ad-996ffebe2b91\") " pod="openshift-ingress/e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24" Oct 30 09:39:06.431408 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:06.431376 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18686e38-6651-454d-b9ad-996ffebe2b91-workload-socket\") pod \"e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24\" (UID: \"18686e38-6651-454d-b9ad-996ffebe2b91\") " pod="openshift-ingress/e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24" Oct 30 09:39:06.432194 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:06.432169 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18686e38-6651-454d-b9ad-996ffebe2b91-workload-socket\") pod \"e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24\" (UID: \"18686e38-6651-454d-b9ad-996ffebe2b91\") " pod="openshift-ingress/e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24" Oct 30 09:39:06.434075 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:06.433634 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18686e38-6651-454d-b9ad-996ffebe2b91-workload-certs\") pod \"e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24\" (UID: \"18686e38-6651-454d-b9ad-996ffebe2b91\") " pod="openshift-ingress/e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24" Oct 30 09:39:06.437710 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:06.436775 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18686e38-6651-454d-b9ad-996ffebe2b91-istio-data\") pod \"e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24\" (UID: \"18686e38-6651-454d-b9ad-996ffebe2b91\") " pod="openshift-ingress/e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24" Oct 30 09:39:06.438655 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:06.438285 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18686e38-6651-454d-b9ad-996ffebe2b91-istiod-ca-cert\") pod \"e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24\" (UID: \"18686e38-6651-454d-b9ad-996ffebe2b91\") " pod="openshift-ingress/e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24" Oct 30 09:39:06.441268 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:06.440010 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18686e38-6651-454d-b9ad-996ffebe2b91-credential-socket\") pod \"e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24\" (UID: \"18686e38-6651-454d-b9ad-996ffebe2b91\") " pod="openshift-ingress/e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24" Oct 30 09:39:06.475228 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:06.475164 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18686e38-6651-454d-b9ad-996ffebe2b91-istio-envoy\") pod \"e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24\" (UID: \"18686e38-6651-454d-b9ad-996ffebe2b91\") " pod="openshift-ingress/e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24" Oct 30 09:39:06.475962 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:06.475941 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18686e38-6651-454d-b9ad-996ffebe2b91-istio-podinfo\") pod \"e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24\" (UID: \"18686e38-6651-454d-b9ad-996ffebe2b91\") " pod="openshift-ingress/e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24" Oct 30 09:39:06.514042 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:06.514002 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxdzg\" (UniqueName: \"kubernetes.io/projected/18686e38-6651-454d-b9ad-996ffebe2b91-kube-api-access-lxdzg\") pod \"e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24\" (UID: \"18686e38-6651-454d-b9ad-996ffebe2b91\") " pod="openshift-ingress/e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24" Oct 30 09:39:06.515173 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:06.515109 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18686e38-6651-454d-b9ad-996ffebe2b91-istio-token\") pod \"e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24\" (UID: \"18686e38-6651-454d-b9ad-996ffebe2b91\") " pod="openshift-ingress/e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24" Oct 30 09:39:06.634368 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:06.634272 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24" Oct 30 09:39:06.951932 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:39:06.951904 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18686e38_6651_454d_b9ad_996ffebe2b91.slice/crio-b56016622c9eaab17c4aa4d150152d152036d3d5ff625a1e3219cb0fd753e61f WatchSource:0}: Error finding container b56016622c9eaab17c4aa4d150152d152036d3d5ff625a1e3219cb0fd753e61f: Status 404 returned error can't find the container with id b56016622c9eaab17c4aa4d150152d152036d3d5ff625a1e3219cb0fd753e61f Oct 30 09:39:06.954645 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:06.954619 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24"] Oct 30 09:39:06.955196 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:06.955166 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:39:06.955243 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:06.955228 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:39:06.955278 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:06.955270 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:39:07.341206 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:07.341114 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24" event={"ID":"18686e38-6651-454d-b9ad-996ffebe2b91","Type":"ContainerStarted","Data":"8164ea6956edc7d2e43109737ac0f7bfbe74b625b3c510d1307756202a96688c"} Oct 30 09:39:07.341206 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:07.341183 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24" event={"ID":"18686e38-6651-454d-b9ad-996ffebe2b91","Type":"ContainerStarted","Data":"b56016622c9eaab17c4aa4d150152d152036d3d5ff625a1e3219cb0fd753e61f"} Oct 30 09:39:07.386938 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:07.386888 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24" podStartSLOduration=1.386874406 podStartE2EDuration="1.386874406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:39:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:39:07.38279313 +0000 UTC m=+2707.644354117" watchObservedRunningTime="2026-10-30 09:39:07.386874406 +0000 UTC m=+2707.648435714" Oct 30 09:39:07.634623 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:07.634542 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24" Oct 30 09:39:08.640764 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:08.640731 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24" Oct 30 09:39:09.354918 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:09.354887 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24" Oct 30 09:39:09.355888 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:09.355867 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24" Oct 30 09:39:15.097095 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:15.097061 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7a213d/auth-test-model-7a213d-kserve-54db88fccc-j8ldx"] Oct 30 09:39:22.126952 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:22.126913 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-6d4b7a-57ddcbd6fb-wzqlg"] Oct 30 09:39:22.162411 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:22.136096 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-6d4b7a-57ddcbd6fb-wzqlg" Oct 30 09:39:22.171404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:22.169970 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-6d4b7a-dockercfg-mpvqw\"" Oct 30 09:39:22.171404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:22.170896 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-6d4b7a\"" Oct 30 09:39:22.179406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:22.172171 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc4c8ab1-cce2-4a36-a929-7ec43a26c17a-plugins-config-volume\") pod \"payload-processing-e2e-autores-6d4b7a-57ddcbd6fb-wzqlg\" (UID: \"bc4c8ab1-cce2-4a36-a929-7ec43a26c17a\") " pod="openshift-ingress/payload-processing-e2e-autores-6d4b7a-57ddcbd6fb-wzqlg" Oct 30 09:39:22.179406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:22.172299 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9c6j\" (UniqueName: \"kubernetes.io/projected/bc4c8ab1-cce2-4a36-a929-7ec43a26c17a-kube-api-access-h9c6j\") pod \"payload-processing-e2e-autores-6d4b7a-57ddcbd6fb-wzqlg\" (UID: \"bc4c8ab1-cce2-4a36-a929-7ec43a26c17a\") " pod="openshift-ingress/payload-processing-e2e-autores-6d4b7a-57ddcbd6fb-wzqlg" Oct 30 09:39:22.295803 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:22.287870 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc4c8ab1-cce2-4a36-a929-7ec43a26c17a-plugins-config-volume\") pod \"payload-processing-e2e-autores-6d4b7a-57ddcbd6fb-wzqlg\" (UID: \"bc4c8ab1-cce2-4a36-a929-7ec43a26c17a\") " pod="openshift-ingress/payload-processing-e2e-autores-6d4b7a-57ddcbd6fb-wzqlg" Oct 30 09:39:22.295803 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:22.288007 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9c6j\" (UniqueName: \"kubernetes.io/projected/bc4c8ab1-cce2-4a36-a929-7ec43a26c17a-kube-api-access-h9c6j\") pod \"payload-processing-e2e-autores-6d4b7a-57ddcbd6fb-wzqlg\" (UID: \"bc4c8ab1-cce2-4a36-a929-7ec43a26c17a\") " pod="openshift-ingress/payload-processing-e2e-autores-6d4b7a-57ddcbd6fb-wzqlg" Oct 30 09:39:22.295803 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:22.289595 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc4c8ab1-cce2-4a36-a929-7ec43a26c17a-plugins-config-volume\") pod \"payload-processing-e2e-autores-6d4b7a-57ddcbd6fb-wzqlg\" (UID: \"bc4c8ab1-cce2-4a36-a929-7ec43a26c17a\") " pod="openshift-ingress/payload-processing-e2e-autores-6d4b7a-57ddcbd6fb-wzqlg" Oct 30 09:39:22.324807 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:22.311786 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-6d4b7a-57ddcbd6fb-wzqlg"] Oct 30 09:39:22.352882 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:22.352405 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9c6j\" (UniqueName: \"kubernetes.io/projected/bc4c8ab1-cce2-4a36-a929-7ec43a26c17a-kube-api-access-h9c6j\") pod \"payload-processing-e2e-autores-6d4b7a-57ddcbd6fb-wzqlg\" (UID: \"bc4c8ab1-cce2-4a36-a929-7ec43a26c17a\") " pod="openshift-ingress/payload-processing-e2e-autores-6d4b7a-57ddcbd6fb-wzqlg" Oct 30 09:39:22.368467 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:22.368432 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-6d4b7a-57ddcbd6fb-wzqlg" Oct 30 09:39:22.689377 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:39:22.689343 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc4c8ab1_cce2_4a36_a929_7ec43a26c17a.slice/crio-9de273a7754301ca8203a0f47cbbe76a1b84bcda565cd82da0f11efcdd2e50ad WatchSource:0}: Error finding container 9de273a7754301ca8203a0f47cbbe76a1b84bcda565cd82da0f11efcdd2e50ad: Status 404 returned error can't find the container with id 9de273a7754301ca8203a0f47cbbe76a1b84bcda565cd82da0f11efcdd2e50ad Oct 30 09:39:22.690428 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:22.690308 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-6d4b7a-57ddcbd6fb-wzqlg"] Oct 30 09:39:23.576601 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:23.576556 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-6d4b7a-57ddcbd6fb-wzqlg" event={"ID":"bc4c8ab1-cce2-4a36-a929-7ec43a26c17a","Type":"ContainerStarted","Data":"e611286eab9db774100a84d38ca05f39b9ec8f41e1cab5abdc8f7819ca1c7a79"} Oct 30 09:39:23.577083 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:23.576656 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-6d4b7a-57ddcbd6fb-wzqlg" event={"ID":"bc4c8ab1-cce2-4a36-a929-7ec43a26c17a","Type":"ContainerStarted","Data":"9de273a7754301ca8203a0f47cbbe76a1b84bcda565cd82da0f11efcdd2e50ad"} Oct 30 09:39:23.577083 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:23.576778 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-6d4b7a-57ddcbd6fb-wzqlg" Oct 30 09:39:24.585735 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:24.585645 4562 generic.go:356] "Generic (PLEG): container finished" podID="aea403b0-3184-48ef-8e1a-b1f91b56d703" containerID="2b2612ef43ed39a643510f2582516db4e90bb96a434e3624eaaf66443eb33d37" exitCode=0 Oct 30 09:39:24.585735 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:24.585719 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h" event={"ID":"aea403b0-3184-48ef-8e1a-b1f91b56d703","Type":"ContainerDied","Data":"2b2612ef43ed39a643510f2582516db4e90bb96a434e3624eaaf66443eb33d37"} Oct 30 09:39:24.586249 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:24.586194 4562 scope.go:122] "RemoveContainer" containerID="2b2612ef43ed39a643510f2582516db4e90bb96a434e3624eaaf66443eb33d37" Oct 30 09:39:24.587740 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:24.587713 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:39:24.587816 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:24.587780 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:39:24.587816 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:24.587811 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:39:24.658701 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:24.658652 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-6d4b7a-57ddcbd6fb-wzqlg" podStartSLOduration=2.658635787 podStartE2EDuration="2.658635787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:39:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:39:23.617454834 +0000 UTC m=+2723.879015850" watchObservedRunningTime="2026-10-30 09:39:24.658635787 +0000 UTC m=+2724.920196775" Oct 30 09:39:25.594469 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:25.594432 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h" event={"ID":"aea403b0-3184-48ef-8e1a-b1f91b56d703","Type":"ContainerStarted","Data":"7a0381a04c9a072371781cd7fb53195ead4de0500e0be0e06ed9bb0ab2fe83ac"} Oct 30 09:39:26.057554 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:26.057516 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h" Oct 30 09:39:26.057749 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:26.057652 4562 patch_prober.go:29] interesting pod/e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body= Oct 30 09:39:26.057749 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:26.057704 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h" podUID="aea403b0-3184-48ef-8e1a-b1f91b56d703" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" Oct 30 09:39:27.060405 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:27.057960 4562 patch_prober.go:29] interesting pod/e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body= Oct 30 09:39:27.060405 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:27.058022 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h" podUID="aea403b0-3184-48ef-8e1a-b1f91b56d703" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" Oct 30 09:39:27.169193 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:27.169155 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp"] Oct 30 09:39:27.169484 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:27.169445 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp" podUID="4406b9a5-a7b3-4469-9760-f3dcbec872a0" containerName="istio-proxy" containerID="cri-o://f99be65fe664122bff556172bc3db3958f832b2ba284800e0b8874822ae34f9a" gracePeriod=30 Oct 30 09:39:28.065460 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:28.061525 4562 patch_prober.go:29] interesting pod/e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body= Oct 30 09:39:28.065460 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:28.061589 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h" podUID="aea403b0-3184-48ef-8e1a-b1f91b56d703" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" Oct 30 09:39:28.724837 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:39:28.724748 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:39:29.062731 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:29.062693 4562 patch_prober.go:29] interesting pod/e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body= Oct 30 09:39:29.065470 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:29.063027 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h" podUID="aea403b0-3184-48ef-8e1a-b1f91b56d703" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" Oct 30 09:39:30.067812 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:30.067777 4562 patch_prober.go:29] interesting pod/e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body= Oct 30 09:39:30.081408 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:30.068444 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h" podUID="aea403b0-3184-48ef-8e1a-b1f91b56d703" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" Oct 30 09:39:30.125439 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:30.125395 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-7a213d/auth-test-model-7a213d-kserve-54db88fccc-j8ldx" podUID="75eb9d8e-5d05-47d3-b923-06404edb57b2" containerName="main" containerID="cri-o://68fdb6dcde6cb7a89b57259897d851965854add2e91f6ea4636168b922c46ac3" gracePeriod=45 Oct 30 09:39:30.482857 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:30.482827 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7a213d/auth-test-model-7a213d-kserve-54db88fccc-j8ldx" Oct 30 09:39:30.595581 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:30.595493 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75eb9d8e-5d05-47d3-b923-06404edb57b2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75eb9d8e-5d05-47d3-b923-06404edb57b2-model-cache\") pod \"75eb9d8e-5d05-47d3-b923-06404edb57b2\" (UID: \"75eb9d8e-5d05-47d3-b923-06404edb57b2\") " Oct 30 09:39:30.595581 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:30.595576 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75eb9d8e-5d05-47d3-b923-06404edb57b2-tls-certs\" (UniqueName: \"kubernetes.io/secret/75eb9d8e-5d05-47d3-b923-06404edb57b2-tls-certs\") pod \"75eb9d8e-5d05-47d3-b923-06404edb57b2\" (UID: \"75eb9d8e-5d05-47d3-b923-06404edb57b2\") " Oct 30 09:39:30.595838 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:30.595638 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75eb9d8e-5d05-47d3-b923-06404edb57b2-kube-api-access-nqwrw\" (UniqueName: \"kubernetes.io/projected/75eb9d8e-5d05-47d3-b923-06404edb57b2-kube-api-access-nqwrw\") pod \"75eb9d8e-5d05-47d3-b923-06404edb57b2\" (UID: \"75eb9d8e-5d05-47d3-b923-06404edb57b2\") " Oct 30 09:39:30.595838 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:30.595671 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75eb9d8e-5d05-47d3-b923-06404edb57b2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75eb9d8e-5d05-47d3-b923-06404edb57b2-tmp-dir\") pod \"75eb9d8e-5d05-47d3-b923-06404edb57b2\" (UID: \"75eb9d8e-5d05-47d3-b923-06404edb57b2\") " Oct 30 09:39:30.595838 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:30.595726 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75eb9d8e-5d05-47d3-b923-06404edb57b2-model-cache" pod "75eb9d8e-5d05-47d3-b923-06404edb57b2" (UID: "75eb9d8e-5d05-47d3-b923-06404edb57b2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:39:30.595838 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:30.595806 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75eb9d8e-5d05-47d3-b923-06404edb57b2-home\" (UniqueName: \"kubernetes.io/empty-dir/75eb9d8e-5d05-47d3-b923-06404edb57b2-home\") pod \"75eb9d8e-5d05-47d3-b923-06404edb57b2\" (UID: \"75eb9d8e-5d05-47d3-b923-06404edb57b2\") " Oct 30 09:39:30.596098 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:30.595883 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75eb9d8e-5d05-47d3-b923-06404edb57b2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/75eb9d8e-5d05-47d3-b923-06404edb57b2-dshm\") pod \"75eb9d8e-5d05-47d3-b923-06404edb57b2\" (UID: \"75eb9d8e-5d05-47d3-b923-06404edb57b2\") " Oct 30 09:39:30.596098 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:30.596089 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75eb9d8e-5d05-47d3-b923-06404edb57b2-tmp-dir" pod "75eb9d8e-5d05-47d3-b923-06404edb57b2" (UID: "75eb9d8e-5d05-47d3-b923-06404edb57b2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:39:30.596262 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:30.596246 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75eb9d8e-5d05-47d3-b923-06404edb57b2-home" pod "75eb9d8e-5d05-47d3-b923-06404edb57b2" (UID: "75eb9d8e-5d05-47d3-b923-06404edb57b2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:39:30.596927 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:30.596558 4562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75eb9d8e-5d05-47d3-b923-06404edb57b2-model-cache\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:39:30.596927 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:30.596582 4562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75eb9d8e-5d05-47d3-b923-06404edb57b2-tmp-dir\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:39:30.596927 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:30.596596 4562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75eb9d8e-5d05-47d3-b923-06404edb57b2-home\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:39:30.598462 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:30.598437 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75eb9d8e-5d05-47d3-b923-06404edb57b2-kube-api-access-nqwrw" pod "75eb9d8e-5d05-47d3-b923-06404edb57b2" (UID: "75eb9d8e-5d05-47d3-b923-06404edb57b2"). InnerVolumeSpecName "kube-api-access-nqwrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:39:30.598798 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:30.598776 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75eb9d8e-5d05-47d3-b923-06404edb57b2-dshm" pod "75eb9d8e-5d05-47d3-b923-06404edb57b2" (UID: "75eb9d8e-5d05-47d3-b923-06404edb57b2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:39:30.599056 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:30.599039 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75eb9d8e-5d05-47d3-b923-06404edb57b2-tls-certs" pod "75eb9d8e-5d05-47d3-b923-06404edb57b2" (UID: "75eb9d8e-5d05-47d3-b923-06404edb57b2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:39:30.679090 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:30.679055 4562 generic.go:356] "Generic (PLEG): container finished" podID="75eb9d8e-5d05-47d3-b923-06404edb57b2" containerID="68fdb6dcde6cb7a89b57259897d851965854add2e91f6ea4636168b922c46ac3" exitCode=0 Oct 30 09:39:30.679285 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:30.679111 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7a213d/auth-test-model-7a213d-kserve-54db88fccc-j8ldx" Oct 30 09:39:30.679285 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:30.679151 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7a213d/auth-test-model-7a213d-kserve-54db88fccc-j8ldx" event={"ID":"75eb9d8e-5d05-47d3-b923-06404edb57b2","Type":"ContainerDied","Data":"68fdb6dcde6cb7a89b57259897d851965854add2e91f6ea4636168b922c46ac3"} Oct 30 09:39:30.679285 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:30.679201 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7a213d/auth-test-model-7a213d-kserve-54db88fccc-j8ldx" event={"ID":"75eb9d8e-5d05-47d3-b923-06404edb57b2","Type":"ContainerDied","Data":"aa0bfede847e016de3aeef16fc69f11fc4153395b2801851f8f4c5e5e8b9e774"} Oct 30 09:39:30.679285 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:30.679226 4562 scope.go:122] "RemoveContainer" containerID="68fdb6dcde6cb7a89b57259897d851965854add2e91f6ea4636168b922c46ac3" Oct 30 09:39:30.689876 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:30.689853 4562 scope.go:122] "RemoveContainer" containerID="68fdb6dcde6cb7a89b57259897d851965854add2e91f6ea4636168b922c46ac3" Oct 30 09:39:30.690178 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:39:30.690156 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68fdb6dcde6cb7a89b57259897d851965854add2e91f6ea4636168b922c46ac3\": container with ID starting with 68fdb6dcde6cb7a89b57259897d851965854add2e91f6ea4636168b922c46ac3 not found: ID does not exist" containerID="68fdb6dcde6cb7a89b57259897d851965854add2e91f6ea4636168b922c46ac3" Oct 30 09:39:30.690287 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:30.690190 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68fdb6dcde6cb7a89b57259897d851965854add2e91f6ea4636168b922c46ac3"} err="failed to get container status \"68fdb6dcde6cb7a89b57259897d851965854add2e91f6ea4636168b922c46ac3\": rpc error: code = NotFound desc = could not find container \"68fdb6dcde6cb7a89b57259897d851965854add2e91f6ea4636168b922c46ac3\": container with ID starting with 68fdb6dcde6cb7a89b57259897d851965854add2e91f6ea4636168b922c46ac3 not found: ID does not exist" Oct 30 09:39:30.698183 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:30.698153 4562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75eb9d8e-5d05-47d3-b923-06404edb57b2-dshm\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:39:30.698183 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:30.698186 4562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75eb9d8e-5d05-47d3-b923-06404edb57b2-tls-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:39:30.698309 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:30.698196 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqwrw\" (UniqueName: \"kubernetes.io/projected/75eb9d8e-5d05-47d3-b923-06404edb57b2-kube-api-access-nqwrw\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:39:30.734583 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:30.734557 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7a213d/auth-test-model-7a213d-kserve-54db88fccc-j8ldx"] Oct 30 09:39:30.737908 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:30.737886 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7a213d/auth-test-model-7a213d-kserve-54db88fccc-j8ldx"] Oct 30 09:39:31.069941 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:31.069897 4562 patch_prober.go:29] interesting pod/e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body= Oct 30 09:39:31.070777 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:31.069962 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h" podUID="aea403b0-3184-48ef-8e1a-b1f91b56d703" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" Oct 30 09:39:31.405540 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:31.405448 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-7a213d/auth-test-model-7a213d-kserve-54db88fccc-j8ldx" podUID="75eb9d8e-5d05-47d3-b923-06404edb57b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.192:8000/ready\": context deadline exceeded" Oct 30 09:39:31.892603 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:31.892560 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75eb9d8e-5d05-47d3-b923-06404edb57b2" path="/var/lib/kubelet/pods/75eb9d8e-5d05-47d3-b923-06404edb57b2/volumes" Oct 30 09:39:32.063607 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:32.063575 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h" Oct 30 09:39:32.063966 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:32.063942 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h" Oct 30 09:39:32.064814 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:32.064790 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h" Oct 30 09:39:32.458397 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:32.458356 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp" Oct 30 09:39:32.528498 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:32.528468 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4406b9a5-a7b3-4469-9760-f3dcbec872a0-kube-api-access-mjbpv\" (UniqueName: \"kubernetes.io/projected/4406b9a5-a7b3-4469-9760-f3dcbec872a0-kube-api-access-mjbpv\") pod \"4406b9a5-a7b3-4469-9760-f3dcbec872a0\" (UID: \"4406b9a5-a7b3-4469-9760-f3dcbec872a0\") " Oct 30 09:39:32.528683 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:32.528523 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4406b9a5-a7b3-4469-9760-f3dcbec872a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4406b9a5-a7b3-4469-9760-f3dcbec872a0-istio-envoy\") pod \"4406b9a5-a7b3-4469-9760-f3dcbec872a0\" (UID: \"4406b9a5-a7b3-4469-9760-f3dcbec872a0\") " Oct 30 09:39:32.528683 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:32.528619 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4406b9a5-a7b3-4469-9760-f3dcbec872a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4406b9a5-a7b3-4469-9760-f3dcbec872a0-workload-socket\") pod \"4406b9a5-a7b3-4469-9760-f3dcbec872a0\" (UID: \"4406b9a5-a7b3-4469-9760-f3dcbec872a0\") " Oct 30 09:39:32.528683 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:32.528661 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4406b9a5-a7b3-4469-9760-f3dcbec872a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4406b9a5-a7b3-4469-9760-f3dcbec872a0-workload-certs\") pod \"4406b9a5-a7b3-4469-9760-f3dcbec872a0\" (UID: \"4406b9a5-a7b3-4469-9760-f3dcbec872a0\") " Oct 30 09:39:32.528876 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:32.528691 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4406b9a5-a7b3-4469-9760-f3dcbec872a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4406b9a5-a7b3-4469-9760-f3dcbec872a0-istio-data\") pod \"4406b9a5-a7b3-4469-9760-f3dcbec872a0\" (UID: \"4406b9a5-a7b3-4469-9760-f3dcbec872a0\") " Oct 30 09:39:32.528876 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:32.528798 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4406b9a5-a7b3-4469-9760-f3dcbec872a0-istio-token\" (UniqueName: \"kubernetes.io/projected/4406b9a5-a7b3-4469-9760-f3dcbec872a0-istio-token\") pod \"4406b9a5-a7b3-4469-9760-f3dcbec872a0\" (UID: \"4406b9a5-a7b3-4469-9760-f3dcbec872a0\") " Oct 30 09:39:32.528997 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:32.528945 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4406b9a5-a7b3-4469-9760-f3dcbec872a0-workload-certs" pod "4406b9a5-a7b3-4469-9760-f3dcbec872a0" (UID: "4406b9a5-a7b3-4469-9760-f3dcbec872a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:39:32.529062 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:32.528994 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4406b9a5-a7b3-4469-9760-f3dcbec872a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4406b9a5-a7b3-4469-9760-f3dcbec872a0-istio-podinfo\") pod \"4406b9a5-a7b3-4469-9760-f3dcbec872a0\" (UID: \"4406b9a5-a7b3-4469-9760-f3dcbec872a0\") " Oct 30 09:39:32.529062 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:32.529014 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4406b9a5-a7b3-4469-9760-f3dcbec872a0-istio-data" pod "4406b9a5-a7b3-4469-9760-f3dcbec872a0" (UID: "4406b9a5-a7b3-4469-9760-f3dcbec872a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:39:32.529062 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:32.529039 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4406b9a5-a7b3-4469-9760-f3dcbec872a0-workload-socket" pod "4406b9a5-a7b3-4469-9760-f3dcbec872a0" (UID: "4406b9a5-a7b3-4469-9760-f3dcbec872a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:39:32.529213 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:32.529075 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4406b9a5-a7b3-4469-9760-f3dcbec872a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4406b9a5-a7b3-4469-9760-f3dcbec872a0-istiod-ca-cert\") pod \"4406b9a5-a7b3-4469-9760-f3dcbec872a0\" (UID: \"4406b9a5-a7b3-4469-9760-f3dcbec872a0\") " Oct 30 09:39:32.529266 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:32.529210 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4406b9a5-a7b3-4469-9760-f3dcbec872a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4406b9a5-a7b3-4469-9760-f3dcbec872a0-credential-socket\") pod \"4406b9a5-a7b3-4469-9760-f3dcbec872a0\" (UID: \"4406b9a5-a7b3-4469-9760-f3dcbec872a0\") " Oct 30 09:39:32.529538 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:32.529510 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4406b9a5-a7b3-4469-9760-f3dcbec872a0-credential-socket" pod "4406b9a5-a7b3-4469-9760-f3dcbec872a0" (UID: "4406b9a5-a7b3-4469-9760-f3dcbec872a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:39:32.529538 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:32.529532 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4406b9a5-a7b3-4469-9760-f3dcbec872a0-istiod-ca-cert" pod "4406b9a5-a7b3-4469-9760-f3dcbec872a0" (UID: "4406b9a5-a7b3-4469-9760-f3dcbec872a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:39:32.529881 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:32.529862 4562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4406b9a5-a7b3-4469-9760-f3dcbec872a0-istiod-ca-cert\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:39:32.529964 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:32.529881 4562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4406b9a5-a7b3-4469-9760-f3dcbec872a0-credential-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:39:32.529964 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:32.529893 4562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4406b9a5-a7b3-4469-9760-f3dcbec872a0-workload-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:39:32.529964 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:32.529902 4562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4406b9a5-a7b3-4469-9760-f3dcbec872a0-workload-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:39:32.529964 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:32.529910 4562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4406b9a5-a7b3-4469-9760-f3dcbec872a0-istio-data\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:39:32.531272 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:32.531241 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4406b9a5-a7b3-4469-9760-f3dcbec872a0-istio-podinfo" pod "4406b9a5-a7b3-4469-9760-f3dcbec872a0" (UID: "4406b9a5-a7b3-4469-9760-f3dcbec872a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:39:32.531530 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:32.531358 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4406b9a5-a7b3-4469-9760-f3dcbec872a0-kube-api-access-mjbpv" pod "4406b9a5-a7b3-4469-9760-f3dcbec872a0" (UID: "4406b9a5-a7b3-4469-9760-f3dcbec872a0"). InnerVolumeSpecName "kube-api-access-mjbpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:39:32.537799 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:32.537774 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4406b9a5-a7b3-4469-9760-f3dcbec872a0-istio-envoy" pod "4406b9a5-a7b3-4469-9760-f3dcbec872a0" (UID: "4406b9a5-a7b3-4469-9760-f3dcbec872a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:39:32.538985 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:32.538960 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4406b9a5-a7b3-4469-9760-f3dcbec872a0-istio-token" pod "4406b9a5-a7b3-4469-9760-f3dcbec872a0" (UID: "4406b9a5-a7b3-4469-9760-f3dcbec872a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:39:32.631306 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:32.631273 4562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4406b9a5-a7b3-4469-9760-f3dcbec872a0-istio-podinfo\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:39:32.631306 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:32.631305 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjbpv\" (UniqueName: \"kubernetes.io/projected/4406b9a5-a7b3-4469-9760-f3dcbec872a0-kube-api-access-mjbpv\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:39:32.631306 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:32.631315 4562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4406b9a5-a7b3-4469-9760-f3dcbec872a0-istio-envoy\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:39:32.631568 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:32.631324 4562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4406b9a5-a7b3-4469-9760-f3dcbec872a0-istio-token\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:39:32.696068 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:32.696033 4562 generic.go:356] "Generic (PLEG): container finished" podID="4406b9a5-a7b3-4469-9760-f3dcbec872a0" containerID="f99be65fe664122bff556172bc3db3958f832b2ba284800e0b8874822ae34f9a" exitCode=0 Oct 30 09:39:32.696251 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:32.696091 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp" Oct 30 09:39:32.696251 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:32.696122 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp" event={"ID":"4406b9a5-a7b3-4469-9760-f3dcbec872a0","Type":"ContainerDied","Data":"f99be65fe664122bff556172bc3db3958f832b2ba284800e0b8874822ae34f9a"} Oct 30 09:39:32.696251 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:32.696177 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp" event={"ID":"4406b9a5-a7b3-4469-9760-f3dcbec872a0","Type":"ContainerDied","Data":"681b7c8a8dcb365c527c8db896989adf75aeace13faf939a8b0d237a0203d9a6"} Oct 30 09:39:32.696251 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:32.696201 4562 scope.go:122] "RemoveContainer" containerID="f99be65fe664122bff556172bc3db3958f832b2ba284800e0b8874822ae34f9a" Oct 30 09:39:32.708130 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:32.708112 4562 scope.go:122] "RemoveContainer" containerID="f99be65fe664122bff556172bc3db3958f832b2ba284800e0b8874822ae34f9a" Oct 30 09:39:32.708495 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:39:32.708374 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f99be65fe664122bff556172bc3db3958f832b2ba284800e0b8874822ae34f9a\": container with ID starting with f99be65fe664122bff556172bc3db3958f832b2ba284800e0b8874822ae34f9a not found: ID does not exist" containerID="f99be65fe664122bff556172bc3db3958f832b2ba284800e0b8874822ae34f9a" Oct 30 09:39:32.708495 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:32.708476 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f99be65fe664122bff556172bc3db3958f832b2ba284800e0b8874822ae34f9a"} err="failed to get container status \"f99be65fe664122bff556172bc3db3958f832b2ba284800e0b8874822ae34f9a\": rpc error: code = NotFound desc = could not find container \"f99be65fe664122bff556172bc3db3958f832b2ba284800e0b8874822ae34f9a\": container with ID starting with f99be65fe664122bff556172bc3db3958f832b2ba284800e0b8874822ae34f9a not found: ID does not exist" Oct 30 09:39:32.753060 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:32.753034 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp"] Oct 30 09:39:32.760339 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:32.760317 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-hdphp"] Oct 30 09:39:33.876193 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:33.876161 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4406b9a5-a7b3-4469-9760-f3dcbec872a0" path="/var/lib/kubelet/pods/4406b9a5-a7b3-4469-9760-f3dcbec872a0/volumes" Oct 30 09:39:34.638871 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:34.638838 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-6d4b7a-57ddcbd6fb-wzqlg" Oct 30 09:39:44.927369 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:44.927275 4562 generic.go:356] "Generic (PLEG): container finished" podID="54cc6629-e1ca-4af7-9455-80e3b4d097f6" containerID="21fc9cf91ea050cde727c5f8eb885da8d61174f5d4c3cd0b9bf497ac5f65a86a" exitCode=0 Oct 30 09:39:44.927369 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:44.927353 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk" event={"ID":"54cc6629-e1ca-4af7-9455-80e3b4d097f6","Type":"ContainerDied","Data":"21fc9cf91ea050cde727c5f8eb885da8d61174f5d4c3cd0b9bf497ac5f65a86a"} Oct 30 09:39:44.927882 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:44.927779 4562 scope.go:122] "RemoveContainer" containerID="21fc9cf91ea050cde727c5f8eb885da8d61174f5d4c3cd0b9bf497ac5f65a86a" Oct 30 09:39:44.929596 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:44.929558 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:39:44.929731 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:44.929631 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:39:44.929731 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:44.929668 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:39:45.936286 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:45.936250 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk" event={"ID":"54cc6629-e1ca-4af7-9455-80e3b4d097f6","Type":"ContainerStarted","Data":"fa62c69fd900306ab5356014658f747fc8cdcdca034f0dd8bb719977a5af299c"} Oct 30 09:39:46.076436 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:46.072526 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk" Oct 30 09:39:46.076436 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:46.076140 4562 patch_prober.go:29] interesting pod/e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Oct 30 09:39:46.076436 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:46.076193 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk" podUID="54cc6629-e1ca-4af7-9455-80e3b4d097f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Oct 30 09:39:47.068422 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:47.067802 4562 patch_prober.go:29] interesting pod/e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Oct 30 09:39:47.068422 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:47.067863 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk" podUID="54cc6629-e1ca-4af7-9455-80e3b4d097f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Oct 30 09:39:48.058456 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:48.054105 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk" Oct 30 09:39:48.079555 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:48.079518 4562 patch_prober.go:29] interesting pod/e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Oct 30 09:39:48.084048 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:48.084010 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk" podUID="54cc6629-e1ca-4af7-9455-80e3b4d097f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Oct 30 09:39:49.066371 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:49.066305 4562 patch_prober.go:29] interesting pod/e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Oct 30 09:39:49.066642 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:49.066423 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk" podUID="54cc6629-e1ca-4af7-9455-80e3b4d097f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Oct 30 09:39:50.068794 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:50.068761 4562 patch_prober.go:29] interesting pod/e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Oct 30 09:39:50.069337 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:50.069303 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk" podUID="54cc6629-e1ca-4af7-9455-80e3b4d097f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Oct 30 09:39:50.474671 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:50.474630 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-3d9b36-5bcbf74867-k9hvd"] Oct 30 09:39:50.476444 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:50.476414 4562 cpu_manager.go:414] "containerMap: removing container" podUID="4406b9a5-a7b3-4469-9760-f3dcbec872a0" containerName="istio-proxy" Oct 30 09:39:50.476621 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:50.476454 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4406b9a5-a7b3-4469-9760-f3dcbec872a0" containerName="istio-proxy" Oct 30 09:39:50.476621 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:50.476567 4562 cpu_manager.go:414] "containerMap: removing container" podUID="75eb9d8e-5d05-47d3-b923-06404edb57b2" containerName="main" Oct 30 09:39:50.476621 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:50.476576 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75eb9d8e-5d05-47d3-b923-06404edb57b2" containerName="main" Oct 30 09:39:50.476938 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:50.476920 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="4406b9a5-a7b3-4469-9760-f3dcbec872a0" containerName="istio-proxy" Oct 30 09:39:50.476997 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:50.476968 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="75eb9d8e-5d05-47d3-b923-06404edb57b2" containerName="main" Oct 30 09:39:50.483250 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:50.483228 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3d9b36-5bcbf74867-k9hvd" Oct 30 09:39:50.495229 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:50.495201 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-3d9b36-5bcbf74867-k9hvd"] Oct 30 09:39:50.504549 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:50.504518 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-3d9b36\"" Oct 30 09:39:50.507513 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:50.504640 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-3d9b36-dockercfg-g2pcw\"" Oct 30 09:39:50.579805 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:50.579772 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-3d9b36-5cf56c76bd-c976p"] Oct 30 09:39:50.612398 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:50.612339 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0dbb750-1c20-41a5-b33f-5784a1f683d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-3d9b36-5bcbf74867-k9hvd\" (UID: \"c0dbb750-1c20-41a5-b33f-5784a1f683d9\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3d9b36-5bcbf74867-k9hvd" Oct 30 09:39:50.612885 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:50.612850 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnx5k\" (UniqueName: \"kubernetes.io/projected/c0dbb750-1c20-41a5-b33f-5784a1f683d9-kube-api-access-vnx5k\") pod \"payload-pre-processing-e2e-extmod-b-3d9b36-5bcbf74867-k9hvd\" (UID: \"c0dbb750-1c20-41a5-b33f-5784a1f683d9\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3d9b36-5bcbf74867-k9hvd" Oct 30 09:39:50.615880 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:50.615857 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-3d9b36-5cf56c76bd-c976p" Oct 30 09:39:50.774475 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:50.764591 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0dbb750-1c20-41a5-b33f-5784a1f683d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-3d9b36-5bcbf74867-k9hvd\" (UID: \"c0dbb750-1c20-41a5-b33f-5784a1f683d9\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3d9b36-5bcbf74867-k9hvd" Oct 30 09:39:50.774475 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:50.764738 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnx5k\" (UniqueName: \"kubernetes.io/projected/c0dbb750-1c20-41a5-b33f-5784a1f683d9-kube-api-access-vnx5k\") pod \"payload-pre-processing-e2e-extmod-b-3d9b36-5bcbf74867-k9hvd\" (UID: \"c0dbb750-1c20-41a5-b33f-5784a1f683d9\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3d9b36-5bcbf74867-k9hvd" Oct 30 09:39:50.774475 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:50.764835 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d696e344-8d35-4896-84dc-9a4dc225f291-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-3d9b36-5cf56c76bd-c976p\" (UID: \"d696e344-8d35-4896-84dc-9a4dc225f291\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-3d9b36-5cf56c76bd-c976p" Oct 30 09:39:50.774475 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:50.764936 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q55n9\" (UniqueName: \"kubernetes.io/projected/d696e344-8d35-4896-84dc-9a4dc225f291-kube-api-access-q55n9\") pod \"payload-processing-e2e-extmod-b-3d9b36-5cf56c76bd-c976p\" (UID: \"d696e344-8d35-4896-84dc-9a4dc225f291\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-3d9b36-5cf56c76bd-c976p" Oct 30 09:39:50.774475 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:50.766219 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0dbb750-1c20-41a5-b33f-5784a1f683d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-3d9b36-5bcbf74867-k9hvd\" (UID: \"c0dbb750-1c20-41a5-b33f-5784a1f683d9\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3d9b36-5bcbf74867-k9hvd" Oct 30 09:39:50.804478 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:50.800529 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-3d9b36-5cf56c76bd-c976p"] Oct 30 09:39:50.804478 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:50.801854 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnx5k\" (UniqueName: \"kubernetes.io/projected/c0dbb750-1c20-41a5-b33f-5784a1f683d9-kube-api-access-vnx5k\") pod \"payload-pre-processing-e2e-extmod-b-3d9b36-5bcbf74867-k9hvd\" (UID: \"c0dbb750-1c20-41a5-b33f-5784a1f683d9\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3d9b36-5bcbf74867-k9hvd" Oct 30 09:39:50.865869 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:50.865829 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q55n9\" (UniqueName: \"kubernetes.io/projected/d696e344-8d35-4896-84dc-9a4dc225f291-kube-api-access-q55n9\") pod \"payload-processing-e2e-extmod-b-3d9b36-5cf56c76bd-c976p\" (UID: \"d696e344-8d35-4896-84dc-9a4dc225f291\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-3d9b36-5cf56c76bd-c976p" Oct 30 09:39:50.866290 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:50.866271 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d696e344-8d35-4896-84dc-9a4dc225f291-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-3d9b36-5cf56c76bd-c976p\" (UID: \"d696e344-8d35-4896-84dc-9a4dc225f291\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-3d9b36-5cf56c76bd-c976p" Oct 30 09:39:50.867259 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:50.867235 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d696e344-8d35-4896-84dc-9a4dc225f291-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-3d9b36-5cf56c76bd-c976p\" (UID: \"d696e344-8d35-4896-84dc-9a4dc225f291\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-3d9b36-5cf56c76bd-c976p" Oct 30 09:39:50.887938 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:50.887914 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q55n9\" (UniqueName: \"kubernetes.io/projected/d696e344-8d35-4896-84dc-9a4dc225f291-kube-api-access-q55n9\") pod \"payload-processing-e2e-extmod-b-3d9b36-5cf56c76bd-c976p\" (UID: \"d696e344-8d35-4896-84dc-9a4dc225f291\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-3d9b36-5cf56c76bd-c976p" Oct 30 09:39:51.066911 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:51.064911 4562 patch_prober.go:29] interesting pod/e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Oct 30 09:39:51.066911 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:51.064974 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk" podUID="54cc6629-e1ca-4af7-9455-80e3b4d097f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Oct 30 09:39:51.100609 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:51.100582 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3d9b36-5bcbf74867-k9hvd" Oct 30 09:39:51.132603 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:51.132575 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-3d9b36-5cf56c76bd-c976p" Oct 30 09:39:51.475113 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:51.475074 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-3d9b36-5bcbf74867-k9hvd"] Oct 30 09:39:51.575779 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:39:51.575752 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd696e344_8d35_4896_84dc_9a4dc225f291.slice/crio-57e38bc4eefd3a27221220af629641496de6f07577ce1e83b200de0c0427e191 WatchSource:0}: Error finding container 57e38bc4eefd3a27221220af629641496de6f07577ce1e83b200de0c0427e191: Status 404 returned error can't find the container with id 57e38bc4eefd3a27221220af629641496de6f07577ce1e83b200de0c0427e191 Oct 30 09:39:51.577374 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:51.577353 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-3d9b36-5cf56c76bd-c976p"] Oct 30 09:39:52.036209 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:52.036166 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-3d9b36-5cf56c76bd-c976p" event={"ID":"d696e344-8d35-4896-84dc-9a4dc225f291","Type":"ContainerStarted","Data":"38db6254771c4c209929a8a6e6928f07fddcbedfc000d88ec0a81e643716de4d"} Oct 30 09:39:52.036441 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:52.036234 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-3d9b36-5cf56c76bd-c976p" event={"ID":"d696e344-8d35-4896-84dc-9a4dc225f291","Type":"ContainerStarted","Data":"57e38bc4eefd3a27221220af629641496de6f07577ce1e83b200de0c0427e191"} Oct 30 09:39:52.036441 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:52.036331 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-3d9b36-5cf56c76bd-c976p" Oct 30 09:39:52.038074 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:52.038045 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3d9b36-5bcbf74867-k9hvd" event={"ID":"c0dbb750-1c20-41a5-b33f-5784a1f683d9","Type":"ContainerStarted","Data":"ba2b86702a749f28cc495af83ae66f51e321838468d53bb897e6adf50741d459"} Oct 30 09:39:52.038217 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:52.038094 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3d9b36-5bcbf74867-k9hvd" event={"ID":"c0dbb750-1c20-41a5-b33f-5784a1f683d9","Type":"ContainerStarted","Data":"804e66c52e1df89f4644f08468788a5ae2148a079e5c80e3b00d48ec101da041"} Oct 30 09:39:52.038840 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:52.038820 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3d9b36-5bcbf74867-k9hvd" Oct 30 09:39:52.081259 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:52.081182 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-3d9b36-5cf56c76bd-c976p" podStartSLOduration=2.081162567 podStartE2EDuration="2.081162567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:39:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:39:52.078863933 +0000 UTC m=+2752.340424921" watchObservedRunningTime="2026-10-30 09:39:52.081162567 +0000 UTC m=+2752.342723560" Oct 30 09:39:52.126088 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:52.126033 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3d9b36-5bcbf74867-k9hvd" podStartSLOduration=2.126018928 podStartE2EDuration="2.126018928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:39:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:39:52.123709283 +0000 UTC m=+2752.385270272" watchObservedRunningTime="2026-10-30 09:39:52.126018928 +0000 UTC m=+2752.387579916" Oct 30 09:39:52.161239 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:52.161146 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk" Oct 30 09:39:52.163639 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:39:52.163612 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk" Oct 30 09:39:58.598038 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:39:58.598000 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:40:03.056898 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:03.056863 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-3d9b36-5cf56c76bd-c976p" Oct 30 09:40:04.055454 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:04.055425 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3d9b36-5bcbf74867-k9hvd" Oct 30 09:40:16.241837 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:16.241752 4562 generic.go:356] "Generic (PLEG): container finished" podID="18686e38-6651-454d-b9ad-996ffebe2b91" containerID="8164ea6956edc7d2e43109737ac0f7bfbe74b625b3c510d1307756202a96688c" exitCode=0 Oct 30 09:40:16.242246 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:16.241826 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24" event={"ID":"18686e38-6651-454d-b9ad-996ffebe2b91","Type":"ContainerDied","Data":"8164ea6956edc7d2e43109737ac0f7bfbe74b625b3c510d1307756202a96688c"} Oct 30 09:40:16.242246 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:16.242242 4562 scope.go:122] "RemoveContainer" containerID="8164ea6956edc7d2e43109737ac0f7bfbe74b625b3c510d1307756202a96688c" Oct 30 09:40:16.244068 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:16.244036 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:40:16.244193 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:16.244125 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:40:16.244193 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:16.244176 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:40:17.279979 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:17.279935 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24" event={"ID":"18686e38-6651-454d-b9ad-996ffebe2b91","Type":"ContainerStarted","Data":"c41e759d6fc5c724727ce44f55e9a2177cb97954f052d3db913c105f08f52e26"} Oct 30 09:40:17.635252 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:17.635162 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24" Oct 30 09:40:17.635451 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:17.635320 4562 patch_prober.go:29] interesting pod/e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Oct 30 09:40:17.635451 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:17.635371 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24" podUID="18686e38-6651-454d-b9ad-996ffebe2b91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Oct 30 09:40:18.635205 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:18.635169 4562 patch_prober.go:29] interesting pod/e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Oct 30 09:40:18.635629 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:18.635233 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24" podUID="18686e38-6651-454d-b9ad-996ffebe2b91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Oct 30 09:40:19.635555 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:19.635517 4562 patch_prober.go:29] interesting pod/e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Oct 30 09:40:19.635991 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:19.635588 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24" podUID="18686e38-6651-454d-b9ad-996ffebe2b91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Oct 30 09:40:20.634708 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:20.634672 4562 patch_prober.go:29] interesting pod/e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Oct 30 09:40:20.634904 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:20.634746 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24" podUID="18686e38-6651-454d-b9ad-996ffebe2b91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Oct 30 09:40:21.635087 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:21.635047 4562 patch_prober.go:29] interesting pod/e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Oct 30 09:40:21.635529 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:21.635101 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24" podUID="18686e38-6651-454d-b9ad-996ffebe2b91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Oct 30 09:40:22.635432 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:22.635364 4562 patch_prober.go:29] interesting pod/e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Oct 30 09:40:22.635831 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:22.635449 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24" podUID="18686e38-6651-454d-b9ad-996ffebe2b91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Oct 30 09:40:23.639781 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:23.639743 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24" Oct 30 09:40:23.640250 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:23.640228 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24" Oct 30 09:40:23.640947 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:23.640928 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24" Oct 30 09:40:24.262721 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:24.262681 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-3d9b36-59cf455fb5-qhp85"] Oct 30 09:40:24.274697 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:24.274279 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3d9b36-59cf455fb5-qhp85" Oct 30 09:40:24.287723 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:24.287677 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-3d9b36-59cf455fb5-qhp85"] Oct 30 09:40:24.297674 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:24.297644 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-3d9b36-serving-cert\"" Oct 30 09:40:24.496405 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:24.453496 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbdrc\" (UniqueName: \"kubernetes.io/projected/0ecc2c25-ad9a-438b-9b2c-ab6e98905576-kube-api-access-wbdrc\") pod \"maas-api-e2e-extmod-b-3d9b36-59cf455fb5-qhp85\" (UID: \"0ecc2c25-ad9a-438b-9b2c-ab6e98905576\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3d9b36-59cf455fb5-qhp85" Oct 30 09:40:24.497661 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:24.497632 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ecc2c25-ad9a-438b-9b2c-ab6e98905576-metrics-certs\") pod \"maas-api-e2e-extmod-b-3d9b36-59cf455fb5-qhp85\" (UID: \"0ecc2c25-ad9a-438b-9b2c-ab6e98905576\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3d9b36-59cf455fb5-qhp85" Oct 30 09:40:24.498018 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:24.497997 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ecc2c25-ad9a-438b-9b2c-ab6e98905576-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-3d9b36-59cf455fb5-qhp85\" (UID: \"0ecc2c25-ad9a-438b-9b2c-ab6e98905576\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3d9b36-59cf455fb5-qhp85" Oct 30 09:40:24.498246 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:24.498232 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ecc2c25-ad9a-438b-9b2c-ab6e98905576-maas-api-tls\") pod \"maas-api-e2e-extmod-b-3d9b36-59cf455fb5-qhp85\" (UID: \"0ecc2c25-ad9a-438b-9b2c-ab6e98905576\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3d9b36-59cf455fb5-qhp85" Oct 30 09:40:24.599852 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:24.599757 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ecc2c25-ad9a-438b-9b2c-ab6e98905576-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-3d9b36-59cf455fb5-qhp85\" (UID: \"0ecc2c25-ad9a-438b-9b2c-ab6e98905576\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3d9b36-59cf455fb5-qhp85" Oct 30 09:40:24.600017 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:24.599856 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ecc2c25-ad9a-438b-9b2c-ab6e98905576-maas-api-tls\") pod \"maas-api-e2e-extmod-b-3d9b36-59cf455fb5-qhp85\" (UID: \"0ecc2c25-ad9a-438b-9b2c-ab6e98905576\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3d9b36-59cf455fb5-qhp85" Oct 30 09:40:24.600017 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:24.599977 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbdrc\" (UniqueName: \"kubernetes.io/projected/0ecc2c25-ad9a-438b-9b2c-ab6e98905576-kube-api-access-wbdrc\") pod \"maas-api-e2e-extmod-b-3d9b36-59cf455fb5-qhp85\" (UID: \"0ecc2c25-ad9a-438b-9b2c-ab6e98905576\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3d9b36-59cf455fb5-qhp85" Oct 30 09:40:24.600017 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:40:24.600007 4562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-3d9b36-serving-cert: secret "maas-api-e2e-extmod-b-3d9b36-serving-cert" not found Oct 30 09:40:24.600188 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:40:24.600090 4562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ecc2c25-ad9a-438b-9b2c-ab6e98905576-maas-api-tls podName:0ecc2c25-ad9a-438b-9b2c-ab6e98905576 nodeName:}" failed. No retries permitted until 2026-10-30 09:40:25.100069138 +0000 UTC m=+2785.361630115 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0ecc2c25-ad9a-438b-9b2c-ab6e98905576-maas-api-tls") pod "maas-api-e2e-extmod-b-3d9b36-59cf455fb5-qhp85" (UID: "0ecc2c25-ad9a-438b-9b2c-ab6e98905576") : secret "maas-api-e2e-extmod-b-3d9b36-serving-cert" not found Oct 30 09:40:24.600188 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:24.600013 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ecc2c25-ad9a-438b-9b2c-ab6e98905576-metrics-certs\") pod \"maas-api-e2e-extmod-b-3d9b36-59cf455fb5-qhp85\" (UID: \"0ecc2c25-ad9a-438b-9b2c-ab6e98905576\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3d9b36-59cf455fb5-qhp85" Oct 30 09:40:24.600645 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:24.600620 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ecc2c25-ad9a-438b-9b2c-ab6e98905576-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-3d9b36-59cf455fb5-qhp85\" (UID: \"0ecc2c25-ad9a-438b-9b2c-ab6e98905576\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3d9b36-59cf455fb5-qhp85" Oct 30 09:40:24.602616 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:24.602591 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ecc2c25-ad9a-438b-9b2c-ab6e98905576-metrics-certs\") pod \"maas-api-e2e-extmod-b-3d9b36-59cf455fb5-qhp85\" (UID: \"0ecc2c25-ad9a-438b-9b2c-ab6e98905576\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3d9b36-59cf455fb5-qhp85" Oct 30 09:40:24.622940 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:24.622910 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbdrc\" (UniqueName: \"kubernetes.io/projected/0ecc2c25-ad9a-438b-9b2c-ab6e98905576-kube-api-access-wbdrc\") pod \"maas-api-e2e-extmod-b-3d9b36-59cf455fb5-qhp85\" (UID: \"0ecc2c25-ad9a-438b-9b2c-ab6e98905576\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3d9b36-59cf455fb5-qhp85" Oct 30 09:40:25.106572 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:25.106538 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ecc2c25-ad9a-438b-9b2c-ab6e98905576-maas-api-tls\") pod \"maas-api-e2e-extmod-b-3d9b36-59cf455fb5-qhp85\" (UID: \"0ecc2c25-ad9a-438b-9b2c-ab6e98905576\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3d9b36-59cf455fb5-qhp85" Oct 30 09:40:25.109208 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:25.109181 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ecc2c25-ad9a-438b-9b2c-ab6e98905576-maas-api-tls\") pod \"maas-api-e2e-extmod-b-3d9b36-59cf455fb5-qhp85\" (UID: \"0ecc2c25-ad9a-438b-9b2c-ab6e98905576\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3d9b36-59cf455fb5-qhp85" Oct 30 09:40:25.257055 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:25.257018 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3d9b36-59cf455fb5-qhp85" Oct 30 09:40:25.660408 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:25.649082 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-3d9b36-59cf455fb5-qhp85"] Oct 30 09:40:25.707029 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:40:25.706993 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ecc2c25_ad9a_438b_9b2c_ab6e98905576.slice/crio-d82ec7a2bc52df9d8f2b1e71de89e6b55b7e3c71ed20ce5e10068f7742a8e866 WatchSource:0}: Error finding container d82ec7a2bc52df9d8f2b1e71de89e6b55b7e3c71ed20ce5e10068f7742a8e866: Status 404 returned error can't find the container with id d82ec7a2bc52df9d8f2b1e71de89e6b55b7e3c71ed20ce5e10068f7742a8e866 Oct 30 09:40:26.356929 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:26.356888 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3d9b36-59cf455fb5-qhp85" event={"ID":"0ecc2c25-ad9a-438b-9b2c-ab6e98905576","Type":"ContainerStarted","Data":"6c84dd2037088c60cac8c86cda595e0be39816f97cdb74ead68954a2b4046f5f"} Oct 30 09:40:26.357443 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:26.356946 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3d9b36-59cf455fb5-qhp85" event={"ID":"0ecc2c25-ad9a-438b-9b2c-ab6e98905576","Type":"ContainerStarted","Data":"d82ec7a2bc52df9d8f2b1e71de89e6b55b7e3c71ed20ce5e10068f7742a8e866"} Oct 30 09:40:26.357443 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:26.356984 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3d9b36-59cf455fb5-qhp85" Oct 30 09:40:26.422769 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:26.422620 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3d9b36-59cf455fb5-qhp85" podStartSLOduration=1.999093105 podStartE2EDuration="2.41179809s" totalImagesPullingTime="412.704985ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:40:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:40:26.397786456 +0000 UTC m=+2786.659347449" watchObservedRunningTime="2026-10-30 09:40:26.41179809 +0000 UTC m=+2786.673359079" Oct 30 09:40:28.756909 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:40:28.756865 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:40:32.369436 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:32.369370 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3d9b36-59cf455fb5-qhp85" Oct 30 09:40:35.569854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:35.569809 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-f58f32-664f5dc478-zb7dn"] Oct 30 09:40:35.596560 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:35.596515 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-f58f32-664f5dc478-zb7dn"] Oct 30 09:40:35.596756 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:35.596617 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-f58f32-664f5dc478-zb7dn" Oct 30 09:40:35.614210 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:35.614182 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-f58f32\"" Oct 30 09:40:35.614885 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:35.614861 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-f58f32-dockercfg-t9h82\"" Oct 30 09:40:35.621733 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:35.621708 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83e4cd09-2917-4dcb-b67e-13b088428c02-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-f58f32-664f5dc478-zb7dn\" (UID: \"83e4cd09-2917-4dcb-b67e-13b088428c02\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-f58f32-664f5dc478-zb7dn" Oct 30 09:40:35.621896 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:35.621872 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5djrd\" (UniqueName: \"kubernetes.io/projected/83e4cd09-2917-4dcb-b67e-13b088428c02-kube-api-access-5djrd\") pod \"payload-processing-e2e-extmod-b-f58f32-664f5dc478-zb7dn\" (UID: \"83e4cd09-2917-4dcb-b67e-13b088428c02\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-f58f32-664f5dc478-zb7dn" Oct 30 09:40:35.723726 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:35.723689 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83e4cd09-2917-4dcb-b67e-13b088428c02-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-f58f32-664f5dc478-zb7dn\" (UID: \"83e4cd09-2917-4dcb-b67e-13b088428c02\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-f58f32-664f5dc478-zb7dn" Oct 30 09:40:35.723892 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:35.723838 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5djrd\" (UniqueName: \"kubernetes.io/projected/83e4cd09-2917-4dcb-b67e-13b088428c02-kube-api-access-5djrd\") pod \"payload-processing-e2e-extmod-b-f58f32-664f5dc478-zb7dn\" (UID: \"83e4cd09-2917-4dcb-b67e-13b088428c02\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-f58f32-664f5dc478-zb7dn" Oct 30 09:40:35.724510 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:35.724491 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83e4cd09-2917-4dcb-b67e-13b088428c02-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-f58f32-664f5dc478-zb7dn\" (UID: \"83e4cd09-2917-4dcb-b67e-13b088428c02\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-f58f32-664f5dc478-zb7dn" Oct 30 09:40:35.746956 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:35.746924 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5djrd\" (UniqueName: \"kubernetes.io/projected/83e4cd09-2917-4dcb-b67e-13b088428c02-kube-api-access-5djrd\") pod \"payload-processing-e2e-extmod-b-f58f32-664f5dc478-zb7dn\" (UID: \"83e4cd09-2917-4dcb-b67e-13b088428c02\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-f58f32-664f5dc478-zb7dn" Oct 30 09:40:35.920669 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:35.920580 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-f58f32-664f5dc478-zb7dn" Oct 30 09:40:36.221799 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:40:36.221765 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83e4cd09_2917_4dcb_b67e_13b088428c02.slice/crio-e8ae1ffb40b132fbaa6b9eaa395ef5b1562847a98945f0cd306919dc53e7d330 WatchSource:0}: Error finding container e8ae1ffb40b132fbaa6b9eaa395ef5b1562847a98945f0cd306919dc53e7d330: Status 404 returned error can't find the container with id e8ae1ffb40b132fbaa6b9eaa395ef5b1562847a98945f0cd306919dc53e7d330 Oct 30 09:40:36.224015 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:36.223988 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-f58f32-664f5dc478-zb7dn"] Oct 30 09:40:36.552317 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:36.552210 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-f58f32-664f5dc478-zb7dn" event={"ID":"83e4cd09-2917-4dcb-b67e-13b088428c02","Type":"ContainerStarted","Data":"bbfcfd1be1f0ea448c958554dd4e441b92c88bba449d4dd44380e49d1f82c2e4"} Oct 30 09:40:36.552317 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:36.552313 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-f58f32-664f5dc478-zb7dn" event={"ID":"83e4cd09-2917-4dcb-b67e-13b088428c02","Type":"ContainerStarted","Data":"e8ae1ffb40b132fbaa6b9eaa395ef5b1562847a98945f0cd306919dc53e7d330"} Oct 30 09:40:36.553511 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:36.553484 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-f58f32-664f5dc478-zb7dn" Oct 30 09:40:36.596555 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:36.596503 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-f58f32-664f5dc478-zb7dn" podStartSLOduration=1.596485438 podStartE2EDuration="1.596485438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:40:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:40:36.593301237 +0000 UTC m=+2796.854862245" watchObservedRunningTime="2026-10-30 09:40:36.596485438 +0000 UTC m=+2796.858046427" Oct 30 09:40:48.599667 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:48.599640 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-f58f32-664f5dc478-zb7dn" Oct 30 09:40:58.599993 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:40:58.599958 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:40:58.640149 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:58.640112 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-035d7d/payload-processing-external-model-e2e-extmod-b-035d7d-8675dchql"] Oct 30 09:40:58.646225 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:58.646204 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-035d7d/payload-processing-external-model-e2e-extmod-b-035d7d-8675dchql" Oct 30 09:40:58.663264 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:58.663239 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-035d7d/payload-processing-external-model-e2e-extmod-b-035d7d-8675dchql"] Oct 30 09:40:58.671835 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:58.671813 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-035d7d\"/\"payload-processing-external-model-e2e--0e40e469-dockercfg-jhqzj\"" Oct 30 09:40:58.671995 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:58.671960 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-035d7d\"/\"agc-cred-6bc5af3a42ea\"" Oct 30 09:40:58.672111 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:58.672054 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-035d7d\"/\"kube-root-ca.crt\"" Oct 30 09:40:58.672216 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:58.672162 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-035d7d\"/\"openshift-service-ca.crt\"" Oct 30 09:40:58.677030 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:58.677012 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-035d7d\"/\"payload-processing-external-model-plugins-e2e-extmod-b-035d7d\"" Oct 30 09:40:58.677136 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:58.677062 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-035d7d\"/\"routing-overlay\"" Oct 30 09:40:58.696692 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:58.696661 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb6eefc6-9548-4c9c-bbe3-4bef240b4716-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-035d7d-8675dchql\" (UID: \"bb6eefc6-9548-4c9c-bbe3-4bef240b4716\") " pod="ai-tenant-e2e-extmod-b-035d7d/payload-processing-external-model-e2e-extmod-b-035d7d-8675dchql" Oct 30 09:40:58.696847 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:58.696763 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bb6eefc6-9548-4c9c-bbe3-4bef240b4716-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-035d7d-8675dchql\" (UID: \"bb6eefc6-9548-4c9c-bbe3-4bef240b4716\") " pod="ai-tenant-e2e-extmod-b-035d7d/payload-processing-external-model-e2e-extmod-b-035d7d-8675dchql" Oct 30 09:40:58.696847 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:58.696792 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bb6eefc6-9548-4c9c-bbe3-4bef240b4716-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-035d7d-8675dchql\" (UID: \"bb6eefc6-9548-4c9c-bbe3-4bef240b4716\") " pod="ai-tenant-e2e-extmod-b-035d7d/payload-processing-external-model-e2e-extmod-b-035d7d-8675dchql" Oct 30 09:40:58.696975 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:58.696929 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfzm6\" (UniqueName: \"kubernetes.io/projected/bb6eefc6-9548-4c9c-bbe3-4bef240b4716-kube-api-access-tfzm6\") pod \"payload-processing-external-model-e2e-extmod-b-035d7d-8675dchql\" (UID: \"bb6eefc6-9548-4c9c-bbe3-4bef240b4716\") " pod="ai-tenant-e2e-extmod-b-035d7d/payload-processing-external-model-e2e-extmod-b-035d7d-8675dchql" Oct 30 09:40:58.809408 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:58.808166 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfzm6\" (UniqueName: \"kubernetes.io/projected/bb6eefc6-9548-4c9c-bbe3-4bef240b4716-kube-api-access-tfzm6\") pod \"payload-processing-external-model-e2e-extmod-b-035d7d-8675dchql\" (UID: \"bb6eefc6-9548-4c9c-bbe3-4bef240b4716\") " pod="ai-tenant-e2e-extmod-b-035d7d/payload-processing-external-model-e2e-extmod-b-035d7d-8675dchql" Oct 30 09:40:58.809408 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:58.808323 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb6eefc6-9548-4c9c-bbe3-4bef240b4716-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-035d7d-8675dchql\" (UID: \"bb6eefc6-9548-4c9c-bbe3-4bef240b4716\") " pod="ai-tenant-e2e-extmod-b-035d7d/payload-processing-external-model-e2e-extmod-b-035d7d-8675dchql" Oct 30 09:40:58.818419 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:58.818016 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb6eefc6-9548-4c9c-bbe3-4bef240b4716-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-035d7d-8675dchql\" (UID: \"bb6eefc6-9548-4c9c-bbe3-4bef240b4716\") " pod="ai-tenant-e2e-extmod-b-035d7d/payload-processing-external-model-e2e-extmod-b-035d7d-8675dchql" Oct 30 09:40:58.830887 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:58.828572 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bb6eefc6-9548-4c9c-bbe3-4bef240b4716-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-035d7d-8675dchql\" (UID: \"bb6eefc6-9548-4c9c-bbe3-4bef240b4716\") " pod="ai-tenant-e2e-extmod-b-035d7d/payload-processing-external-model-e2e-extmod-b-035d7d-8675dchql" Oct 30 09:40:58.830887 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:58.828637 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bb6eefc6-9548-4c9c-bbe3-4bef240b4716-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-035d7d-8675dchql\" (UID: \"bb6eefc6-9548-4c9c-bbe3-4bef240b4716\") " pod="ai-tenant-e2e-extmod-b-035d7d/payload-processing-external-model-e2e-extmod-b-035d7d-8675dchql" Oct 30 09:40:58.830887 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:58.830701 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bb6eefc6-9548-4c9c-bbe3-4bef240b4716-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-035d7d-8675dchql\" (UID: \"bb6eefc6-9548-4c9c-bbe3-4bef240b4716\") " pod="ai-tenant-e2e-extmod-b-035d7d/payload-processing-external-model-e2e-extmod-b-035d7d-8675dchql" Oct 30 09:40:58.857241 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:58.857136 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfzm6\" (UniqueName: \"kubernetes.io/projected/bb6eefc6-9548-4c9c-bbe3-4bef240b4716-kube-api-access-tfzm6\") pod \"payload-processing-external-model-e2e-extmod-b-035d7d-8675dchql\" (UID: \"bb6eefc6-9548-4c9c-bbe3-4bef240b4716\") " pod="ai-tenant-e2e-extmod-b-035d7d/payload-processing-external-model-e2e-extmod-b-035d7d-8675dchql" Oct 30 09:40:58.869169 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:58.869101 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bb6eefc6-9548-4c9c-bbe3-4bef240b4716-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-035d7d-8675dchql\" (UID: \"bb6eefc6-9548-4c9c-bbe3-4bef240b4716\") " pod="ai-tenant-e2e-extmod-b-035d7d/payload-processing-external-model-e2e-extmod-b-035d7d-8675dchql" Oct 30 09:40:58.963812 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:58.962507 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-035d7d/payload-processing-external-model-e2e-extmod-b-035d7d-8675dchql" Oct 30 09:40:59.266701 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:40:59.266672 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb6eefc6_9548_4c9c_bbe3_4bef240b4716.slice/crio-1a9ce71e03bef8041872a601bfcf41427e208c65c6eb00361504867f218ed1b2 WatchSource:0}: Error finding container 1a9ce71e03bef8041872a601bfcf41427e208c65c6eb00361504867f218ed1b2: Status 404 returned error can't find the container with id 1a9ce71e03bef8041872a601bfcf41427e208c65c6eb00361504867f218ed1b2 Oct 30 09:40:59.269927 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:59.269898 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-035d7d/payload-processing-external-model-e2e-extmod-b-035d7d-8675dchql"] Oct 30 09:40:59.852814 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:59.852774 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-035d7d/payload-processing-external-model-e2e-extmod-b-035d7d-8675dchql" event={"ID":"bb6eefc6-9548-4c9c-bbe3-4bef240b4716","Type":"ContainerStarted","Data":"6823908a52d73431fcdf50506a02e476d4754d7dde12610e7b497a264c203eb4"} Oct 30 09:40:59.853252 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:59.852844 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-035d7d/payload-processing-external-model-e2e-extmod-b-035d7d-8675dchql" event={"ID":"bb6eefc6-9548-4c9c-bbe3-4bef240b4716","Type":"ContainerStarted","Data":"1a9ce71e03bef8041872a601bfcf41427e208c65c6eb00361504867f218ed1b2"} Oct 30 09:40:59.853252 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:59.852972 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-035d7d/payload-processing-external-model-e2e-extmod-b-035d7d-8675dchql" Oct 30 09:40:59.896817 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:40:59.896767 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-035d7d/payload-processing-external-model-e2e-extmod-b-035d7d-8675dchql" podStartSLOduration=1.896753425 podStartE2EDuration="1.896753425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:40:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:40:59.892602718 +0000 UTC m=+2820.154163706" watchObservedRunningTime="2026-10-30 09:40:59.896753425 +0000 UTC m=+2820.158314422" Oct 30 09:41:10.864674 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:10.864639 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-035d7d/payload-processing-external-model-e2e-extmod-b-035d7d-8675dchql" Oct 30 09:41:13.457878 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:13.457792 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5d5c2gpxv"] Oct 30 09:41:13.463004 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:13.462985 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5d5c2gpxv" Oct 30 09:41:13.482096 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:13.482065 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5d5c2gpxv"] Oct 30 09:41:13.489117 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:13.489086 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3d9b36\"/\"routing-overlay\"" Oct 30 09:41:13.489252 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:13.489131 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3d9b36\"/\"openshift-service-ca.crt\"" Oct 30 09:41:13.489252 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:13.489131 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3d9b36\"/\"agc-cred-b1b80af3b434\"" Oct 30 09:41:13.489353 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:13.489305 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3d9b36\"/\"payload-processing-external-model-plugins-e2e-extmod-b-3d9b36\"" Oct 30 09:41:13.489444 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:13.489426 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3d9b36\"/\"kube-root-ca.crt\"" Oct 30 09:41:13.491453 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:13.491418 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3d9b36\"/\"payload-processing-external-model-e2e--dd1a53db-dockercfg-lznsm\"" Oct 30 09:41:13.608411 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:13.596598 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c08f4e10-1e67-440f-99a4-7e066bb0c23e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-5d5c2gpxv\" (UID: \"c08f4e10-1e67-440f-99a4-7e066bb0c23e\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5d5c2gpxv" Oct 30 09:41:13.608931 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:13.608894 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c08f4e10-1e67-440f-99a4-7e066bb0c23e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-5d5c2gpxv\" (UID: \"c08f4e10-1e67-440f-99a4-7e066bb0c23e\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5d5c2gpxv" Oct 30 09:41:13.609218 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:13.609183 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c08f4e10-1e67-440f-99a4-7e066bb0c23e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-5d5c2gpxv\" (UID: \"c08f4e10-1e67-440f-99a4-7e066bb0c23e\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5d5c2gpxv" Oct 30 09:41:13.609745 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:13.609712 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgkvl\" (UniqueName: \"kubernetes.io/projected/c08f4e10-1e67-440f-99a4-7e066bb0c23e-kube-api-access-fgkvl\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-5d5c2gpxv\" (UID: \"c08f4e10-1e67-440f-99a4-7e066bb0c23e\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5d5c2gpxv" Oct 30 09:41:13.713706 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:13.711544 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgkvl\" (UniqueName: \"kubernetes.io/projected/c08f4e10-1e67-440f-99a4-7e066bb0c23e-kube-api-access-fgkvl\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-5d5c2gpxv\" (UID: \"c08f4e10-1e67-440f-99a4-7e066bb0c23e\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5d5c2gpxv" Oct 30 09:41:13.713706 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:13.711687 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c08f4e10-1e67-440f-99a4-7e066bb0c23e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-5d5c2gpxv\" (UID: \"c08f4e10-1e67-440f-99a4-7e066bb0c23e\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5d5c2gpxv" Oct 30 09:41:13.713706 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:13.711783 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c08f4e10-1e67-440f-99a4-7e066bb0c23e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-5d5c2gpxv\" (UID: \"c08f4e10-1e67-440f-99a4-7e066bb0c23e\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5d5c2gpxv" Oct 30 09:41:13.713706 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:13.711850 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c08f4e10-1e67-440f-99a4-7e066bb0c23e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-5d5c2gpxv\" (UID: \"c08f4e10-1e67-440f-99a4-7e066bb0c23e\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5d5c2gpxv" Oct 30 09:41:13.713706 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:13.712928 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c08f4e10-1e67-440f-99a4-7e066bb0c23e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-5d5c2gpxv\" (UID: \"c08f4e10-1e67-440f-99a4-7e066bb0c23e\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5d5c2gpxv" Oct 30 09:41:13.714861 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:13.714833 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c08f4e10-1e67-440f-99a4-7e066bb0c23e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-5d5c2gpxv\" (UID: \"c08f4e10-1e67-440f-99a4-7e066bb0c23e\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5d5c2gpxv" Oct 30 09:41:13.733247 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:13.733172 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c08f4e10-1e67-440f-99a4-7e066bb0c23e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-5d5c2gpxv\" (UID: \"c08f4e10-1e67-440f-99a4-7e066bb0c23e\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5d5c2gpxv" Oct 30 09:41:13.734372 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:13.734348 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgkvl\" (UniqueName: \"kubernetes.io/projected/c08f4e10-1e67-440f-99a4-7e066bb0c23e-kube-api-access-fgkvl\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-5d5c2gpxv\" (UID: \"c08f4e10-1e67-440f-99a4-7e066bb0c23e\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5d5c2gpxv" Oct 30 09:41:13.777774 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:13.777743 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5d5c2gpxv" Oct 30 09:41:14.101986 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:41:14.101951 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc08f4e10_1e67_440f_99a4_7e066bb0c23e.slice/crio-a37999533fe09a068e939019458a9cd54d49916f780a1fcb969e3876e862b3dc WatchSource:0}: Error finding container a37999533fe09a068e939019458a9cd54d49916f780a1fcb969e3876e862b3dc: Status 404 returned error can't find the container with id a37999533fe09a068e939019458a9cd54d49916f780a1fcb969e3876e862b3dc Oct 30 09:41:14.103152 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:14.103124 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5d5c2gpxv"] Oct 30 09:41:15.047798 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:15.047764 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5d5c2gpxv" event={"ID":"c08f4e10-1e67-440f-99a4-7e066bb0c23e","Type":"ContainerStarted","Data":"40812b0c7d4ece334d9c920047e7b251b68ed6ec2e6c98b305385d4f26487207"} Oct 30 09:41:15.048279 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:15.047815 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5d5c2gpxv" event={"ID":"c08f4e10-1e67-440f-99a4-7e066bb0c23e","Type":"ContainerStarted","Data":"a37999533fe09a068e939019458a9cd54d49916f780a1fcb969e3876e862b3dc"} Oct 30 09:41:15.048279 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:15.047851 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5d5c2gpxv" Oct 30 09:41:15.089188 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:15.089139 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5d5c2gpxv" podStartSLOduration=2.089124927 podStartE2EDuration="2.089124927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:41:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:41:15.088953838 +0000 UTC m=+2835.350514841" watchObservedRunningTime="2026-10-30 09:41:15.089124927 +0000 UTC m=+2835.350685915" Oct 30 09:41:20.755230 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:20.755182 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-6d4b7a-57ddcbd6fb-wzqlg"] Oct 30 09:41:25.756171 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:25.756128 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-6d4b7a-57ddcbd6fb-wzqlg" podUID="bc4c8ab1-cce2-4a36-a929-7ec43a26c17a" containerName="payload-processing" containerID="cri-o://e611286eab9db774100a84d38ca05f39b9ec8f41e1cab5abdc8f7819ca1c7a79" gracePeriod=25 Oct 30 09:41:26.058195 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:26.058172 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5d5c2gpxv" Oct 30 09:41:26.080439 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:26.080415 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-6d4b7a-57ddcbd6fb-wzqlg" Oct 30 09:41:26.137944 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:26.137908 4562 generic.go:356] "Generic (PLEG): container finished" podID="bc4c8ab1-cce2-4a36-a929-7ec43a26c17a" containerID="e611286eab9db774100a84d38ca05f39b9ec8f41e1cab5abdc8f7819ca1c7a79" exitCode=0 Oct 30 09:41:26.138159 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:26.137977 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-6d4b7a-57ddcbd6fb-wzqlg" Oct 30 09:41:26.138159 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:26.138087 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-6d4b7a-57ddcbd6fb-wzqlg" event={"ID":"bc4c8ab1-cce2-4a36-a929-7ec43a26c17a","Type":"ContainerDied","Data":"e611286eab9db774100a84d38ca05f39b9ec8f41e1cab5abdc8f7819ca1c7a79"} Oct 30 09:41:26.138159 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:26.138122 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-6d4b7a-57ddcbd6fb-wzqlg" event={"ID":"bc4c8ab1-cce2-4a36-a929-7ec43a26c17a","Type":"ContainerDied","Data":"9de273a7754301ca8203a0f47cbbe76a1b84bcda565cd82da0f11efcdd2e50ad"} Oct 30 09:41:26.138159 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:26.138146 4562 scope.go:122] "RemoveContainer" containerID="e611286eab9db774100a84d38ca05f39b9ec8f41e1cab5abdc8f7819ca1c7a79" Oct 30 09:41:26.155686 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:26.155653 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc4c8ab1-cce2-4a36-a929-7ec43a26c17a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc4c8ab1-cce2-4a36-a929-7ec43a26c17a-plugins-config-volume\") pod \"bc4c8ab1-cce2-4a36-a929-7ec43a26c17a\" (UID: \"bc4c8ab1-cce2-4a36-a929-7ec43a26c17a\") " Oct 30 09:41:26.155878 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:26.155859 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc4c8ab1-cce2-4a36-a929-7ec43a26c17a-kube-api-access-h9c6j\" (UniqueName: \"kubernetes.io/projected/bc4c8ab1-cce2-4a36-a929-7ec43a26c17a-kube-api-access-h9c6j\") pod \"bc4c8ab1-cce2-4a36-a929-7ec43a26c17a\" (UID: \"bc4c8ab1-cce2-4a36-a929-7ec43a26c17a\") " Oct 30 09:41:26.158470 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:26.158329 4562 scope.go:122] "RemoveContainer" containerID="e611286eab9db774100a84d38ca05f39b9ec8f41e1cab5abdc8f7819ca1c7a79" Oct 30 09:41:26.159394 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:41:26.158756 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e611286eab9db774100a84d38ca05f39b9ec8f41e1cab5abdc8f7819ca1c7a79\": container with ID starting with e611286eab9db774100a84d38ca05f39b9ec8f41e1cab5abdc8f7819ca1c7a79 not found: ID does not exist" containerID="e611286eab9db774100a84d38ca05f39b9ec8f41e1cab5abdc8f7819ca1c7a79" Oct 30 09:41:26.159580 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:26.158795 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e611286eab9db774100a84d38ca05f39b9ec8f41e1cab5abdc8f7819ca1c7a79"} err="failed to get container status \"e611286eab9db774100a84d38ca05f39b9ec8f41e1cab5abdc8f7819ca1c7a79\": rpc error: code = NotFound desc = could not find container \"e611286eab9db774100a84d38ca05f39b9ec8f41e1cab5abdc8f7819ca1c7a79\": container with ID starting with e611286eab9db774100a84d38ca05f39b9ec8f41e1cab5abdc8f7819ca1c7a79 not found: ID does not exist" Oct 30 09:41:26.159853 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:26.159830 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc4c8ab1-cce2-4a36-a929-7ec43a26c17a-plugins-config-volume" pod "bc4c8ab1-cce2-4a36-a929-7ec43a26c17a" (UID: "bc4c8ab1-cce2-4a36-a929-7ec43a26c17a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:41:26.160472 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:26.160449 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc4c8ab1-cce2-4a36-a929-7ec43a26c17a-kube-api-access-h9c6j" pod "bc4c8ab1-cce2-4a36-a929-7ec43a26c17a" (UID: "bc4c8ab1-cce2-4a36-a929-7ec43a26c17a"). InnerVolumeSpecName "kube-api-access-h9c6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:41:26.285444 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:26.263701 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9c6j\" (UniqueName: \"kubernetes.io/projected/bc4c8ab1-cce2-4a36-a929-7ec43a26c17a-kube-api-access-h9c6j\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:41:26.285444 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:26.285263 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc4c8ab1-cce2-4a36-a929-7ec43a26c17a-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:41:26.489281 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:26.489248 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-6d4b7a-57ddcbd6fb-wzqlg"] Oct 30 09:41:26.492938 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:26.492916 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-6d4b7a-57ddcbd6fb-wzqlg"] Oct 30 09:41:27.876539 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:27.876505 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc4c8ab1-cce2-4a36-a929-7ec43a26c17a" path="/var/lib/kubelet/pods/bc4c8ab1-cce2-4a36-a929-7ec43a26c17a/volumes" Oct 30 09:41:28.597199 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:41:28.597160 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:41:58.600786 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:41:58.600735 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:41:59.573129 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:59.573089 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-f58f32/payload-processing-external-model-e2e-extmod-b-f58f32-84d5lhn8r"] Oct 30 09:41:59.573930 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:59.573909 4562 cpu_manager.go:414] "containerMap: removing container" podUID="bc4c8ab1-cce2-4a36-a929-7ec43a26c17a" containerName="payload-processing" Oct 30 09:41:59.573930 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:59.573930 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc4c8ab1-cce2-4a36-a929-7ec43a26c17a" containerName="payload-processing" Oct 30 09:41:59.574147 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:59.574132 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc4c8ab1-cce2-4a36-a929-7ec43a26c17a" containerName="payload-processing" Oct 30 09:41:59.579493 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:59.579472 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f58f32/payload-processing-external-model-e2e-extmod-b-f58f32-84d5lhn8r" Oct 30 09:41:59.597588 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:59.597561 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-f58f32/payload-processing-external-model-e2e-extmod-b-f58f32-84d5lhn8r"] Oct 30 09:41:59.607419 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:59.606594 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f58f32\"/\"kube-root-ca.crt\"" Oct 30 09:41:59.607419 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:59.606611 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f58f32\"/\"openshift-service-ca.crt\"" Oct 30 09:41:59.607419 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:59.606671 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-f58f32\"/\"agc-cred-23ecbe553736\"" Oct 30 09:41:59.607419 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:59.607238 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f58f32\"/\"routing-overlay\"" Oct 30 09:41:59.607994 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:59.607979 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-f58f32\"/\"payload-processing-external-model-e2e--525d068c-dockercfg-wn6nq\"" Oct 30 09:41:59.608227 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:59.608207 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f58f32\"/\"payload-processing-external-model-plugins-e2e-extmod-b-f58f32\"" Oct 30 09:41:59.645118 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:59.645085 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6411fcd0-5e54-47e6-97e6-ba80f1434df2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f58f32-84d5lhn8r\" (UID: \"6411fcd0-5e54-47e6-97e6-ba80f1434df2\") " pod="ai-tenant-e2e-extmod-b-f58f32/payload-processing-external-model-e2e-extmod-b-f58f32-84d5lhn8r" Oct 30 09:41:59.645287 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:59.645208 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6411fcd0-5e54-47e6-97e6-ba80f1434df2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f58f32-84d5lhn8r\" (UID: \"6411fcd0-5e54-47e6-97e6-ba80f1434df2\") " pod="ai-tenant-e2e-extmod-b-f58f32/payload-processing-external-model-e2e-extmod-b-f58f32-84d5lhn8r" Oct 30 09:41:59.645287 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:59.645272 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6411fcd0-5e54-47e6-97e6-ba80f1434df2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f58f32-84d5lhn8r\" (UID: \"6411fcd0-5e54-47e6-97e6-ba80f1434df2\") " pod="ai-tenant-e2e-extmod-b-f58f32/payload-processing-external-model-e2e-extmod-b-f58f32-84d5lhn8r" Oct 30 09:41:59.645438 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:59.645302 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgvqs\" (UniqueName: \"kubernetes.io/projected/6411fcd0-5e54-47e6-97e6-ba80f1434df2-kube-api-access-hgvqs\") pod \"payload-processing-external-model-e2e-extmod-b-f58f32-84d5lhn8r\" (UID: \"6411fcd0-5e54-47e6-97e6-ba80f1434df2\") " pod="ai-tenant-e2e-extmod-b-f58f32/payload-processing-external-model-e2e-extmod-b-f58f32-84d5lhn8r" Oct 30 09:41:59.760402 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:59.760320 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6411fcd0-5e54-47e6-97e6-ba80f1434df2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f58f32-84d5lhn8r\" (UID: \"6411fcd0-5e54-47e6-97e6-ba80f1434df2\") " pod="ai-tenant-e2e-extmod-b-f58f32/payload-processing-external-model-e2e-extmod-b-f58f32-84d5lhn8r" Oct 30 09:41:59.760615 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:59.760465 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6411fcd0-5e54-47e6-97e6-ba80f1434df2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f58f32-84d5lhn8r\" (UID: \"6411fcd0-5e54-47e6-97e6-ba80f1434df2\") " pod="ai-tenant-e2e-extmod-b-f58f32/payload-processing-external-model-e2e-extmod-b-f58f32-84d5lhn8r" Oct 30 09:41:59.760615 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:59.760512 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6411fcd0-5e54-47e6-97e6-ba80f1434df2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f58f32-84d5lhn8r\" (UID: \"6411fcd0-5e54-47e6-97e6-ba80f1434df2\") " pod="ai-tenant-e2e-extmod-b-f58f32/payload-processing-external-model-e2e-extmod-b-f58f32-84d5lhn8r" Oct 30 09:41:59.760615 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:59.760543 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgvqs\" (UniqueName: \"kubernetes.io/projected/6411fcd0-5e54-47e6-97e6-ba80f1434df2-kube-api-access-hgvqs\") pod \"payload-processing-external-model-e2e-extmod-b-f58f32-84d5lhn8r\" (UID: \"6411fcd0-5e54-47e6-97e6-ba80f1434df2\") " pod="ai-tenant-e2e-extmod-b-f58f32/payload-processing-external-model-e2e-extmod-b-f58f32-84d5lhn8r" Oct 30 09:41:59.761811 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:59.761783 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6411fcd0-5e54-47e6-97e6-ba80f1434df2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f58f32-84d5lhn8r\" (UID: \"6411fcd0-5e54-47e6-97e6-ba80f1434df2\") " pod="ai-tenant-e2e-extmod-b-f58f32/payload-processing-external-model-e2e-extmod-b-f58f32-84d5lhn8r" Oct 30 09:41:59.785790 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:59.776281 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6411fcd0-5e54-47e6-97e6-ba80f1434df2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f58f32-84d5lhn8r\" (UID: \"6411fcd0-5e54-47e6-97e6-ba80f1434df2\") " pod="ai-tenant-e2e-extmod-b-f58f32/payload-processing-external-model-e2e-extmod-b-f58f32-84d5lhn8r" Oct 30 09:41:59.786003 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:59.785709 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6411fcd0-5e54-47e6-97e6-ba80f1434df2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f58f32-84d5lhn8r\" (UID: \"6411fcd0-5e54-47e6-97e6-ba80f1434df2\") " pod="ai-tenant-e2e-extmod-b-f58f32/payload-processing-external-model-e2e-extmod-b-f58f32-84d5lhn8r" Oct 30 09:41:59.818294 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:59.818219 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgvqs\" (UniqueName: \"kubernetes.io/projected/6411fcd0-5e54-47e6-97e6-ba80f1434df2-kube-api-access-hgvqs\") pod \"payload-processing-external-model-e2e-extmod-b-f58f32-84d5lhn8r\" (UID: \"6411fcd0-5e54-47e6-97e6-ba80f1434df2\") " pod="ai-tenant-e2e-extmod-b-f58f32/payload-processing-external-model-e2e-extmod-b-f58f32-84d5lhn8r" Oct 30 09:41:59.909304 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:59.909206 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-f58f32\"/\"payload-processing-external-model-e2e--525d068c-dockercfg-wn6nq\"" Oct 30 09:41:59.915194 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:41:59.915168 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f58f32/payload-processing-external-model-e2e-extmod-b-f58f32-84d5lhn8r" Oct 30 09:42:00.225479 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:42:00.225446 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6411fcd0_5e54_47e6_97e6_ba80f1434df2.slice/crio-8768a6eda700f556038f763bd7abc1ee03f2826b10d5518b8360225c255c880b WatchSource:0}: Error finding container 8768a6eda700f556038f763bd7abc1ee03f2826b10d5518b8360225c255c880b: Status 404 returned error can't find the container with id 8768a6eda700f556038f763bd7abc1ee03f2826b10d5518b8360225c255c880b Oct 30 09:42:00.227481 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:42:00.227455 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-f58f32/payload-processing-external-model-e2e-extmod-b-f58f32-84d5lhn8r"] Oct 30 09:42:00.442266 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:42:00.442228 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f58f32/payload-processing-external-model-e2e-extmod-b-f58f32-84d5lhn8r" event={"ID":"6411fcd0-5e54-47e6-97e6-ba80f1434df2","Type":"ContainerStarted","Data":"b2d864067d4d624414bd20ba4b672fa53ec612950c979567188bee27709b7eab"} Oct 30 09:42:00.442484 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:42:00.442293 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f58f32/payload-processing-external-model-e2e-extmod-b-f58f32-84d5lhn8r" event={"ID":"6411fcd0-5e54-47e6-97e6-ba80f1434df2","Type":"ContainerStarted","Data":"8768a6eda700f556038f763bd7abc1ee03f2826b10d5518b8360225c255c880b"} Oct 30 09:42:00.442484 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:42:00.442321 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-f58f32/payload-processing-external-model-e2e-extmod-b-f58f32-84d5lhn8r" Oct 30 09:42:00.484356 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:42:00.484253 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-f58f32/payload-processing-external-model-e2e-extmod-b-f58f32-84d5lhn8r" podStartSLOduration=1.48423829 podStartE2EDuration="1.48423829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:41:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:42:00.481458446 +0000 UTC m=+2880.743019435" watchObservedRunningTime="2026-10-30 09:42:00.48423829 +0000 UTC m=+2880.745799278" Oct 30 09:42:11.452302 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:42:11.452269 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-f58f32/payload-processing-external-model-e2e-extmod-b-f58f32-84d5lhn8r" Oct 30 09:42:28.597774 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:42:28.597733 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:42:30.593712 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:42:30.593678 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-035d7d/payload-processing-external-model-e2e-extmod-b-035d7d-8675dchql"] Oct 30 09:42:35.594021 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:42:35.593977 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-035d7d/payload-processing-external-model-e2e-extmod-b-035d7d-8675dchql" podUID="bb6eefc6-9548-4c9c-bbe3-4bef240b4716" containerName="payload-processing" containerID="cri-o://6823908a52d73431fcdf50506a02e476d4754d7dde12610e7b497a264c203eb4" gracePeriod=25 Oct 30 09:42:35.799737 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:42:35.799707 4562 generic.go:356] "Generic (PLEG): container finished" podID="bb6eefc6-9548-4c9c-bbe3-4bef240b4716" containerID="6823908a52d73431fcdf50506a02e476d4754d7dde12610e7b497a264c203eb4" exitCode=0 Oct 30 09:42:35.799898 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:42:35.799781 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-035d7d/payload-processing-external-model-e2e-extmod-b-035d7d-8675dchql" event={"ID":"bb6eefc6-9548-4c9c-bbe3-4bef240b4716","Type":"ContainerDied","Data":"6823908a52d73431fcdf50506a02e476d4754d7dde12610e7b497a264c203eb4"} Oct 30 09:42:35.890625 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:42:35.890598 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-035d7d/payload-processing-external-model-e2e-extmod-b-035d7d-8675dchql" Oct 30 09:42:35.986470 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:42:35.986430 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb6eefc6-9548-4c9c-bbe3-4bef240b4716-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb6eefc6-9548-4c9c-bbe3-4bef240b4716-plugins-config-volume\") pod \"bb6eefc6-9548-4c9c-bbe3-4bef240b4716\" (UID: \"bb6eefc6-9548-4c9c-bbe3-4bef240b4716\") " Oct 30 09:42:35.986672 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:42:35.986629 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb6eefc6-9548-4c9c-bbe3-4bef240b4716-provider-credentials\" (UniqueName: \"kubernetes.io/projected/bb6eefc6-9548-4c9c-bbe3-4bef240b4716-provider-credentials\") pod \"bb6eefc6-9548-4c9c-bbe3-4bef240b4716\" (UID: \"bb6eefc6-9548-4c9c-bbe3-4bef240b4716\") " Oct 30 09:42:35.986755 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:42:35.986680 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb6eefc6-9548-4c9c-bbe3-4bef240b4716-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bb6eefc6-9548-4c9c-bbe3-4bef240b4716-routing-overlay-volume\") pod \"bb6eefc6-9548-4c9c-bbe3-4bef240b4716\" (UID: \"bb6eefc6-9548-4c9c-bbe3-4bef240b4716\") " Oct 30 09:42:35.986755 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:42:35.986727 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb6eefc6-9548-4c9c-bbe3-4bef240b4716-kube-api-access-tfzm6\" (UniqueName: \"kubernetes.io/projected/bb6eefc6-9548-4c9c-bbe3-4bef240b4716-kube-api-access-tfzm6\") pod \"bb6eefc6-9548-4c9c-bbe3-4bef240b4716\" (UID: \"bb6eefc6-9548-4c9c-bbe3-4bef240b4716\") " Oct 30 09:42:35.986978 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:42:35.986940 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb6eefc6-9548-4c9c-bbe3-4bef240b4716-plugins-config-volume" pod "bb6eefc6-9548-4c9c-bbe3-4bef240b4716" (UID: "bb6eefc6-9548-4c9c-bbe3-4bef240b4716"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:42:35.987143 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:42:35.987119 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb6eefc6-9548-4c9c-bbe3-4bef240b4716-routing-overlay-volume" pod "bb6eefc6-9548-4c9c-bbe3-4bef240b4716" (UID: "bb6eefc6-9548-4c9c-bbe3-4bef240b4716"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:42:35.987528 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:42:35.987504 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb6eefc6-9548-4c9c-bbe3-4bef240b4716-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:42:35.987646 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:42:35.987532 4562 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bb6eefc6-9548-4c9c-bbe3-4bef240b4716-routing-overlay-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:42:35.989190 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:42:35.989160 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb6eefc6-9548-4c9c-bbe3-4bef240b4716-kube-api-access-tfzm6" pod "bb6eefc6-9548-4c9c-bbe3-4bef240b4716" (UID: "bb6eefc6-9548-4c9c-bbe3-4bef240b4716"). InnerVolumeSpecName "kube-api-access-tfzm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:42:35.989287 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:42:35.989199 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb6eefc6-9548-4c9c-bbe3-4bef240b4716-provider-credentials" pod "bb6eefc6-9548-4c9c-bbe3-4bef240b4716" (UID: "bb6eefc6-9548-4c9c-bbe3-4bef240b4716"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:42:36.089003 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:42:36.088972 4562 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bb6eefc6-9548-4c9c-bbe3-4bef240b4716-provider-credentials\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:42:36.089003 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:42:36.089002 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfzm6\" (UniqueName: \"kubernetes.io/projected/bb6eefc6-9548-4c9c-bbe3-4bef240b4716-kube-api-access-tfzm6\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:42:36.808037 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:42:36.808003 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-035d7d/payload-processing-external-model-e2e-extmod-b-035d7d-8675dchql" event={"ID":"bb6eefc6-9548-4c9c-bbe3-4bef240b4716","Type":"ContainerDied","Data":"1a9ce71e03bef8041872a601bfcf41427e208c65c6eb00361504867f218ed1b2"} Oct 30 09:42:36.808491 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:42:36.808065 4562 scope.go:122] "RemoveContainer" containerID="6823908a52d73431fcdf50506a02e476d4754d7dde12610e7b497a264c203eb4" Oct 30 09:42:36.808491 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:42:36.808014 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-035d7d/payload-processing-external-model-e2e-extmod-b-035d7d-8675dchql" Oct 30 09:42:36.863954 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:42:36.863921 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-035d7d/payload-processing-external-model-e2e-extmod-b-035d7d-8675dchql"] Oct 30 09:42:36.876748 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:42:36.876722 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-035d7d/payload-processing-external-model-e2e-extmod-b-035d7d-8675dchql"] Oct 30 09:42:37.875644 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:42:37.875610 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb6eefc6-9548-4c9c-bbe3-4bef240b4716" path="/var/lib/kubelet/pods/bb6eefc6-9548-4c9c-bbe3-4bef240b4716/volumes" Oct 30 09:42:50.950572 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:42:50.950494 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-7555n6htd"] Oct 30 09:42:50.951325 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:42:50.951307 4562 cpu_manager.go:414] "containerMap: removing container" podUID="bb6eefc6-9548-4c9c-bbe3-4bef240b4716" containerName="payload-processing" Oct 30 09:42:50.951373 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:42:50.951326 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb6eefc6-9548-4c9c-bbe3-4bef240b4716" containerName="payload-processing" Oct 30 09:42:50.951459 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:42:50.951449 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb6eefc6-9548-4c9c-bbe3-4bef240b4716" containerName="payload-processing" Oct 30 09:42:50.956535 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:42:50.956513 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-7555n6htd" Oct 30 09:42:50.972414 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:42:50.972366 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-7555n6htd"] Oct 30 09:42:50.981056 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:42:50.981034 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3d9b36\"/\"agc-cred-a64cac9b1414\"" Oct 30 09:42:51.047649 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:42:51.047582 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ffe2fbb7-b680-4353-852d-c571af522c9f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-7555n6htd\" (UID: \"ffe2fbb7-b680-4353-852d-c571af522c9f\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-7555n6htd" Oct 30 09:42:51.047854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:42:51.047655 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ffe2fbb7-b680-4353-852d-c571af522c9f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-7555n6htd\" (UID: \"ffe2fbb7-b680-4353-852d-c571af522c9f\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-7555n6htd" Oct 30 09:42:51.047854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:42:51.047748 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffe2fbb7-b680-4353-852d-c571af522c9f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-7555n6htd\" (UID: \"ffe2fbb7-b680-4353-852d-c571af522c9f\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-7555n6htd" Oct 30 09:42:51.047982 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:42:51.047867 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w7wh\" (UniqueName: \"kubernetes.io/projected/ffe2fbb7-b680-4353-852d-c571af522c9f-kube-api-access-4w7wh\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-7555n6htd\" (UID: \"ffe2fbb7-b680-4353-852d-c571af522c9f\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-7555n6htd" Oct 30 09:42:51.149853 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:42:51.149818 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w7wh\" (UniqueName: \"kubernetes.io/projected/ffe2fbb7-b680-4353-852d-c571af522c9f-kube-api-access-4w7wh\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-7555n6htd\" (UID: \"ffe2fbb7-b680-4353-852d-c571af522c9f\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-7555n6htd" Oct 30 09:42:51.150087 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:42:51.150068 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ffe2fbb7-b680-4353-852d-c571af522c9f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-7555n6htd\" (UID: \"ffe2fbb7-b680-4353-852d-c571af522c9f\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-7555n6htd" Oct 30 09:42:51.150165 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:42:51.150114 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ffe2fbb7-b680-4353-852d-c571af522c9f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-7555n6htd\" (UID: \"ffe2fbb7-b680-4353-852d-c571af522c9f\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-7555n6htd" Oct 30 09:42:51.150222 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:42:51.150162 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffe2fbb7-b680-4353-852d-c571af522c9f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-7555n6htd\" (UID: \"ffe2fbb7-b680-4353-852d-c571af522c9f\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-7555n6htd" Oct 30 09:42:51.150959 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:42:51.150932 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ffe2fbb7-b680-4353-852d-c571af522c9f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-7555n6htd\" (UID: \"ffe2fbb7-b680-4353-852d-c571af522c9f\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-7555n6htd" Oct 30 09:42:51.151089 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:42:51.151016 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffe2fbb7-b680-4353-852d-c571af522c9f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-7555n6htd\" (UID: \"ffe2fbb7-b680-4353-852d-c571af522c9f\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-7555n6htd" Oct 30 09:42:51.152765 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:42:51.152739 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ffe2fbb7-b680-4353-852d-c571af522c9f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-7555n6htd\" (UID: \"ffe2fbb7-b680-4353-852d-c571af522c9f\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-7555n6htd" Oct 30 09:42:51.169827 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:42:51.169800 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w7wh\" (UniqueName: \"kubernetes.io/projected/ffe2fbb7-b680-4353-852d-c571af522c9f-kube-api-access-4w7wh\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-7555n6htd\" (UID: \"ffe2fbb7-b680-4353-852d-c571af522c9f\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-7555n6htd" Oct 30 09:42:51.271664 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:42:51.271590 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-7555n6htd" Oct 30 09:42:51.600137 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:42:51.600108 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffe2fbb7_b680_4353_852d_c571af522c9f.slice/crio-aa77844aedc87ccea8afaccb6190e3ff1a6b81d9f0683d0332fcfe7e0e44e866 WatchSource:0}: Error finding container aa77844aedc87ccea8afaccb6190e3ff1a6b81d9f0683d0332fcfe7e0e44e866: Status 404 returned error can't find the container with id aa77844aedc87ccea8afaccb6190e3ff1a6b81d9f0683d0332fcfe7e0e44e866 Oct 30 09:42:51.603417 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:42:51.603362 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-7555n6htd"] Oct 30 09:42:52.008552 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:42:52.008508 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-7555n6htd" event={"ID":"ffe2fbb7-b680-4353-852d-c571af522c9f","Type":"ContainerStarted","Data":"361d1e783c0ba0d28d66e14e8df5ea7fadeb6308a08c5b9223018cd8e2a26056"} Oct 30 09:42:52.009041 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:42:52.008573 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-7555n6htd" event={"ID":"ffe2fbb7-b680-4353-852d-c571af522c9f","Type":"ContainerStarted","Data":"aa77844aedc87ccea8afaccb6190e3ff1a6b81d9f0683d0332fcfe7e0e44e866"} Oct 30 09:42:52.009041 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:42:52.008752 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-7555n6htd" Oct 30 09:42:52.053993 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:42:52.053949 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-7555n6htd" podStartSLOduration=2.053936329 podStartE2EDuration="2.053936329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:42:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:42:52.050072889 +0000 UTC m=+2932.311633913" watchObservedRunningTime="2026-10-30 09:42:52.053936329 +0000 UTC m=+2932.315497316" Oct 30 09:42:58.596896 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:42:58.596856 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:43:03.052429 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:03.052373 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-7555n6htd" Oct 30 09:43:03.122524 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:03.122487 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5d5c2gpxv"] Oct 30 09:43:08.123398 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:08.123351 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5d5c2gpxv" podUID="c08f4e10-1e67-440f-99a4-7e066bb0c23e" containerName="payload-processing" containerID="cri-o://40812b0c7d4ece334d9c920047e7b251b68ed6ec2e6c98b305385d4f26487207" gracePeriod=25 Oct 30 09:43:08.413827 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:08.413795 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5d5c2gpxv" Oct 30 09:43:08.466478 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:08.466359 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c08f4e10-1e67-440f-99a4-7e066bb0c23e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c08f4e10-1e67-440f-99a4-7e066bb0c23e-provider-credentials\") pod \"c08f4e10-1e67-440f-99a4-7e066bb0c23e\" (UID: \"c08f4e10-1e67-440f-99a4-7e066bb0c23e\") " Oct 30 09:43:08.467558 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:08.466430 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c08f4e10-1e67-440f-99a4-7e066bb0c23e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c08f4e10-1e67-440f-99a4-7e066bb0c23e-routing-overlay-volume\") pod \"c08f4e10-1e67-440f-99a4-7e066bb0c23e\" (UID: \"c08f4e10-1e67-440f-99a4-7e066bb0c23e\") " Oct 30 09:43:08.467558 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:08.467481 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c08f4e10-1e67-440f-99a4-7e066bb0c23e-routing-overlay-volume" pod "c08f4e10-1e67-440f-99a4-7e066bb0c23e" (UID: "c08f4e10-1e67-440f-99a4-7e066bb0c23e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:43:08.468364 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:08.467861 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c08f4e10-1e67-440f-99a4-7e066bb0c23e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c08f4e10-1e67-440f-99a4-7e066bb0c23e-plugins-config-volume\") pod \"c08f4e10-1e67-440f-99a4-7e066bb0c23e\" (UID: \"c08f4e10-1e67-440f-99a4-7e066bb0c23e\") " Oct 30 09:43:08.468364 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:08.468334 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c08f4e10-1e67-440f-99a4-7e066bb0c23e-plugins-config-volume" pod "c08f4e10-1e67-440f-99a4-7e066bb0c23e" (UID: "c08f4e10-1e67-440f-99a4-7e066bb0c23e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:43:08.469019 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:08.468720 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c08f4e10-1e67-440f-99a4-7e066bb0c23e-kube-api-access-fgkvl\" (UniqueName: \"kubernetes.io/projected/c08f4e10-1e67-440f-99a4-7e066bb0c23e-kube-api-access-fgkvl\") pod \"c08f4e10-1e67-440f-99a4-7e066bb0c23e\" (UID: \"c08f4e10-1e67-440f-99a4-7e066bb0c23e\") " Oct 30 09:43:08.469960 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:08.469910 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c08f4e10-1e67-440f-99a4-7e066bb0c23e-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:43:08.469960 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:08.469938 4562 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c08f4e10-1e67-440f-99a4-7e066bb0c23e-routing-overlay-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:43:08.478576 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:08.478548 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c08f4e10-1e67-440f-99a4-7e066bb0c23e-kube-api-access-fgkvl" pod "c08f4e10-1e67-440f-99a4-7e066bb0c23e" (UID: "c08f4e10-1e67-440f-99a4-7e066bb0c23e"). InnerVolumeSpecName "kube-api-access-fgkvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:43:08.478744 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:08.478589 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c08f4e10-1e67-440f-99a4-7e066bb0c23e-provider-credentials" pod "c08f4e10-1e67-440f-99a4-7e066bb0c23e" (UID: "c08f4e10-1e67-440f-99a4-7e066bb0c23e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:43:08.571746 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:08.571705 4562 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c08f4e10-1e67-440f-99a4-7e066bb0c23e-provider-credentials\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:43:08.571746 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:08.571748 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgkvl\" (UniqueName: \"kubernetes.io/projected/c08f4e10-1e67-440f-99a4-7e066bb0c23e-kube-api-access-fgkvl\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:43:09.213793 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:09.213757 4562 generic.go:356] "Generic (PLEG): container finished" podID="c08f4e10-1e67-440f-99a4-7e066bb0c23e" containerID="40812b0c7d4ece334d9c920047e7b251b68ed6ec2e6c98b305385d4f26487207" exitCode=0 Oct 30 09:43:09.214193 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:09.213805 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5d5c2gpxv" Oct 30 09:43:09.214193 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:09.213847 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5d5c2gpxv" event={"ID":"c08f4e10-1e67-440f-99a4-7e066bb0c23e","Type":"ContainerDied","Data":"40812b0c7d4ece334d9c920047e7b251b68ed6ec2e6c98b305385d4f26487207"} Oct 30 09:43:09.214193 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:09.213899 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5d5c2gpxv" event={"ID":"c08f4e10-1e67-440f-99a4-7e066bb0c23e","Type":"ContainerDied","Data":"a37999533fe09a068e939019458a9cd54d49916f780a1fcb969e3876e862b3dc"} Oct 30 09:43:09.214193 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:09.213924 4562 scope.go:122] "RemoveContainer" containerID="40812b0c7d4ece334d9c920047e7b251b68ed6ec2e6c98b305385d4f26487207" Oct 30 09:43:09.225709 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:09.225692 4562 scope.go:122] "RemoveContainer" containerID="40812b0c7d4ece334d9c920047e7b251b68ed6ec2e6c98b305385d4f26487207" Oct 30 09:43:09.225973 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:43:09.225956 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40812b0c7d4ece334d9c920047e7b251b68ed6ec2e6c98b305385d4f26487207\": container with ID starting with 40812b0c7d4ece334d9c920047e7b251b68ed6ec2e6c98b305385d4f26487207 not found: ID does not exist" containerID="40812b0c7d4ece334d9c920047e7b251b68ed6ec2e6c98b305385d4f26487207" Oct 30 09:43:09.226035 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:09.225980 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40812b0c7d4ece334d9c920047e7b251b68ed6ec2e6c98b305385d4f26487207"} err="failed to get container status \"40812b0c7d4ece334d9c920047e7b251b68ed6ec2e6c98b305385d4f26487207\": rpc error: code = NotFound desc = could not find container \"40812b0c7d4ece334d9c920047e7b251b68ed6ec2e6c98b305385d4f26487207\": container with ID starting with 40812b0c7d4ece334d9c920047e7b251b68ed6ec2e6c98b305385d4f26487207 not found: ID does not exist" Oct 30 09:43:09.268994 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:09.268965 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5d5c2gpxv"] Oct 30 09:43:09.281406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:09.281368 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5d5c2gpxv"] Oct 30 09:43:09.960299 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:09.959335 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c08f4e10-1e67-440f-99a4-7e066bb0c23e" path="/var/lib/kubelet/pods/c08f4e10-1e67-440f-99a4-7e066bb0c23e/volumes" Oct 30 09:43:11.115074 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:11.115040 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-f58f32/payload-processing-external-model-e2e-extmod-b-f58f32-65c7kh4xp"] Oct 30 09:43:11.115901 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:11.115875 4562 cpu_manager.go:414] "containerMap: removing container" podUID="c08f4e10-1e67-440f-99a4-7e066bb0c23e" containerName="payload-processing" Oct 30 09:43:11.115901 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:11.115897 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c08f4e10-1e67-440f-99a4-7e066bb0c23e" containerName="payload-processing" Oct 30 09:43:11.116142 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:11.116073 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="c08f4e10-1e67-440f-99a4-7e066bb0c23e" containerName="payload-processing" Oct 30 09:43:11.121074 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:11.121055 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f58f32/payload-processing-external-model-e2e-extmod-b-f58f32-65c7kh4xp" Oct 30 09:43:11.136019 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:11.135995 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-f58f32/payload-processing-external-model-e2e-extmod-b-f58f32-65c7kh4xp"] Oct 30 09:43:11.144927 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:11.144903 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-f58f32\"/\"agc-cred-8e68fd3db4b1\"" Oct 30 09:43:11.228689 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:11.228658 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19c6e7c8-d41e-463e-8543-31f5fdc072b8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f58f32-65c7kh4xp\" (UID: \"19c6e7c8-d41e-463e-8543-31f5fdc072b8\") " pod="ai-tenant-e2e-extmod-b-f58f32/payload-processing-external-model-e2e-extmod-b-f58f32-65c7kh4xp" Oct 30 09:43:11.228689 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:11.228690 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/19c6e7c8-d41e-463e-8543-31f5fdc072b8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f58f32-65c7kh4xp\" (UID: \"19c6e7c8-d41e-463e-8543-31f5fdc072b8\") " pod="ai-tenant-e2e-extmod-b-f58f32/payload-processing-external-model-e2e-extmod-b-f58f32-65c7kh4xp" Oct 30 09:43:11.228882 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:11.228864 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9bjn\" (UniqueName: \"kubernetes.io/projected/19c6e7c8-d41e-463e-8543-31f5fdc072b8-kube-api-access-w9bjn\") pod \"payload-processing-external-model-e2e-extmod-b-f58f32-65c7kh4xp\" (UID: \"19c6e7c8-d41e-463e-8543-31f5fdc072b8\") " pod="ai-tenant-e2e-extmod-b-f58f32/payload-processing-external-model-e2e-extmod-b-f58f32-65c7kh4xp" Oct 30 09:43:11.229010 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:11.228996 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19c6e7c8-d41e-463e-8543-31f5fdc072b8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f58f32-65c7kh4xp\" (UID: \"19c6e7c8-d41e-463e-8543-31f5fdc072b8\") " pod="ai-tenant-e2e-extmod-b-f58f32/payload-processing-external-model-e2e-extmod-b-f58f32-65c7kh4xp" Oct 30 09:43:11.330837 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:11.330797 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19c6e7c8-d41e-463e-8543-31f5fdc072b8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f58f32-65c7kh4xp\" (UID: \"19c6e7c8-d41e-463e-8543-31f5fdc072b8\") " pod="ai-tenant-e2e-extmod-b-f58f32/payload-processing-external-model-e2e-extmod-b-f58f32-65c7kh4xp" Oct 30 09:43:11.330999 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:11.330857 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19c6e7c8-d41e-463e-8543-31f5fdc072b8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f58f32-65c7kh4xp\" (UID: \"19c6e7c8-d41e-463e-8543-31f5fdc072b8\") " pod="ai-tenant-e2e-extmod-b-f58f32/payload-processing-external-model-e2e-extmod-b-f58f32-65c7kh4xp" Oct 30 09:43:11.330999 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:11.330883 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/19c6e7c8-d41e-463e-8543-31f5fdc072b8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f58f32-65c7kh4xp\" (UID: \"19c6e7c8-d41e-463e-8543-31f5fdc072b8\") " pod="ai-tenant-e2e-extmod-b-f58f32/payload-processing-external-model-e2e-extmod-b-f58f32-65c7kh4xp" Oct 30 09:43:11.331185 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:11.331063 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9bjn\" (UniqueName: \"kubernetes.io/projected/19c6e7c8-d41e-463e-8543-31f5fdc072b8-kube-api-access-w9bjn\") pod \"payload-processing-external-model-e2e-extmod-b-f58f32-65c7kh4xp\" (UID: \"19c6e7c8-d41e-463e-8543-31f5fdc072b8\") " pod="ai-tenant-e2e-extmod-b-f58f32/payload-processing-external-model-e2e-extmod-b-f58f32-65c7kh4xp" Oct 30 09:43:11.331583 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:11.331560 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19c6e7c8-d41e-463e-8543-31f5fdc072b8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f58f32-65c7kh4xp\" (UID: \"19c6e7c8-d41e-463e-8543-31f5fdc072b8\") " pod="ai-tenant-e2e-extmod-b-f58f32/payload-processing-external-model-e2e-extmod-b-f58f32-65c7kh4xp" Oct 30 09:43:11.331734 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:11.331712 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19c6e7c8-d41e-463e-8543-31f5fdc072b8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f58f32-65c7kh4xp\" (UID: \"19c6e7c8-d41e-463e-8543-31f5fdc072b8\") " pod="ai-tenant-e2e-extmod-b-f58f32/payload-processing-external-model-e2e-extmod-b-f58f32-65c7kh4xp" Oct 30 09:43:11.333525 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:11.333501 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/19c6e7c8-d41e-463e-8543-31f5fdc072b8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f58f32-65c7kh4xp\" (UID: \"19c6e7c8-d41e-463e-8543-31f5fdc072b8\") " pod="ai-tenant-e2e-extmod-b-f58f32/payload-processing-external-model-e2e-extmod-b-f58f32-65c7kh4xp" Oct 30 09:43:11.350865 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:11.350841 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9bjn\" (UniqueName: \"kubernetes.io/projected/19c6e7c8-d41e-463e-8543-31f5fdc072b8-kube-api-access-w9bjn\") pod \"payload-processing-external-model-e2e-extmod-b-f58f32-65c7kh4xp\" (UID: \"19c6e7c8-d41e-463e-8543-31f5fdc072b8\") " pod="ai-tenant-e2e-extmod-b-f58f32/payload-processing-external-model-e2e-extmod-b-f58f32-65c7kh4xp" Oct 30 09:43:11.435634 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:11.435556 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f58f32/payload-processing-external-model-e2e-extmod-b-f58f32-65c7kh4xp" Oct 30 09:43:11.735021 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:43:11.734985 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19c6e7c8_d41e_463e_8543_31f5fdc072b8.slice/crio-7a5eb40b92fdec5a7512cda7d535feda7ebe97b50ca54a507fb42b92db78d8bb WatchSource:0}: Error finding container 7a5eb40b92fdec5a7512cda7d535feda7ebe97b50ca54a507fb42b92db78d8bb: Status 404 returned error can't find the container with id 7a5eb40b92fdec5a7512cda7d535feda7ebe97b50ca54a507fb42b92db78d8bb Oct 30 09:43:11.736936 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:11.736908 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-f58f32/payload-processing-external-model-e2e-extmod-b-f58f32-65c7kh4xp"] Oct 30 09:43:12.252673 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:12.252627 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f58f32/payload-processing-external-model-e2e-extmod-b-f58f32-65c7kh4xp" event={"ID":"19c6e7c8-d41e-463e-8543-31f5fdc072b8","Type":"ContainerStarted","Data":"78b45c493fe94d3de07137c9247516aa70c3279f14e5629948791409d84c92f6"} Oct 30 09:43:12.253088 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:12.252819 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-f58f32/payload-processing-external-model-e2e-extmod-b-f58f32-65c7kh4xp" Oct 30 09:43:12.253088 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:12.252858 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f58f32/payload-processing-external-model-e2e-extmod-b-f58f32-65c7kh4xp" event={"ID":"19c6e7c8-d41e-463e-8543-31f5fdc072b8","Type":"ContainerStarted","Data":"7a5eb40b92fdec5a7512cda7d535feda7ebe97b50ca54a507fb42b92db78d8bb"} Oct 30 09:43:12.300282 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:12.300225 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-f58f32/payload-processing-external-model-e2e-extmod-b-f58f32-65c7kh4xp" podStartSLOduration=1.30021128 podStartE2EDuration="1.30021128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:43:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:43:12.297545787 +0000 UTC m=+2952.559106781" watchObservedRunningTime="2026-10-30 09:43:12.30021128 +0000 UTC m=+2952.561772312" Oct 30 09:43:17.512899 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:17.512862 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-f58f32/payload-processing-external-model-e2e-extmod-b-f58f32-65c7kh4xp"] Oct 30 09:43:17.516662 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:17.516636 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-f58f32/payload-processing-external-model-e2e-extmod-b-f58f32-65c7kh4xp" Oct 30 09:43:18.781376 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:18.781340 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-85fdfbb6b6-vkrhd"] Oct 30 09:43:18.786367 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:18.786347 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-85fdfbb6b6-vkrhd" Oct 30 09:43:18.804308 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:18.804265 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-85fdfbb6b6-vkrhd"] Oct 30 09:43:18.811434 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:18.811407 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-dbd224b134a2\"" Oct 30 09:43:18.811581 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:18.811457 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 30 09:43:18.811581 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:18.811505 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 30 09:43:18.811581 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:18.811572 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 30 09:43:18.816878 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:18.816857 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 30 09:43:18.817077 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:18.817040 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-pwntl\"" Oct 30 09:43:18.838200 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:18.838168 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7fa57945-98fb-4135-bbdb-db117285ee4e-provider-credentials\") pod \"payload-processing-external-model-85fdfbb6b6-vkrhd\" (UID: \"7fa57945-98fb-4135-bbdb-db117285ee4e\") " pod="models-as-a-service/payload-processing-external-model-85fdfbb6b6-vkrhd" Oct 30 09:43:18.838342 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:18.838323 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fa57945-98fb-4135-bbdb-db117285ee4e-plugins-config-volume\") pod \"payload-processing-external-model-85fdfbb6b6-vkrhd\" (UID: \"7fa57945-98fb-4135-bbdb-db117285ee4e\") " pod="models-as-a-service/payload-processing-external-model-85fdfbb6b6-vkrhd" Oct 30 09:43:18.838492 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:18.838474 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92nfd\" (UniqueName: \"kubernetes.io/projected/7fa57945-98fb-4135-bbdb-db117285ee4e-kube-api-access-92nfd\") pod \"payload-processing-external-model-85fdfbb6b6-vkrhd\" (UID: \"7fa57945-98fb-4135-bbdb-db117285ee4e\") " pod="models-as-a-service/payload-processing-external-model-85fdfbb6b6-vkrhd" Oct 30 09:43:18.838536 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:18.838510 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7fa57945-98fb-4135-bbdb-db117285ee4e-routing-overlay-volume\") pod \"payload-processing-external-model-85fdfbb6b6-vkrhd\" (UID: \"7fa57945-98fb-4135-bbdb-db117285ee4e\") " pod="models-as-a-service/payload-processing-external-model-85fdfbb6b6-vkrhd" Oct 30 09:43:19.013403 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:19.002850 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92nfd\" (UniqueName: \"kubernetes.io/projected/7fa57945-98fb-4135-bbdb-db117285ee4e-kube-api-access-92nfd\") pod \"payload-processing-external-model-85fdfbb6b6-vkrhd\" (UID: \"7fa57945-98fb-4135-bbdb-db117285ee4e\") " pod="models-as-a-service/payload-processing-external-model-85fdfbb6b6-vkrhd" Oct 30 09:43:19.013403 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:19.002915 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7fa57945-98fb-4135-bbdb-db117285ee4e-routing-overlay-volume\") pod \"payload-processing-external-model-85fdfbb6b6-vkrhd\" (UID: \"7fa57945-98fb-4135-bbdb-db117285ee4e\") " pod="models-as-a-service/payload-processing-external-model-85fdfbb6b6-vkrhd" Oct 30 09:43:19.013403 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:19.003129 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7fa57945-98fb-4135-bbdb-db117285ee4e-provider-credentials\") pod \"payload-processing-external-model-85fdfbb6b6-vkrhd\" (UID: \"7fa57945-98fb-4135-bbdb-db117285ee4e\") " pod="models-as-a-service/payload-processing-external-model-85fdfbb6b6-vkrhd" Oct 30 09:43:19.013403 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:19.003253 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fa57945-98fb-4135-bbdb-db117285ee4e-plugins-config-volume\") pod \"payload-processing-external-model-85fdfbb6b6-vkrhd\" (UID: \"7fa57945-98fb-4135-bbdb-db117285ee4e\") " pod="models-as-a-service/payload-processing-external-model-85fdfbb6b6-vkrhd" Oct 30 09:43:19.013403 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:19.004260 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fa57945-98fb-4135-bbdb-db117285ee4e-plugins-config-volume\") pod \"payload-processing-external-model-85fdfbb6b6-vkrhd\" (UID: \"7fa57945-98fb-4135-bbdb-db117285ee4e\") " pod="models-as-a-service/payload-processing-external-model-85fdfbb6b6-vkrhd" Oct 30 09:43:19.031681 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:19.028296 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7fa57945-98fb-4135-bbdb-db117285ee4e-routing-overlay-volume\") pod \"payload-processing-external-model-85fdfbb6b6-vkrhd\" (UID: \"7fa57945-98fb-4135-bbdb-db117285ee4e\") " pod="models-as-a-service/payload-processing-external-model-85fdfbb6b6-vkrhd" Oct 30 09:43:19.045545 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:19.045506 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7fa57945-98fb-4135-bbdb-db117285ee4e-provider-credentials\") pod \"payload-processing-external-model-85fdfbb6b6-vkrhd\" (UID: \"7fa57945-98fb-4135-bbdb-db117285ee4e\") " pod="models-as-a-service/payload-processing-external-model-85fdfbb6b6-vkrhd" Oct 30 09:43:19.087294 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:19.087259 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92nfd\" (UniqueName: \"kubernetes.io/projected/7fa57945-98fb-4135-bbdb-db117285ee4e-kube-api-access-92nfd\") pod \"payload-processing-external-model-85fdfbb6b6-vkrhd\" (UID: \"7fa57945-98fb-4135-bbdb-db117285ee4e\") " pod="models-as-a-service/payload-processing-external-model-85fdfbb6b6-vkrhd" Oct 30 09:43:19.102969 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:19.102933 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-85fdfbb6b6-vkrhd" Oct 30 09:43:19.419460 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:43:19.419426 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fa57945_98fb_4135_bbdb_db117285ee4e.slice/crio-03a9c207797a4d7e939d53c890f69e4866ee3c978551dc6a3e00635e5db60e09 WatchSource:0}: Error finding container 03a9c207797a4d7e939d53c890f69e4866ee3c978551dc6a3e00635e5db60e09: Status 404 returned error can't find the container with id 03a9c207797a4d7e939d53c890f69e4866ee3c978551dc6a3e00635e5db60e09 Oct 30 09:43:19.420569 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:19.420544 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-85fdfbb6b6-vkrhd"] Oct 30 09:43:20.334231 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:20.334189 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-85fdfbb6b6-vkrhd" event={"ID":"7fa57945-98fb-4135-bbdb-db117285ee4e","Type":"ContainerStarted","Data":"ae1c96d3fc5987e9473151e98bce72a52b260e3c0887abcbf101ed343bb1d168"} Oct 30 09:43:20.334231 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:20.334238 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-85fdfbb6b6-vkrhd" event={"ID":"7fa57945-98fb-4135-bbdb-db117285ee4e","Type":"ContainerStarted","Data":"03a9c207797a4d7e939d53c890f69e4866ee3c978551dc6a3e00635e5db60e09"} Oct 30 09:43:20.334741 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:20.334262 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-85fdfbb6b6-vkrhd" Oct 30 09:43:22.513206 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:22.513151 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-f58f32/payload-processing-external-model-e2e-extmod-b-f58f32-65c7kh4xp" podUID="19c6e7c8-d41e-463e-8543-31f5fdc072b8" containerName="payload-processing" containerID="cri-o://78b45c493fe94d3de07137c9247516aa70c3279f14e5629948791409d84c92f6" gracePeriod=25 Oct 30 09:43:22.807790 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:22.807763 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f58f32/payload-processing-external-model-e2e-extmod-b-f58f32-65c7kh4xp" Oct 30 09:43:22.853481 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:22.853417 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-85fdfbb6b6-vkrhd" podStartSLOduration=4.85339581 podStartE2EDuration="4.85339581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:43:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:43:20.380606294 +0000 UTC m=+2960.642167288" watchObservedRunningTime="2026-10-30 09:43:22.85339581 +0000 UTC m=+2963.114956791" Oct 30 09:43:22.870193 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:22.870158 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19c6e7c8-d41e-463e-8543-31f5fdc072b8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/19c6e7c8-d41e-463e-8543-31f5fdc072b8-provider-credentials\") pod \"19c6e7c8-d41e-463e-8543-31f5fdc072b8\" (UID: \"19c6e7c8-d41e-463e-8543-31f5fdc072b8\") " Oct 30 09:43:22.870439 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:22.870245 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19c6e7c8-d41e-463e-8543-31f5fdc072b8-kube-api-access-w9bjn\" (UniqueName: \"kubernetes.io/projected/19c6e7c8-d41e-463e-8543-31f5fdc072b8-kube-api-access-w9bjn\") pod \"19c6e7c8-d41e-463e-8543-31f5fdc072b8\" (UID: \"19c6e7c8-d41e-463e-8543-31f5fdc072b8\") " Oct 30 09:43:22.870685 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:22.870655 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19c6e7c8-d41e-463e-8543-31f5fdc072b8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19c6e7c8-d41e-463e-8543-31f5fdc072b8-plugins-config-volume\") pod \"19c6e7c8-d41e-463e-8543-31f5fdc072b8\" (UID: \"19c6e7c8-d41e-463e-8543-31f5fdc072b8\") " Oct 30 09:43:22.870918 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:22.870894 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19c6e7c8-d41e-463e-8543-31f5fdc072b8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19c6e7c8-d41e-463e-8543-31f5fdc072b8-routing-overlay-volume\") pod \"19c6e7c8-d41e-463e-8543-31f5fdc072b8\" (UID: \"19c6e7c8-d41e-463e-8543-31f5fdc072b8\") " Oct 30 09:43:22.871462 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:22.871437 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19c6e7c8-d41e-463e-8543-31f5fdc072b8-plugins-config-volume" pod "19c6e7c8-d41e-463e-8543-31f5fdc072b8" (UID: "19c6e7c8-d41e-463e-8543-31f5fdc072b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:43:22.871630 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:22.871612 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19c6e7c8-d41e-463e-8543-31f5fdc072b8-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:43:22.871851 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:22.871825 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19c6e7c8-d41e-463e-8543-31f5fdc072b8-routing-overlay-volume" pod "19c6e7c8-d41e-463e-8543-31f5fdc072b8" (UID: "19c6e7c8-d41e-463e-8543-31f5fdc072b8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:43:22.873522 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:22.873489 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19c6e7c8-d41e-463e-8543-31f5fdc072b8-kube-api-access-w9bjn" pod "19c6e7c8-d41e-463e-8543-31f5fdc072b8" (UID: "19c6e7c8-d41e-463e-8543-31f5fdc072b8"). InnerVolumeSpecName "kube-api-access-w9bjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:43:22.873632 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:22.873543 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19c6e7c8-d41e-463e-8543-31f5fdc072b8-provider-credentials" pod "19c6e7c8-d41e-463e-8543-31f5fdc072b8" (UID: "19c6e7c8-d41e-463e-8543-31f5fdc072b8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:43:22.973591 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:22.973553 4562 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19c6e7c8-d41e-463e-8543-31f5fdc072b8-routing-overlay-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:43:22.973591 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:22.973585 4562 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/19c6e7c8-d41e-463e-8543-31f5fdc072b8-provider-credentials\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:43:22.973804 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:22.973600 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9bjn\" (UniqueName: \"kubernetes.io/projected/19c6e7c8-d41e-463e-8543-31f5fdc072b8-kube-api-access-w9bjn\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:43:23.369929 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:23.369891 4562 generic.go:356] "Generic (PLEG): container finished" podID="19c6e7c8-d41e-463e-8543-31f5fdc072b8" containerID="78b45c493fe94d3de07137c9247516aa70c3279f14e5629948791409d84c92f6" exitCode=0 Oct 30 09:43:23.370111 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:23.369950 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f58f32/payload-processing-external-model-e2e-extmod-b-f58f32-65c7kh4xp" Oct 30 09:43:23.370111 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:23.369969 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f58f32/payload-processing-external-model-e2e-extmod-b-f58f32-65c7kh4xp" event={"ID":"19c6e7c8-d41e-463e-8543-31f5fdc072b8","Type":"ContainerDied","Data":"78b45c493fe94d3de07137c9247516aa70c3279f14e5629948791409d84c92f6"} Oct 30 09:43:23.370111 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:23.370056 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f58f32/payload-processing-external-model-e2e-extmod-b-f58f32-65c7kh4xp" event={"ID":"19c6e7c8-d41e-463e-8543-31f5fdc072b8","Type":"ContainerDied","Data":"7a5eb40b92fdec5a7512cda7d535feda7ebe97b50ca54a507fb42b92db78d8bb"} Oct 30 09:43:23.370111 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:23.370086 4562 scope.go:122] "RemoveContainer" containerID="78b45c493fe94d3de07137c9247516aa70c3279f14e5629948791409d84c92f6" Oct 30 09:43:23.381561 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:23.381541 4562 scope.go:122] "RemoveContainer" containerID="78b45c493fe94d3de07137c9247516aa70c3279f14e5629948791409d84c92f6" Oct 30 09:43:23.381836 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:43:23.381811 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78b45c493fe94d3de07137c9247516aa70c3279f14e5629948791409d84c92f6\": container with ID starting with 78b45c493fe94d3de07137c9247516aa70c3279f14e5629948791409d84c92f6 not found: ID does not exist" containerID="78b45c493fe94d3de07137c9247516aa70c3279f14e5629948791409d84c92f6" Oct 30 09:43:23.381938 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:23.381846 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78b45c493fe94d3de07137c9247516aa70c3279f14e5629948791409d84c92f6"} err="failed to get container status \"78b45c493fe94d3de07137c9247516aa70c3279f14e5629948791409d84c92f6\": rpc error: code = NotFound desc = could not find container \"78b45c493fe94d3de07137c9247516aa70c3279f14e5629948791409d84c92f6\": container with ID starting with 78b45c493fe94d3de07137c9247516aa70c3279f14e5629948791409d84c92f6 not found: ID does not exist" Oct 30 09:43:23.453234 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:23.450305 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-f58f32/payload-processing-external-model-e2e-extmod-b-f58f32-65c7kh4xp"] Oct 30 09:43:23.454564 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:23.454537 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-f58f32/payload-processing-external-model-e2e-extmod-b-f58f32-65c7kh4xp"] Oct 30 09:43:23.876304 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:23.876273 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19c6e7c8-d41e-463e-8543-31f5fdc072b8" path="/var/lib/kubelet/pods/19c6e7c8-d41e-463e-8543-31f5fdc072b8/volumes" Oct 30 09:43:28.598204 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:43:28.598168 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:43:31.344999 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:31.344968 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-85fdfbb6b6-vkrhd" Oct 30 09:43:33.124219 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:33.124182 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-f58f32/payload-processing-external-model-e2e-extmod-b-f58f32-84d5lhn8r"] Oct 30 09:43:38.125006 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:38.124937 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-f58f32/payload-processing-external-model-e2e-extmod-b-f58f32-84d5lhn8r" podUID="6411fcd0-5e54-47e6-97e6-ba80f1434df2" containerName="payload-processing" containerID="cri-o://b2d864067d4d624414bd20ba4b672fa53ec612950c979567188bee27709b7eab" gracePeriod=25 Oct 30 09:43:38.417125 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:38.417099 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f58f32/payload-processing-external-model-e2e-extmod-b-f58f32-84d5lhn8r" Oct 30 09:43:38.493564 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:38.493523 4562 generic.go:356] "Generic (PLEG): container finished" podID="6411fcd0-5e54-47e6-97e6-ba80f1434df2" containerID="b2d864067d4d624414bd20ba4b672fa53ec612950c979567188bee27709b7eab" exitCode=0 Oct 30 09:43:38.493753 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:38.493573 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f58f32/payload-processing-external-model-e2e-extmod-b-f58f32-84d5lhn8r" Oct 30 09:43:38.493753 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:38.493604 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f58f32/payload-processing-external-model-e2e-extmod-b-f58f32-84d5lhn8r" event={"ID":"6411fcd0-5e54-47e6-97e6-ba80f1434df2","Type":"ContainerDied","Data":"b2d864067d4d624414bd20ba4b672fa53ec612950c979567188bee27709b7eab"} Oct 30 09:43:38.493753 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:38.493660 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f58f32/payload-processing-external-model-e2e-extmod-b-f58f32-84d5lhn8r" event={"ID":"6411fcd0-5e54-47e6-97e6-ba80f1434df2","Type":"ContainerDied","Data":"8768a6eda700f556038f763bd7abc1ee03f2826b10d5518b8360225c255c880b"} Oct 30 09:43:38.493753 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:38.493686 4562 scope.go:122] "RemoveContainer" containerID="b2d864067d4d624414bd20ba4b672fa53ec612950c979567188bee27709b7eab" Oct 30 09:43:38.504716 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:38.504683 4562 scope.go:122] "RemoveContainer" containerID="b2d864067d4d624414bd20ba4b672fa53ec612950c979567188bee27709b7eab" Oct 30 09:43:38.505026 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:43:38.505006 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2d864067d4d624414bd20ba4b672fa53ec612950c979567188bee27709b7eab\": container with ID starting with b2d864067d4d624414bd20ba4b672fa53ec612950c979567188bee27709b7eab not found: ID does not exist" containerID="b2d864067d4d624414bd20ba4b672fa53ec612950c979567188bee27709b7eab" Oct 30 09:43:38.505105 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:38.505034 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2d864067d4d624414bd20ba4b672fa53ec612950c979567188bee27709b7eab"} err="failed to get container status \"b2d864067d4d624414bd20ba4b672fa53ec612950c979567188bee27709b7eab\": rpc error: code = NotFound desc = could not find container \"b2d864067d4d624414bd20ba4b672fa53ec612950c979567188bee27709b7eab\": container with ID starting with b2d864067d4d624414bd20ba4b672fa53ec612950c979567188bee27709b7eab not found: ID does not exist" Oct 30 09:43:38.524757 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:38.524731 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6411fcd0-5e54-47e6-97e6-ba80f1434df2-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6411fcd0-5e54-47e6-97e6-ba80f1434df2-routing-overlay-volume\") pod \"6411fcd0-5e54-47e6-97e6-ba80f1434df2\" (UID: \"6411fcd0-5e54-47e6-97e6-ba80f1434df2\") " Oct 30 09:43:38.524887 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:38.524873 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6411fcd0-5e54-47e6-97e6-ba80f1434df2-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6411fcd0-5e54-47e6-97e6-ba80f1434df2-provider-credentials\") pod \"6411fcd0-5e54-47e6-97e6-ba80f1434df2\" (UID: \"6411fcd0-5e54-47e6-97e6-ba80f1434df2\") " Oct 30 09:43:38.524960 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:38.524915 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6411fcd0-5e54-47e6-97e6-ba80f1434df2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6411fcd0-5e54-47e6-97e6-ba80f1434df2-plugins-config-volume\") pod \"6411fcd0-5e54-47e6-97e6-ba80f1434df2\" (UID: \"6411fcd0-5e54-47e6-97e6-ba80f1434df2\") " Oct 30 09:43:38.525095 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:38.525076 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6411fcd0-5e54-47e6-97e6-ba80f1434df2-kube-api-access-hgvqs\" (UniqueName: \"kubernetes.io/projected/6411fcd0-5e54-47e6-97e6-ba80f1434df2-kube-api-access-hgvqs\") pod \"6411fcd0-5e54-47e6-97e6-ba80f1434df2\" (UID: \"6411fcd0-5e54-47e6-97e6-ba80f1434df2\") " Oct 30 09:43:38.525230 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:38.525106 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6411fcd0-5e54-47e6-97e6-ba80f1434df2-routing-overlay-volume" pod "6411fcd0-5e54-47e6-97e6-ba80f1434df2" (UID: "6411fcd0-5e54-47e6-97e6-ba80f1434df2"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:43:38.525440 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:38.525365 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6411fcd0-5e54-47e6-97e6-ba80f1434df2-plugins-config-volume" pod "6411fcd0-5e54-47e6-97e6-ba80f1434df2" (UID: "6411fcd0-5e54-47e6-97e6-ba80f1434df2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:43:38.525754 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:38.525728 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6411fcd0-5e54-47e6-97e6-ba80f1434df2-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:43:38.525754 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:38.525754 4562 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6411fcd0-5e54-47e6-97e6-ba80f1434df2-routing-overlay-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:43:38.527098 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:38.527078 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6411fcd0-5e54-47e6-97e6-ba80f1434df2-provider-credentials" pod "6411fcd0-5e54-47e6-97e6-ba80f1434df2" (UID: "6411fcd0-5e54-47e6-97e6-ba80f1434df2"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:43:38.527239 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:38.527217 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6411fcd0-5e54-47e6-97e6-ba80f1434df2-kube-api-access-hgvqs" pod "6411fcd0-5e54-47e6-97e6-ba80f1434df2" (UID: "6411fcd0-5e54-47e6-97e6-ba80f1434df2"). InnerVolumeSpecName "kube-api-access-hgvqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:43:38.640154 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:38.637719 4562 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6411fcd0-5e54-47e6-97e6-ba80f1434df2-provider-credentials\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:43:38.640154 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:38.637761 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgvqs\" (UniqueName: \"kubernetes.io/projected/6411fcd0-5e54-47e6-97e6-ba80f1434df2-kube-api-access-hgvqs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:43:38.845007 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:38.844975 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-f58f32/payload-processing-external-model-e2e-extmod-b-f58f32-84d5lhn8r"] Oct 30 09:43:38.848624 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:38.848593 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-f58f32/payload-processing-external-model-e2e-extmod-b-f58f32-84d5lhn8r"] Oct 30 09:43:39.879513 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:39.879480 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6411fcd0-5e54-47e6-97e6-ba80f1434df2" path="/var/lib/kubelet/pods/6411fcd0-5e54-47e6-97e6-ba80f1434df2/volumes" Oct 30 09:43:47.670662 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:47.670622 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-6c845vbkc"] Oct 30 09:43:47.671375 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:47.671361 4562 cpu_manager.go:414] "containerMap: removing container" podUID="19c6e7c8-d41e-463e-8543-31f5fdc072b8" containerName="payload-processing" Oct 30 09:43:47.671375 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:47.671375 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19c6e7c8-d41e-463e-8543-31f5fdc072b8" containerName="payload-processing" Oct 30 09:43:47.671503 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:47.671427 4562 cpu_manager.go:414] "containerMap: removing container" podUID="6411fcd0-5e54-47e6-97e6-ba80f1434df2" containerName="payload-processing" Oct 30 09:43:47.671503 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:47.671433 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6411fcd0-5e54-47e6-97e6-ba80f1434df2" containerName="payload-processing" Oct 30 09:43:47.671573 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:47.671535 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="19c6e7c8-d41e-463e-8543-31f5fdc072b8" containerName="payload-processing" Oct 30 09:43:47.671573 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:47.671548 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="6411fcd0-5e54-47e6-97e6-ba80f1434df2" containerName="payload-processing" Oct 30 09:43:47.676788 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:47.676768 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-6c845vbkc" Oct 30 09:43:47.696784 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:47.696760 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-6c845vbkc"] Oct 30 09:43:47.705363 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:47.705340 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3d9b36\"/\"agc-cred-a052f8898cfd\"" Oct 30 09:43:47.759850 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:47.759807 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52ee1a5d-2e5c-460c-a8c8-92eeb570bb5b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-6c845vbkc\" (UID: \"52ee1a5d-2e5c-460c-a8c8-92eeb570bb5b\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-6c845vbkc" Oct 30 09:43:47.760045 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:47.760020 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/52ee1a5d-2e5c-460c-a8c8-92eeb570bb5b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-6c845vbkc\" (UID: \"52ee1a5d-2e5c-460c-a8c8-92eeb570bb5b\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-6c845vbkc" Oct 30 09:43:47.760298 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:47.760274 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmz2s\" (UniqueName: \"kubernetes.io/projected/52ee1a5d-2e5c-460c-a8c8-92eeb570bb5b-kube-api-access-bmz2s\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-6c845vbkc\" (UID: \"52ee1a5d-2e5c-460c-a8c8-92eeb570bb5b\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-6c845vbkc" Oct 30 09:43:47.760414 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:47.760370 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/52ee1a5d-2e5c-460c-a8c8-92eeb570bb5b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-6c845vbkc\" (UID: \"52ee1a5d-2e5c-460c-a8c8-92eeb570bb5b\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-6c845vbkc" Oct 30 09:43:47.861700 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:47.861664 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/52ee1a5d-2e5c-460c-a8c8-92eeb570bb5b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-6c845vbkc\" (UID: \"52ee1a5d-2e5c-460c-a8c8-92eeb570bb5b\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-6c845vbkc" Oct 30 09:43:47.861875 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:47.861725 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52ee1a5d-2e5c-460c-a8c8-92eeb570bb5b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-6c845vbkc\" (UID: \"52ee1a5d-2e5c-460c-a8c8-92eeb570bb5b\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-6c845vbkc" Oct 30 09:43:47.861875 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:47.861864 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/52ee1a5d-2e5c-460c-a8c8-92eeb570bb5b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-6c845vbkc\" (UID: \"52ee1a5d-2e5c-460c-a8c8-92eeb570bb5b\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-6c845vbkc" Oct 30 09:43:47.862095 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:47.862075 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmz2s\" (UniqueName: \"kubernetes.io/projected/52ee1a5d-2e5c-460c-a8c8-92eeb570bb5b-kube-api-access-bmz2s\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-6c845vbkc\" (UID: \"52ee1a5d-2e5c-460c-a8c8-92eeb570bb5b\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-6c845vbkc" Oct 30 09:43:47.862534 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:47.862512 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/52ee1a5d-2e5c-460c-a8c8-92eeb570bb5b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-6c845vbkc\" (UID: \"52ee1a5d-2e5c-460c-a8c8-92eeb570bb5b\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-6c845vbkc" Oct 30 09:43:47.862648 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:47.862548 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52ee1a5d-2e5c-460c-a8c8-92eeb570bb5b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-6c845vbkc\" (UID: \"52ee1a5d-2e5c-460c-a8c8-92eeb570bb5b\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-6c845vbkc" Oct 30 09:43:47.864333 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:47.864315 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/52ee1a5d-2e5c-460c-a8c8-92eeb570bb5b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-6c845vbkc\" (UID: \"52ee1a5d-2e5c-460c-a8c8-92eeb570bb5b\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-6c845vbkc" Oct 30 09:43:47.882952 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:47.882929 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmz2s\" (UniqueName: \"kubernetes.io/projected/52ee1a5d-2e5c-460c-a8c8-92eeb570bb5b-kube-api-access-bmz2s\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-6c845vbkc\" (UID: \"52ee1a5d-2e5c-460c-a8c8-92eeb570bb5b\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-6c845vbkc" Oct 30 09:43:47.992450 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:47.992406 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-6c845vbkc" Oct 30 09:43:48.301727 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:43:48.301695 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52ee1a5d_2e5c_460c_a8c8_92eeb570bb5b.slice/crio-3cf1d4ee7fbc9000a3fddea2b5b1dc16470d606058e357e5e413786f62fd3eb7 WatchSource:0}: Error finding container 3cf1d4ee7fbc9000a3fddea2b5b1dc16470d606058e357e5e413786f62fd3eb7: Status 404 returned error can't find the container with id 3cf1d4ee7fbc9000a3fddea2b5b1dc16470d606058e357e5e413786f62fd3eb7 Oct 30 09:43:48.302434 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:48.302404 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-6c845vbkc"] Oct 30 09:43:48.633747 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:48.633652 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-6c845vbkc" event={"ID":"52ee1a5d-2e5c-460c-a8c8-92eeb570bb5b","Type":"ContainerStarted","Data":"5ffb5f2b789e421a7c2852caa33f092737b85ada4bac293f824740e9bbf38532"} Oct 30 09:43:48.633747 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:48.633718 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-6c845vbkc" event={"ID":"52ee1a5d-2e5c-460c-a8c8-92eeb570bb5b","Type":"ContainerStarted","Data":"3cf1d4ee7fbc9000a3fddea2b5b1dc16470d606058e357e5e413786f62fd3eb7"} Oct 30 09:43:48.633942 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:48.633749 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-6c845vbkc" Oct 30 09:43:48.675154 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:48.675099 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-6c845vbkc" podStartSLOduration=1.675084258 podStartE2EDuration="1.675084258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:43:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:43:48.673775915 +0000 UTC m=+2988.935336904" watchObservedRunningTime="2026-10-30 09:43:48.675084258 +0000 UTC m=+2988.936645246" Oct 30 09:43:56.605562 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:56.605522 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d66d22-7bccc4d786-z6qjp"] Oct 30 09:43:56.611337 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:56.611312 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-d66d22-7bccc4d786-z6qjp" Oct 30 09:43:56.638257 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:56.638221 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-d66d22\"" Oct 30 09:43:56.642336 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:56.642311 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d66d22-7bccc4d786-z6qjp"] Oct 30 09:43:56.797565 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:56.791986 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgc4v\" (UniqueName: \"kubernetes.io/projected/c8073ce8-37bf-4ad9-9311-d17bdc00fbb0-kube-api-access-qgc4v\") pod \"payload-processing-e2e-extmod-b-d66d22-7bccc4d786-z6qjp\" (UID: \"c8073ce8-37bf-4ad9-9311-d17bdc00fbb0\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d66d22-7bccc4d786-z6qjp" Oct 30 09:43:56.809694 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:56.809652 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8073ce8-37bf-4ad9-9311-d17bdc00fbb0-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-d66d22-7bccc4d786-z6qjp\" (UID: \"c8073ce8-37bf-4ad9-9311-d17bdc00fbb0\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d66d22-7bccc4d786-z6qjp" Oct 30 09:43:56.914835 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:56.914723 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8073ce8-37bf-4ad9-9311-d17bdc00fbb0-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-d66d22-7bccc4d786-z6qjp\" (UID: \"c8073ce8-37bf-4ad9-9311-d17bdc00fbb0\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d66d22-7bccc4d786-z6qjp" Oct 30 09:43:56.915029 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:56.914976 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgc4v\" (UniqueName: \"kubernetes.io/projected/c8073ce8-37bf-4ad9-9311-d17bdc00fbb0-kube-api-access-qgc4v\") pod \"payload-processing-e2e-extmod-b-d66d22-7bccc4d786-z6qjp\" (UID: \"c8073ce8-37bf-4ad9-9311-d17bdc00fbb0\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d66d22-7bccc4d786-z6qjp" Oct 30 09:43:56.917973 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:56.917938 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8073ce8-37bf-4ad9-9311-d17bdc00fbb0-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-d66d22-7bccc4d786-z6qjp\" (UID: \"c8073ce8-37bf-4ad9-9311-d17bdc00fbb0\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d66d22-7bccc4d786-z6qjp" Oct 30 09:43:56.955415 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:56.953520 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgc4v\" (UniqueName: \"kubernetes.io/projected/c8073ce8-37bf-4ad9-9311-d17bdc00fbb0-kube-api-access-qgc4v\") pod \"payload-processing-e2e-extmod-b-d66d22-7bccc4d786-z6qjp\" (UID: \"c8073ce8-37bf-4ad9-9311-d17bdc00fbb0\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d66d22-7bccc4d786-z6qjp" Oct 30 09:43:57.227680 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:57.227644 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-d66d22-7bccc4d786-z6qjp" Oct 30 09:43:57.536704 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:43:57.536674 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8073ce8_37bf_4ad9_9311_d17bdc00fbb0.slice/crio-8a8d201e1ea4841d5bf9e95fe38aee8c7442bf47f90c7d333d055f5760c472a1 WatchSource:0}: Error finding container 8a8d201e1ea4841d5bf9e95fe38aee8c7442bf47f90c7d333d055f5760c472a1: Status 404 returned error can't find the container with id 8a8d201e1ea4841d5bf9e95fe38aee8c7442bf47f90c7d333d055f5760c472a1 Oct 30 09:43:57.538719 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:57.538693 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d66d22-7bccc4d786-z6qjp"] Oct 30 09:43:57.714812 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:57.714762 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-d66d22-7bccc4d786-z6qjp" event={"ID":"c8073ce8-37bf-4ad9-9311-d17bdc00fbb0","Type":"ContainerStarted","Data":"76fecc19b8e65f4c5b62f01ae82c02e62b583d33352272559248ee08fa36b3fd"} Oct 30 09:43:57.715242 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:57.714841 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-d66d22-7bccc4d786-z6qjp" event={"ID":"c8073ce8-37bf-4ad9-9311-d17bdc00fbb0","Type":"ContainerStarted","Data":"8a8d201e1ea4841d5bf9e95fe38aee8c7442bf47f90c7d333d055f5760c472a1"} Oct 30 09:43:57.715242 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:57.714874 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-d66d22-7bccc4d786-z6qjp" Oct 30 09:43:57.762736 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:57.762686 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-d66d22-7bccc4d786-z6qjp" podStartSLOduration=1.7626718829999999 podStartE2EDuration="1.762671883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:43:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:43:57.759321389 +0000 UTC m=+2998.020882378" watchObservedRunningTime="2026-10-30 09:43:57.762671883 +0000 UTC m=+2998.024232872" Oct 30 09:43:58.598276 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:43:58.598240 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:43:59.644509 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:59.644481 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-6c845vbkc" Oct 30 09:43:59.724140 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:43:59.724089 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-7555n6htd"] Oct 30 09:44:00.303893 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:00.303865 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6q7n_1f51db5a-5720-4dea-ad02-f336b35d43b1/ovn-acl-logging/0.log" Oct 30 09:44:00.321804 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:00.321774 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6q7n_1f51db5a-5720-4dea-ad02-f336b35d43b1/ovn-acl-logging/0.log" Oct 30 09:44:02.305486 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:02.305449 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-85fdfbb6b6-vkrhd"] Oct 30 09:44:04.725590 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:04.725525 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-7555n6htd" podUID="ffe2fbb7-b680-4353-852d-c571af522c9f" containerName="payload-processing" containerID="cri-o://361d1e783c0ba0d28d66e14e8df5ea7fadeb6308a08c5b9223018cd8e2a26056" gracePeriod=25 Oct 30 09:44:05.012501 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:05.012472 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-7555n6htd" Oct 30 09:44:05.170117 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:05.170084 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffe2fbb7-b680-4353-852d-c571af522c9f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ffe2fbb7-b680-4353-852d-c571af522c9f-routing-overlay-volume\") pod \"ffe2fbb7-b680-4353-852d-c571af522c9f\" (UID: \"ffe2fbb7-b680-4353-852d-c571af522c9f\") " Oct 30 09:44:05.170503 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:05.170152 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffe2fbb7-b680-4353-852d-c571af522c9f-kube-api-access-4w7wh\" (UniqueName: \"kubernetes.io/projected/ffe2fbb7-b680-4353-852d-c571af522c9f-kube-api-access-4w7wh\") pod \"ffe2fbb7-b680-4353-852d-c571af522c9f\" (UID: \"ffe2fbb7-b680-4353-852d-c571af522c9f\") " Oct 30 09:44:05.170503 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:05.170396 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffe2fbb7-b680-4353-852d-c571af522c9f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffe2fbb7-b680-4353-852d-c571af522c9f-plugins-config-volume\") pod \"ffe2fbb7-b680-4353-852d-c571af522c9f\" (UID: \"ffe2fbb7-b680-4353-852d-c571af522c9f\") " Oct 30 09:44:05.170503 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:05.170434 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffe2fbb7-b680-4353-852d-c571af522c9f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ffe2fbb7-b680-4353-852d-c571af522c9f-provider-credentials\") pod \"ffe2fbb7-b680-4353-852d-c571af522c9f\" (UID: \"ffe2fbb7-b680-4353-852d-c571af522c9f\") " Oct 30 09:44:05.171716 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:05.171691 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffe2fbb7-b680-4353-852d-c571af522c9f-routing-overlay-volume" pod "ffe2fbb7-b680-4353-852d-c571af522c9f" (UID: "ffe2fbb7-b680-4353-852d-c571af522c9f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:44:05.171845 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:05.171760 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffe2fbb7-b680-4353-852d-c571af522c9f-plugins-config-volume" pod "ffe2fbb7-b680-4353-852d-c571af522c9f" (UID: "ffe2fbb7-b680-4353-852d-c571af522c9f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:44:05.173229 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:05.173202 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffe2fbb7-b680-4353-852d-c571af522c9f-provider-credentials" pod "ffe2fbb7-b680-4353-852d-c571af522c9f" (UID: "ffe2fbb7-b680-4353-852d-c571af522c9f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:44:05.173373 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:05.173352 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffe2fbb7-b680-4353-852d-c571af522c9f-kube-api-access-4w7wh" pod "ffe2fbb7-b680-4353-852d-c571af522c9f" (UID: "ffe2fbb7-b680-4353-852d-c571af522c9f"). InnerVolumeSpecName "kube-api-access-4w7wh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:44:05.272591 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:05.272512 4562 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ffe2fbb7-b680-4353-852d-c571af522c9f-routing-overlay-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:44:05.272591 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:05.272543 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w7wh\" (UniqueName: \"kubernetes.io/projected/ffe2fbb7-b680-4353-852d-c571af522c9f-kube-api-access-4w7wh\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:44:05.272591 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:05.272554 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffe2fbb7-b680-4353-852d-c571af522c9f-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:44:05.272591 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:05.272563 4562 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ffe2fbb7-b680-4353-852d-c571af522c9f-provider-credentials\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:44:05.787442 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:05.787399 4562 generic.go:356] "Generic (PLEG): container finished" podID="ffe2fbb7-b680-4353-852d-c571af522c9f" containerID="361d1e783c0ba0d28d66e14e8df5ea7fadeb6308a08c5b9223018cd8e2a26056" exitCode=0 Oct 30 09:44:05.787940 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:05.787463 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-7555n6htd" Oct 30 09:44:05.787940 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:05.787489 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-7555n6htd" event={"ID":"ffe2fbb7-b680-4353-852d-c571af522c9f","Type":"ContainerDied","Data":"361d1e783c0ba0d28d66e14e8df5ea7fadeb6308a08c5b9223018cd8e2a26056"} Oct 30 09:44:05.787940 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:05.787565 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-7555n6htd" event={"ID":"ffe2fbb7-b680-4353-852d-c571af522c9f","Type":"ContainerDied","Data":"aa77844aedc87ccea8afaccb6190e3ff1a6b81d9f0683d0332fcfe7e0e44e866"} Oct 30 09:44:05.787940 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:05.787589 4562 scope.go:122] "RemoveContainer" containerID="361d1e783c0ba0d28d66e14e8df5ea7fadeb6308a08c5b9223018cd8e2a26056" Oct 30 09:44:05.800431 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:05.800411 4562 scope.go:122] "RemoveContainer" containerID="361d1e783c0ba0d28d66e14e8df5ea7fadeb6308a08c5b9223018cd8e2a26056" Oct 30 09:44:05.800750 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:44:05.800729 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"361d1e783c0ba0d28d66e14e8df5ea7fadeb6308a08c5b9223018cd8e2a26056\": container with ID starting with 361d1e783c0ba0d28d66e14e8df5ea7fadeb6308a08c5b9223018cd8e2a26056 not found: ID does not exist" containerID="361d1e783c0ba0d28d66e14e8df5ea7fadeb6308a08c5b9223018cd8e2a26056" Oct 30 09:44:05.800806 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:05.800759 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"361d1e783c0ba0d28d66e14e8df5ea7fadeb6308a08c5b9223018cd8e2a26056"} err="failed to get container status \"361d1e783c0ba0d28d66e14e8df5ea7fadeb6308a08c5b9223018cd8e2a26056\": rpc error: code = NotFound desc = could not find container \"361d1e783c0ba0d28d66e14e8df5ea7fadeb6308a08c5b9223018cd8e2a26056\": container with ID starting with 361d1e783c0ba0d28d66e14e8df5ea7fadeb6308a08c5b9223018cd8e2a26056 not found: ID does not exist" Oct 30 09:44:05.844770 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:05.844734 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-7555n6htd"] Oct 30 09:44:05.855844 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:05.855812 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-7555n6htd"] Oct 30 09:44:05.879247 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:05.879203 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffe2fbb7-b680-4353-852d-c571af522c9f" path="/var/lib/kubelet/pods/ffe2fbb7-b680-4353-852d-c571af522c9f/volumes" Oct 30 09:44:07.305992 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:07.305948 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-85fdfbb6b6-vkrhd" podUID="7fa57945-98fb-4135-bbdb-db117285ee4e" containerName="payload-processing" containerID="cri-o://ae1c96d3fc5987e9473151e98bce72a52b260e3c0887abcbf101ed343bb1d168" gracePeriod=25 Oct 30 09:44:07.611629 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:07.611606 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-85fdfbb6b6-vkrhd" Oct 30 09:44:07.710284 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:07.710246 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fa57945-98fb-4135-bbdb-db117285ee4e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fa57945-98fb-4135-bbdb-db117285ee4e-plugins-config-volume\") pod \"7fa57945-98fb-4135-bbdb-db117285ee4e\" (UID: \"7fa57945-98fb-4135-bbdb-db117285ee4e\") " Oct 30 09:44:07.710515 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:07.710291 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fa57945-98fb-4135-bbdb-db117285ee4e-kube-api-access-92nfd\" (UniqueName: \"kubernetes.io/projected/7fa57945-98fb-4135-bbdb-db117285ee4e-kube-api-access-92nfd\") pod \"7fa57945-98fb-4135-bbdb-db117285ee4e\" (UID: \"7fa57945-98fb-4135-bbdb-db117285ee4e\") " Oct 30 09:44:07.710515 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:07.710348 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fa57945-98fb-4135-bbdb-db117285ee4e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7fa57945-98fb-4135-bbdb-db117285ee4e-provider-credentials\") pod \"7fa57945-98fb-4135-bbdb-db117285ee4e\" (UID: \"7fa57945-98fb-4135-bbdb-db117285ee4e\") " Oct 30 09:44:07.710515 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:07.710412 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fa57945-98fb-4135-bbdb-db117285ee4e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7fa57945-98fb-4135-bbdb-db117285ee4e-routing-overlay-volume\") pod \"7fa57945-98fb-4135-bbdb-db117285ee4e\" (UID: \"7fa57945-98fb-4135-bbdb-db117285ee4e\") " Oct 30 09:44:07.710826 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:07.710784 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fa57945-98fb-4135-bbdb-db117285ee4e-plugins-config-volume" pod "7fa57945-98fb-4135-bbdb-db117285ee4e" (UID: "7fa57945-98fb-4135-bbdb-db117285ee4e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:44:07.710968 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:07.710845 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fa57945-98fb-4135-bbdb-db117285ee4e-routing-overlay-volume" pod "7fa57945-98fb-4135-bbdb-db117285ee4e" (UID: "7fa57945-98fb-4135-bbdb-db117285ee4e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:44:07.712981 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:07.712950 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fa57945-98fb-4135-bbdb-db117285ee4e-kube-api-access-92nfd" pod "7fa57945-98fb-4135-bbdb-db117285ee4e" (UID: "7fa57945-98fb-4135-bbdb-db117285ee4e"). InnerVolumeSpecName "kube-api-access-92nfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:44:07.712981 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:07.712971 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fa57945-98fb-4135-bbdb-db117285ee4e-provider-credentials" pod "7fa57945-98fb-4135-bbdb-db117285ee4e" (UID: "7fa57945-98fb-4135-bbdb-db117285ee4e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:44:07.804400 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:07.804344 4562 generic.go:356] "Generic (PLEG): container finished" podID="7fa57945-98fb-4135-bbdb-db117285ee4e" containerID="ae1c96d3fc5987e9473151e98bce72a52b260e3c0887abcbf101ed343bb1d168" exitCode=0 Oct 30 09:44:07.804577 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:07.804406 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-85fdfbb6b6-vkrhd" event={"ID":"7fa57945-98fb-4135-bbdb-db117285ee4e","Type":"ContainerDied","Data":"ae1c96d3fc5987e9473151e98bce72a52b260e3c0887abcbf101ed343bb1d168"} Oct 30 09:44:07.804577 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:07.804424 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-85fdfbb6b6-vkrhd" Oct 30 09:44:07.804577 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:07.804489 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-85fdfbb6b6-vkrhd" event={"ID":"7fa57945-98fb-4135-bbdb-db117285ee4e","Type":"ContainerDied","Data":"03a9c207797a4d7e939d53c890f69e4866ee3c978551dc6a3e00635e5db60e09"} Oct 30 09:44:07.804577 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:07.804513 4562 scope.go:122] "RemoveContainer" containerID="ae1c96d3fc5987e9473151e98bce72a52b260e3c0887abcbf101ed343bb1d168" Oct 30 09:44:07.812228 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:07.812199 4562 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7fa57945-98fb-4135-bbdb-db117285ee4e-provider-credentials\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:44:07.812372 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:07.812233 4562 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7fa57945-98fb-4135-bbdb-db117285ee4e-routing-overlay-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:44:07.812372 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:07.812250 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fa57945-98fb-4135-bbdb-db117285ee4e-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:44:07.812372 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:07.812263 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92nfd\" (UniqueName: \"kubernetes.io/projected/7fa57945-98fb-4135-bbdb-db117285ee4e-kube-api-access-92nfd\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:44:07.815992 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:07.815972 4562 scope.go:122] "RemoveContainer" containerID="ae1c96d3fc5987e9473151e98bce72a52b260e3c0887abcbf101ed343bb1d168" Oct 30 09:44:07.816283 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:44:07.816248 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae1c96d3fc5987e9473151e98bce72a52b260e3c0887abcbf101ed343bb1d168\": container with ID starting with ae1c96d3fc5987e9473151e98bce72a52b260e3c0887abcbf101ed343bb1d168 not found: ID does not exist" containerID="ae1c96d3fc5987e9473151e98bce72a52b260e3c0887abcbf101ed343bb1d168" Oct 30 09:44:07.816333 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:07.816290 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae1c96d3fc5987e9473151e98bce72a52b260e3c0887abcbf101ed343bb1d168"} err="failed to get container status \"ae1c96d3fc5987e9473151e98bce72a52b260e3c0887abcbf101ed343bb1d168\": rpc error: code = NotFound desc = could not find container \"ae1c96d3fc5987e9473151e98bce72a52b260e3c0887abcbf101ed343bb1d168\": container with ID starting with ae1c96d3fc5987e9473151e98bce72a52b260e3c0887abcbf101ed343bb1d168 not found: ID does not exist" Oct 30 09:44:07.882105 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:07.882021 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-85fdfbb6b6-vkrhd"] Oct 30 09:44:07.884617 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:07.884590 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-85fdfbb6b6-vkrhd"] Oct 30 09:44:08.727583 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:08.727555 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-d66d22-7bccc4d786-z6qjp" Oct 30 09:44:09.878247 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:09.878206 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fa57945-98fb-4135-bbdb-db117285ee4e" path="/var/lib/kubelet/pods/7fa57945-98fb-4135-bbdb-db117285ee4e/volumes" Oct 30 09:44:25.454862 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:25.454775 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dwcq7"] Oct 30 09:44:25.455844 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:25.455815 4562 cpu_manager.go:414] "containerMap: removing container" podUID="7fa57945-98fb-4135-bbdb-db117285ee4e" containerName="payload-processing" Oct 30 09:44:25.455844 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:25.455841 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fa57945-98fb-4135-bbdb-db117285ee4e" containerName="payload-processing" Oct 30 09:44:25.456027 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:25.455892 4562 cpu_manager.go:414] "containerMap: removing container" podUID="ffe2fbb7-b680-4353-852d-c571af522c9f" containerName="payload-processing" Oct 30 09:44:25.456027 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:25.455902 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffe2fbb7-b680-4353-852d-c571af522c9f" containerName="payload-processing" Oct 30 09:44:25.456027 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:25.456016 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fa57945-98fb-4135-bbdb-db117285ee4e" containerName="payload-processing" Oct 30 09:44:25.456122 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:25.456029 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffe2fbb7-b680-4353-852d-c571af522c9f" containerName="payload-processing" Oct 30 09:44:25.461474 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:25.461455 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dwcq7" Oct 30 09:44:25.486223 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:25.486195 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dwcq7"] Oct 30 09:44:25.558530 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:25.558497 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb668c39-39ae-443f-9648-c9d5cc7e2d2e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-dwcq7\" (UID: \"cb668c39-39ae-443f-9648-c9d5cc7e2d2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dwcq7" Oct 30 09:44:25.558708 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:25.558643 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb668c39-39ae-443f-9648-c9d5cc7e2d2e-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-dwcq7\" (UID: \"cb668c39-39ae-443f-9648-c9d5cc7e2d2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dwcq7" Oct 30 09:44:25.558759 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:25.558727 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb668c39-39ae-443f-9648-c9d5cc7e2d2e-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-dwcq7\" (UID: \"cb668c39-39ae-443f-9648-c9d5cc7e2d2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dwcq7" Oct 30 09:44:25.558797 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:25.558780 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb668c39-39ae-443f-9648-c9d5cc7e2d2e-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-dwcq7\" (UID: \"cb668c39-39ae-443f-9648-c9d5cc7e2d2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dwcq7" Oct 30 09:44:25.558891 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:25.558874 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85m7k\" (UniqueName: \"kubernetes.io/projected/cb668c39-39ae-443f-9648-c9d5cc7e2d2e-kube-api-access-85m7k\") pod \"maas-default-gateway-openshift-default-77f556d647-dwcq7\" (UID: \"cb668c39-39ae-443f-9648-c9d5cc7e2d2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dwcq7" Oct 30 09:44:25.558931 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:25.558905 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb668c39-39ae-443f-9648-c9d5cc7e2d2e-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-dwcq7\" (UID: \"cb668c39-39ae-443f-9648-c9d5cc7e2d2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dwcq7" Oct 30 09:44:25.559012 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:25.558992 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb668c39-39ae-443f-9648-c9d5cc7e2d2e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-dwcq7\" (UID: \"cb668c39-39ae-443f-9648-c9d5cc7e2d2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dwcq7" Oct 30 09:44:25.559049 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:25.559034 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb668c39-39ae-443f-9648-c9d5cc7e2d2e-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-dwcq7\" (UID: \"cb668c39-39ae-443f-9648-c9d5cc7e2d2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dwcq7" Oct 30 09:44:25.559081 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:25.559058 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb668c39-39ae-443f-9648-c9d5cc7e2d2e-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-dwcq7\" (UID: \"cb668c39-39ae-443f-9648-c9d5cc7e2d2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dwcq7" Oct 30 09:44:25.660374 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:25.660339 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb668c39-39ae-443f-9648-c9d5cc7e2d2e-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-dwcq7\" (UID: \"cb668c39-39ae-443f-9648-c9d5cc7e2d2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dwcq7" Oct 30 09:44:25.660571 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:25.660420 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb668c39-39ae-443f-9648-c9d5cc7e2d2e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-dwcq7\" (UID: \"cb668c39-39ae-443f-9648-c9d5cc7e2d2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dwcq7" Oct 30 09:44:25.660571 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:25.660452 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb668c39-39ae-443f-9648-c9d5cc7e2d2e-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-dwcq7\" (UID: \"cb668c39-39ae-443f-9648-c9d5cc7e2d2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dwcq7" Oct 30 09:44:25.660571 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:25.660470 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb668c39-39ae-443f-9648-c9d5cc7e2d2e-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-dwcq7\" (UID: \"cb668c39-39ae-443f-9648-c9d5cc7e2d2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dwcq7" Oct 30 09:44:25.660571 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:25.660511 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb668c39-39ae-443f-9648-c9d5cc7e2d2e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-dwcq7\" (UID: \"cb668c39-39ae-443f-9648-c9d5cc7e2d2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dwcq7" Oct 30 09:44:25.660806 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:25.660628 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb668c39-39ae-443f-9648-c9d5cc7e2d2e-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-dwcq7\" (UID: \"cb668c39-39ae-443f-9648-c9d5cc7e2d2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dwcq7" Oct 30 09:44:25.660806 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:25.660681 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb668c39-39ae-443f-9648-c9d5cc7e2d2e-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-dwcq7\" (UID: \"cb668c39-39ae-443f-9648-c9d5cc7e2d2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dwcq7" Oct 30 09:44:25.660806 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:25.660745 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb668c39-39ae-443f-9648-c9d5cc7e2d2e-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-dwcq7\" (UID: \"cb668c39-39ae-443f-9648-c9d5cc7e2d2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dwcq7" Oct 30 09:44:25.661335 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:25.660818 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85m7k\" (UniqueName: \"kubernetes.io/projected/cb668c39-39ae-443f-9648-c9d5cc7e2d2e-kube-api-access-85m7k\") pod \"maas-default-gateway-openshift-default-77f556d647-dwcq7\" (UID: \"cb668c39-39ae-443f-9648-c9d5cc7e2d2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dwcq7" Oct 30 09:44:25.661335 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:25.660847 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb668c39-39ae-443f-9648-c9d5cc7e2d2e-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-dwcq7\" (UID: \"cb668c39-39ae-443f-9648-c9d5cc7e2d2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dwcq7" Oct 30 09:44:25.661335 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:25.660961 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb668c39-39ae-443f-9648-c9d5cc7e2d2e-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-dwcq7\" (UID: \"cb668c39-39ae-443f-9648-c9d5cc7e2d2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dwcq7" Oct 30 09:44:25.661335 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:25.661039 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb668c39-39ae-443f-9648-c9d5cc7e2d2e-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-dwcq7\" (UID: \"cb668c39-39ae-443f-9648-c9d5cc7e2d2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dwcq7" Oct 30 09:44:25.661335 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:25.661059 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb668c39-39ae-443f-9648-c9d5cc7e2d2e-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-dwcq7\" (UID: \"cb668c39-39ae-443f-9648-c9d5cc7e2d2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dwcq7" Oct 30 09:44:25.661570 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:25.661400 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb668c39-39ae-443f-9648-c9d5cc7e2d2e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-dwcq7\" (UID: \"cb668c39-39ae-443f-9648-c9d5cc7e2d2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dwcq7" Oct 30 09:44:25.663024 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:25.663001 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb668c39-39ae-443f-9648-c9d5cc7e2d2e-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-dwcq7\" (UID: \"cb668c39-39ae-443f-9648-c9d5cc7e2d2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dwcq7" Oct 30 09:44:25.663219 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:25.663199 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb668c39-39ae-443f-9648-c9d5cc7e2d2e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-dwcq7\" (UID: \"cb668c39-39ae-443f-9648-c9d5cc7e2d2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dwcq7" Oct 30 09:44:25.688440 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:25.688397 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb668c39-39ae-443f-9648-c9d5cc7e2d2e-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-dwcq7\" (UID: \"cb668c39-39ae-443f-9648-c9d5cc7e2d2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dwcq7" Oct 30 09:44:25.688627 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:25.688605 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85m7k\" (UniqueName: \"kubernetes.io/projected/cb668c39-39ae-443f-9648-c9d5cc7e2d2e-kube-api-access-85m7k\") pod \"maas-default-gateway-openshift-default-77f556d647-dwcq7\" (UID: \"cb668c39-39ae-443f-9648-c9d5cc7e2d2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dwcq7" Oct 30 09:44:25.777022 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:25.776931 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dwcq7" Oct 30 09:44:26.112059 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:44:26.112023 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb668c39_39ae_443f_9648_c9d5cc7e2d2e.slice/crio-60c6c6a2b545ef54e4ec3fe3a8d06e367586d2ba2a527b36e9e58ccaba21b2bd WatchSource:0}: Error finding container 60c6c6a2b545ef54e4ec3fe3a8d06e367586d2ba2a527b36e9e58ccaba21b2bd: Status 404 returned error can't find the container with id 60c6c6a2b545ef54e4ec3fe3a8d06e367586d2ba2a527b36e9e58ccaba21b2bd Oct 30 09:44:26.112214 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:26.112190 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dwcq7"] Oct 30 09:44:26.116142 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:26.115405 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:44:26.116142 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:26.115560 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:44:26.116142 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:26.115655 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:44:27.096060 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:27.096017 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dwcq7" event={"ID":"cb668c39-39ae-443f-9648-c9d5cc7e2d2e","Type":"ContainerStarted","Data":"40191ed0a14c3ee6a690dfa6034eed64cb9062769c65d488e5d97a20545e0f97"} Oct 30 09:44:27.096620 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:27.096086 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dwcq7" event={"ID":"cb668c39-39ae-443f-9648-c9d5cc7e2d2e","Type":"ContainerStarted","Data":"60c6c6a2b545ef54e4ec3fe3a8d06e367586d2ba2a527b36e9e58ccaba21b2bd"} Oct 30 09:44:27.174581 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:27.174524 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dwcq7" podStartSLOduration=2.174508781 podStartE2EDuration="2.174508781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:44:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:44:27.160924017 +0000 UTC m=+3027.422485030" watchObservedRunningTime="2026-10-30 09:44:27.174508781 +0000 UTC m=+3027.436069770" Oct 30 09:44:27.777526 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:27.777489 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dwcq7" Oct 30 09:44:27.779159 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:27.779132 4562 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-dwcq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body= Oct 30 09:44:27.779274 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:27.779183 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dwcq7" podUID="cb668c39-39ae-443f-9648-c9d5cc7e2d2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" Oct 30 09:44:28.733149 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:44:28.733107 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:44:28.778484 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:28.777801 4562 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-dwcq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body= Oct 30 09:44:28.778484 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:28.777878 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dwcq7" podUID="cb668c39-39ae-443f-9648-c9d5cc7e2d2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" Oct 30 09:44:29.778404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:29.778350 4562 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-dwcq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body= Oct 30 09:44:29.778906 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:29.778435 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dwcq7" podUID="cb668c39-39ae-443f-9648-c9d5cc7e2d2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" Oct 30 09:44:30.175536 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:30.167131 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-5d5cmp6tj"] Oct 30 09:44:30.185781 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:30.185750 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-5d5cmp6tj"] Oct 30 09:44:30.185965 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:30.185860 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-5d5cmp6tj" Oct 30 09:44:30.201502 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:30.201467 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-881f83\"/\"agc-cred-fa5ab9b30048\"" Oct 30 09:44:30.201747 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:30.201603 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-881f83\"/\"openshift-service-ca.crt\"" Oct 30 09:44:30.205080 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:30.205051 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-881f83\"/\"routing-overlay\"" Oct 30 09:44:30.205080 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:30.205117 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-881f83\"/\"kube-root-ca.crt\"" Oct 30 09:44:30.214738 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:30.214709 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-881f83\"/\"payload-processing-external-model-plugins-e2e-extmod-a-881f83\"" Oct 30 09:44:30.232583 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:30.215050 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-881f83\"/\"payload-processing-external-model-e2e--d06f4d1b-dockercfg-nr74x\"" Oct 30 09:44:30.232865 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:30.216452 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-881f83\"/\"agc-cred-cf9bd9a0dc6b\"" Oct 30 09:44:30.485405 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:30.484356 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5mz7\" (UniqueName: \"kubernetes.io/projected/3fca0ddf-66ff-4161-8f0c-94d895001d3e-kube-api-access-v5mz7\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-5d5cmp6tj\" (UID: \"3fca0ddf-66ff-4161-8f0c-94d895001d3e\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-5d5cmp6tj" Oct 30 09:44:30.485405 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:30.484560 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3fca0ddf-66ff-4161-8f0c-94d895001d3e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-5d5cmp6tj\" (UID: \"3fca0ddf-66ff-4161-8f0c-94d895001d3e\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-5d5cmp6tj" Oct 30 09:44:30.485405 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:30.484638 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3fca0ddf-66ff-4161-8f0c-94d895001d3e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-5d5cmp6tj\" (UID: \"3fca0ddf-66ff-4161-8f0c-94d895001d3e\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-5d5cmp6tj" Oct 30 09:44:30.485405 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:30.484682 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fca0ddf-66ff-4161-8f0c-94d895001d3e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-5d5cmp6tj\" (UID: \"3fca0ddf-66ff-4161-8f0c-94d895001d3e\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-5d5cmp6tj" Oct 30 09:44:30.585846 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:30.585808 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5mz7\" (UniqueName: \"kubernetes.io/projected/3fca0ddf-66ff-4161-8f0c-94d895001d3e-kube-api-access-v5mz7\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-5d5cmp6tj\" (UID: \"3fca0ddf-66ff-4161-8f0c-94d895001d3e\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-5d5cmp6tj" Oct 30 09:44:30.586027 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:30.585913 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3fca0ddf-66ff-4161-8f0c-94d895001d3e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-5d5cmp6tj\" (UID: \"3fca0ddf-66ff-4161-8f0c-94d895001d3e\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-5d5cmp6tj" Oct 30 09:44:30.586027 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:30.585951 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3fca0ddf-66ff-4161-8f0c-94d895001d3e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-5d5cmp6tj\" (UID: \"3fca0ddf-66ff-4161-8f0c-94d895001d3e\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-5d5cmp6tj" Oct 30 09:44:30.586027 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:30.585980 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fca0ddf-66ff-4161-8f0c-94d895001d3e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-5d5cmp6tj\" (UID: \"3fca0ddf-66ff-4161-8f0c-94d895001d3e\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-5d5cmp6tj" Oct 30 09:44:30.586831 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:30.586794 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3fca0ddf-66ff-4161-8f0c-94d895001d3e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-5d5cmp6tj\" (UID: \"3fca0ddf-66ff-4161-8f0c-94d895001d3e\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-5d5cmp6tj" Oct 30 09:44:30.586916 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:30.586899 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fca0ddf-66ff-4161-8f0c-94d895001d3e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-5d5cmp6tj\" (UID: \"3fca0ddf-66ff-4161-8f0c-94d895001d3e\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-5d5cmp6tj" Oct 30 09:44:30.588546 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:30.588526 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3fca0ddf-66ff-4161-8f0c-94d895001d3e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-5d5cmp6tj\" (UID: \"3fca0ddf-66ff-4161-8f0c-94d895001d3e\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-5d5cmp6tj" Oct 30 09:44:30.606099 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:30.606073 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5mz7\" (UniqueName: \"kubernetes.io/projected/3fca0ddf-66ff-4161-8f0c-94d895001d3e-kube-api-access-v5mz7\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-5d5cmp6tj\" (UID: \"3fca0ddf-66ff-4161-8f0c-94d895001d3e\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-5d5cmp6tj" Oct 30 09:44:30.691110 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:30.690786 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-5d5cmp6tj" Oct 30 09:44:30.779009 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:30.778369 4562 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-dwcq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body= Oct 30 09:44:30.779009 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:30.778457 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dwcq7" podUID="cb668c39-39ae-443f-9648-c9d5cc7e2d2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" Oct 30 09:44:31.035012 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:31.034985 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-5d5cmp6tj"] Oct 30 09:44:31.199348 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:31.199314 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-5d5cmp6tj" event={"ID":"3fca0ddf-66ff-4161-8f0c-94d895001d3e","Type":"ContainerStarted","Data":"2c92b6811a41607fd3e012eaef1b64f79c2ebc5bd83248f06a83a2b24fafa77e"} Oct 30 09:44:31.199540 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:31.199371 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-5d5cmp6tj" event={"ID":"3fca0ddf-66ff-4161-8f0c-94d895001d3e","Type":"ContainerStarted","Data":"449bc2298c60f8547c6bd0ab1a8b7819bdb5e116a3d1e161ae920c9ee77c2264"} Oct 30 09:44:31.199540 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:31.199420 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-5d5cmp6tj" Oct 30 09:44:31.210035 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:31.210006 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-5d5cmp6tj"] Oct 30 09:44:31.264543 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:31.264486 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-5d5cmp6tj" podStartSLOduration=1.2644712839999999 podStartE2EDuration="1.264471284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:44:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:44:31.261693336 +0000 UTC m=+3031.523254324" watchObservedRunningTime="2026-10-30 09:44:31.264471284 +0000 UTC m=+3031.526032271" Oct 30 09:44:31.778322 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:31.778282 4562 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-dwcq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body= Oct 30 09:44:31.778575 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:31.778340 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dwcq7" podUID="cb668c39-39ae-443f-9648-c9d5cc7e2d2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" Oct 30 09:44:32.777867 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:32.777829 4562 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-dwcq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body= Oct 30 09:44:32.778355 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:32.777903 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dwcq7" podUID="cb668c39-39ae-443f-9648-c9d5cc7e2d2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" Oct 30 09:44:33.778183 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:33.778148 4562 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-dwcq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body= Oct 30 09:44:33.778635 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:33.778202 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dwcq7" podUID="cb668c39-39ae-443f-9648-c9d5cc7e2d2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" Oct 30 09:44:35.285963 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:35.285929 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dwcq7" Oct 30 09:44:35.286507 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:35.286475 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dwcq7" Oct 30 09:44:35.294525 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:35.294483 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dwcq7" Oct 30 09:44:38.214462 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:38.214421 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-5d5cmp6tj" podUID="3fca0ddf-66ff-4161-8f0c-94d895001d3e" containerName="payload-processing" containerID="cri-o://2c92b6811a41607fd3e012eaef1b64f79c2ebc5bd83248f06a83a2b24fafa77e" gracePeriod=25 Oct 30 09:44:38.358914 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:38.358871 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d66d22-f86b5c5b-7wsft"] Oct 30 09:44:38.363902 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:38.363871 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d66d22-f86b5c5b-7wsft" Oct 30 09:44:38.374654 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:38.374625 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d66d22-f86b5c5b-7wsft"] Oct 30 09:44:38.382623 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:38.380998 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-d66d22-serving-cert\"" Oct 30 09:44:38.477640 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:38.477556 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17acd011-7a83-4ac2-9689-3a5568b7f94b-maas-api-tls\") pod \"maas-api-e2e-extmod-b-d66d22-f86b5c5b-7wsft\" (UID: \"17acd011-7a83-4ac2-9689-3a5568b7f94b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d66d22-f86b5c5b-7wsft" Oct 30 09:44:38.477808 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:38.477699 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17acd011-7a83-4ac2-9689-3a5568b7f94b-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-d66d22-f86b5c5b-7wsft\" (UID: \"17acd011-7a83-4ac2-9689-3a5568b7f94b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d66d22-f86b5c5b-7wsft" Oct 30 09:44:38.477808 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:38.477763 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xstl\" (UniqueName: \"kubernetes.io/projected/17acd011-7a83-4ac2-9689-3a5568b7f94b-kube-api-access-4xstl\") pod \"maas-api-e2e-extmod-b-d66d22-f86b5c5b-7wsft\" (UID: \"17acd011-7a83-4ac2-9689-3a5568b7f94b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d66d22-f86b5c5b-7wsft" Oct 30 09:44:38.477902 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:38.477883 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17acd011-7a83-4ac2-9689-3a5568b7f94b-metrics-certs\") pod \"maas-api-e2e-extmod-b-d66d22-f86b5c5b-7wsft\" (UID: \"17acd011-7a83-4ac2-9689-3a5568b7f94b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d66d22-f86b5c5b-7wsft" Oct 30 09:44:38.544169 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:38.544143 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-5d5cmp6tj" Oct 30 09:44:38.591406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:38.589766 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17acd011-7a83-4ac2-9689-3a5568b7f94b-maas-api-tls\") pod \"maas-api-e2e-extmod-b-d66d22-f86b5c5b-7wsft\" (UID: \"17acd011-7a83-4ac2-9689-3a5568b7f94b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d66d22-f86b5c5b-7wsft" Oct 30 09:44:38.591406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:38.589882 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17acd011-7a83-4ac2-9689-3a5568b7f94b-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-d66d22-f86b5c5b-7wsft\" (UID: \"17acd011-7a83-4ac2-9689-3a5568b7f94b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d66d22-f86b5c5b-7wsft" Oct 30 09:44:38.591406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:38.589937 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xstl\" (UniqueName: \"kubernetes.io/projected/17acd011-7a83-4ac2-9689-3a5568b7f94b-kube-api-access-4xstl\") pod \"maas-api-e2e-extmod-b-d66d22-f86b5c5b-7wsft\" (UID: \"17acd011-7a83-4ac2-9689-3a5568b7f94b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d66d22-f86b5c5b-7wsft" Oct 30 09:44:38.591406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:38.590015 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17acd011-7a83-4ac2-9689-3a5568b7f94b-metrics-certs\") pod \"maas-api-e2e-extmod-b-d66d22-f86b5c5b-7wsft\" (UID: \"17acd011-7a83-4ac2-9689-3a5568b7f94b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d66d22-f86b5c5b-7wsft" Oct 30 09:44:38.597317 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:38.597281 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17acd011-7a83-4ac2-9689-3a5568b7f94b-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-d66d22-f86b5c5b-7wsft\" (UID: \"17acd011-7a83-4ac2-9689-3a5568b7f94b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d66d22-f86b5c5b-7wsft" Oct 30 09:44:38.618046 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:38.605995 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17acd011-7a83-4ac2-9689-3a5568b7f94b-maas-api-tls\") pod \"maas-api-e2e-extmod-b-d66d22-f86b5c5b-7wsft\" (UID: \"17acd011-7a83-4ac2-9689-3a5568b7f94b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d66d22-f86b5c5b-7wsft" Oct 30 09:44:38.639154 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:38.639100 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xstl\" (UniqueName: \"kubernetes.io/projected/17acd011-7a83-4ac2-9689-3a5568b7f94b-kube-api-access-4xstl\") pod \"maas-api-e2e-extmod-b-d66d22-f86b5c5b-7wsft\" (UID: \"17acd011-7a83-4ac2-9689-3a5568b7f94b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d66d22-f86b5c5b-7wsft" Oct 30 09:44:38.644836 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:38.644806 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17acd011-7a83-4ac2-9689-3a5568b7f94b-metrics-certs\") pod \"maas-api-e2e-extmod-b-d66d22-f86b5c5b-7wsft\" (UID: \"17acd011-7a83-4ac2-9689-3a5568b7f94b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d66d22-f86b5c5b-7wsft" Oct 30 09:44:38.688773 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:38.688743 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d66d22-f86b5c5b-7wsft" Oct 30 09:44:38.717800 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:38.717759 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fca0ddf-66ff-4161-8f0c-94d895001d3e-kube-api-access-v5mz7\" (UniqueName: \"kubernetes.io/projected/3fca0ddf-66ff-4161-8f0c-94d895001d3e-kube-api-access-v5mz7\") pod \"3fca0ddf-66ff-4161-8f0c-94d895001d3e\" (UID: \"3fca0ddf-66ff-4161-8f0c-94d895001d3e\") " Oct 30 09:44:38.718005 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:38.717929 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fca0ddf-66ff-4161-8f0c-94d895001d3e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3fca0ddf-66ff-4161-8f0c-94d895001d3e-routing-overlay-volume\") pod \"3fca0ddf-66ff-4161-8f0c-94d895001d3e\" (UID: \"3fca0ddf-66ff-4161-8f0c-94d895001d3e\") " Oct 30 09:44:38.718113 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:38.718089 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fca0ddf-66ff-4161-8f0c-94d895001d3e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3fca0ddf-66ff-4161-8f0c-94d895001d3e-provider-credentials\") pod \"3fca0ddf-66ff-4161-8f0c-94d895001d3e\" (UID: \"3fca0ddf-66ff-4161-8f0c-94d895001d3e\") " Oct 30 09:44:38.718208 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:38.718193 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fca0ddf-66ff-4161-8f0c-94d895001d3e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fca0ddf-66ff-4161-8f0c-94d895001d3e-plugins-config-volume\") pod \"3fca0ddf-66ff-4161-8f0c-94d895001d3e\" (UID: \"3fca0ddf-66ff-4161-8f0c-94d895001d3e\") " Oct 30 09:44:38.718538 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:38.718507 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fca0ddf-66ff-4161-8f0c-94d895001d3e-routing-overlay-volume" pod "3fca0ddf-66ff-4161-8f0c-94d895001d3e" (UID: "3fca0ddf-66ff-4161-8f0c-94d895001d3e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:44:38.718746 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:38.718722 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fca0ddf-66ff-4161-8f0c-94d895001d3e-plugins-config-volume" pod "3fca0ddf-66ff-4161-8f0c-94d895001d3e" (UID: "3fca0ddf-66ff-4161-8f0c-94d895001d3e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:44:38.718889 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:38.718873 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fca0ddf-66ff-4161-8f0c-94d895001d3e-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:44:38.718952 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:38.718890 4562 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3fca0ddf-66ff-4161-8f0c-94d895001d3e-routing-overlay-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:44:38.720668 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:38.720640 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fca0ddf-66ff-4161-8f0c-94d895001d3e-kube-api-access-v5mz7" pod "3fca0ddf-66ff-4161-8f0c-94d895001d3e" (UID: "3fca0ddf-66ff-4161-8f0c-94d895001d3e"). InnerVolumeSpecName "kube-api-access-v5mz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:44:38.720855 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:38.720827 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fca0ddf-66ff-4161-8f0c-94d895001d3e-provider-credentials" pod "3fca0ddf-66ff-4161-8f0c-94d895001d3e" (UID: "3fca0ddf-66ff-4161-8f0c-94d895001d3e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:44:38.820218 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:38.820178 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5mz7\" (UniqueName: \"kubernetes.io/projected/3fca0ddf-66ff-4161-8f0c-94d895001d3e-kube-api-access-v5mz7\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:44:38.820218 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:38.820215 4562 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3fca0ddf-66ff-4161-8f0c-94d895001d3e-provider-credentials\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:44:38.999558 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:44:38.999527 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17acd011_7a83_4ac2_9689_3a5568b7f94b.slice/crio-e6d270ba86d4a9f86b2379a538531df2d26c184ee1534432eca2df77746b9044 WatchSource:0}: Error finding container e6d270ba86d4a9f86b2379a538531df2d26c184ee1534432eca2df77746b9044: Status 404 returned error can't find the container with id e6d270ba86d4a9f86b2379a538531df2d26c184ee1534432eca2df77746b9044 Oct 30 09:44:39.000931 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:39.000912 4562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 09:44:39.001291 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:39.001270 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d66d22-f86b5c5b-7wsft"] Oct 30 09:44:39.284626 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:39.284586 4562 generic.go:356] "Generic (PLEG): container finished" podID="3fca0ddf-66ff-4161-8f0c-94d895001d3e" containerID="2c92b6811a41607fd3e012eaef1b64f79c2ebc5bd83248f06a83a2b24fafa77e" exitCode=0 Oct 30 09:44:39.285133 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:39.284640 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-5d5cmp6tj" Oct 30 09:44:39.285133 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:39.284676 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-5d5cmp6tj" event={"ID":"3fca0ddf-66ff-4161-8f0c-94d895001d3e","Type":"ContainerDied","Data":"2c92b6811a41607fd3e012eaef1b64f79c2ebc5bd83248f06a83a2b24fafa77e"} Oct 30 09:44:39.285133 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:39.284747 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-5d5cmp6tj" event={"ID":"3fca0ddf-66ff-4161-8f0c-94d895001d3e","Type":"ContainerDied","Data":"449bc2298c60f8547c6bd0ab1a8b7819bdb5e116a3d1e161ae920c9ee77c2264"} Oct 30 09:44:39.285133 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:39.284785 4562 scope.go:122] "RemoveContainer" containerID="2c92b6811a41607fd3e012eaef1b64f79c2ebc5bd83248f06a83a2b24fafa77e" Oct 30 09:44:39.286824 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:39.286781 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d66d22-f86b5c5b-7wsft" event={"ID":"17acd011-7a83-4ac2-9689-3a5568b7f94b","Type":"ContainerStarted","Data":"e6d270ba86d4a9f86b2379a538531df2d26c184ee1534432eca2df77746b9044"} Oct 30 09:44:39.297210 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:39.297191 4562 scope.go:122] "RemoveContainer" containerID="2c92b6811a41607fd3e012eaef1b64f79c2ebc5bd83248f06a83a2b24fafa77e" Oct 30 09:44:39.297557 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:44:39.297537 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c92b6811a41607fd3e012eaef1b64f79c2ebc5bd83248f06a83a2b24fafa77e\": container with ID starting with 2c92b6811a41607fd3e012eaef1b64f79c2ebc5bd83248f06a83a2b24fafa77e not found: ID does not exist" containerID="2c92b6811a41607fd3e012eaef1b64f79c2ebc5bd83248f06a83a2b24fafa77e" Oct 30 09:44:39.297625 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:39.297565 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c92b6811a41607fd3e012eaef1b64f79c2ebc5bd83248f06a83a2b24fafa77e"} err="failed to get container status \"2c92b6811a41607fd3e012eaef1b64f79c2ebc5bd83248f06a83a2b24fafa77e\": rpc error: code = NotFound desc = could not find container \"2c92b6811a41607fd3e012eaef1b64f79c2ebc5bd83248f06a83a2b24fafa77e\": container with ID starting with 2c92b6811a41607fd3e012eaef1b64f79c2ebc5bd83248f06a83a2b24fafa77e not found: ID does not exist" Oct 30 09:44:39.341934 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:39.341909 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-5d5cmp6tj"] Oct 30 09:44:39.357514 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:39.357487 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-5d5cmp6tj"] Oct 30 09:44:39.878619 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:39.878584 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fca0ddf-66ff-4161-8f0c-94d895001d3e" path="/var/lib/kubelet/pods/3fca0ddf-66ff-4161-8f0c-94d895001d3e/volumes" Oct 30 09:44:40.296559 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:40.296515 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d66d22-f86b5c5b-7wsft" event={"ID":"17acd011-7a83-4ac2-9689-3a5568b7f94b","Type":"ContainerStarted","Data":"673152ac959d04641228d6e3ac7decbec9deafda8237ca6e86d095729fc16e58"} Oct 30 09:44:40.296951 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:40.296694 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d66d22-f86b5c5b-7wsft" Oct 30 09:44:40.371402 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:40.371029 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d66d22-f86b5c5b-7wsft" podStartSLOduration=2.040256503 podStartE2EDuration="2.371007745s" totalImagesPullingTime="330.751242ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:44:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:44:40.356748724 +0000 UTC m=+3040.618309751" watchObservedRunningTime="2026-10-30 09:44:40.371007745 +0000 UTC m=+3040.632568735" Oct 30 09:44:46.309674 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:46.309639 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d66d22-f86b5c5b-7wsft" Oct 30 09:44:48.063924 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:48.063891 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5d5cqnr4z"] Oct 30 09:44:48.064892 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:48.064868 4562 cpu_manager.go:414] "containerMap: removing container" podUID="3fca0ddf-66ff-4161-8f0c-94d895001d3e" containerName="payload-processing" Oct 30 09:44:48.064892 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:48.064893 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fca0ddf-66ff-4161-8f0c-94d895001d3e" containerName="payload-processing" Oct 30 09:44:48.065142 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:48.065126 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fca0ddf-66ff-4161-8f0c-94d895001d3e" containerName="payload-processing" Oct 30 09:44:48.069027 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:48.069006 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5d5cqnr4z" Oct 30 09:44:48.111772 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:48.111735 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5d5cqnr4z"] Oct 30 09:44:48.199576 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:48.183962 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/370afa2f-9b22-491c-8574-985a715e84c2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-5d5cqnr4z\" (UID: \"370afa2f-9b22-491c-8574-985a715e84c2\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5d5cqnr4z" Oct 30 09:44:48.199576 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:48.184040 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/370afa2f-9b22-491c-8574-985a715e84c2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-5d5cqnr4z\" (UID: \"370afa2f-9b22-491c-8574-985a715e84c2\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5d5cqnr4z" Oct 30 09:44:48.199576 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:48.184536 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp94g\" (UniqueName: \"kubernetes.io/projected/370afa2f-9b22-491c-8574-985a715e84c2-kube-api-access-pp94g\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-5d5cqnr4z\" (UID: \"370afa2f-9b22-491c-8574-985a715e84c2\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5d5cqnr4z" Oct 30 09:44:48.199576 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:48.184596 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/370afa2f-9b22-491c-8574-985a715e84c2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-5d5cqnr4z\" (UID: \"370afa2f-9b22-491c-8574-985a715e84c2\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5d5cqnr4z" Oct 30 09:44:48.287274 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:48.287218 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp94g\" (UniqueName: \"kubernetes.io/projected/370afa2f-9b22-491c-8574-985a715e84c2-kube-api-access-pp94g\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-5d5cqnr4z\" (UID: \"370afa2f-9b22-491c-8574-985a715e84c2\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5d5cqnr4z" Oct 30 09:44:48.287513 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:48.287298 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/370afa2f-9b22-491c-8574-985a715e84c2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-5d5cqnr4z\" (UID: \"370afa2f-9b22-491c-8574-985a715e84c2\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5d5cqnr4z" Oct 30 09:44:48.297827 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:48.294442 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/370afa2f-9b22-491c-8574-985a715e84c2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-5d5cqnr4z\" (UID: \"370afa2f-9b22-491c-8574-985a715e84c2\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5d5cqnr4z" Oct 30 09:44:48.308633 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:48.308598 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/370afa2f-9b22-491c-8574-985a715e84c2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-5d5cqnr4z\" (UID: \"370afa2f-9b22-491c-8574-985a715e84c2\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5d5cqnr4z" Oct 30 09:44:48.308834 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:48.308676 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/370afa2f-9b22-491c-8574-985a715e84c2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-5d5cqnr4z\" (UID: \"370afa2f-9b22-491c-8574-985a715e84c2\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5d5cqnr4z" Oct 30 09:44:48.309696 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:48.309659 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/370afa2f-9b22-491c-8574-985a715e84c2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-5d5cqnr4z\" (UID: \"370afa2f-9b22-491c-8574-985a715e84c2\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5d5cqnr4z" Oct 30 09:44:48.316998 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:48.316929 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/370afa2f-9b22-491c-8574-985a715e84c2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-5d5cqnr4z\" (UID: \"370afa2f-9b22-491c-8574-985a715e84c2\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5d5cqnr4z" Oct 30 09:44:48.319458 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:48.319373 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp94g\" (UniqueName: \"kubernetes.io/projected/370afa2f-9b22-491c-8574-985a715e84c2-kube-api-access-pp94g\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-5d5cqnr4z\" (UID: \"370afa2f-9b22-491c-8574-985a715e84c2\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5d5cqnr4z" Oct 30 09:44:48.387129 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:48.387090 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5d5cqnr4z" Oct 30 09:44:48.708023 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:44:48.707983 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod370afa2f_9b22_491c_8574_985a715e84c2.slice/crio-99497a6857512676025efaff542167092501198d5cb7265b7800113ac0926cf1 WatchSource:0}: Error finding container 99497a6857512676025efaff542167092501198d5cb7265b7800113ac0926cf1: Status 404 returned error can't find the container with id 99497a6857512676025efaff542167092501198d5cb7265b7800113ac0926cf1 Oct 30 09:44:48.711622 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:48.711593 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5d5cqnr4z"] Oct 30 09:44:49.373251 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:49.373206 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5d5cqnr4z" event={"ID":"370afa2f-9b22-491c-8574-985a715e84c2","Type":"ContainerStarted","Data":"197ab4a0dd214b995e5eb8d1347a96d9e77224f92e2886689e05dac668f1e1d0"} Oct 30 09:44:49.373709 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:49.373280 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5d5cqnr4z" event={"ID":"370afa2f-9b22-491c-8574-985a715e84c2","Type":"ContainerStarted","Data":"99497a6857512676025efaff542167092501198d5cb7265b7800113ac0926cf1"} Oct 30 09:44:49.373709 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:49.373309 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5d5cqnr4z" Oct 30 09:44:49.414685 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:49.414632 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5d5cqnr4z" podStartSLOduration=1.414616656 podStartE2EDuration="1.414616656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:44:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:44:49.413212886 +0000 UTC m=+3049.674773875" watchObservedRunningTime="2026-10-30 09:44:49.414616656 +0000 UTC m=+3049.676177656" Oct 30 09:44:58.634431 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:44:58.634100 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:44:59.684077 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:59.684044 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5d5cqnr4z"] Oct 30 09:44:59.686722 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:59.686696 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5d5cqnr4z" Oct 30 09:44:59.696448 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:44:59.696414 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-6c845vbkc"] Oct 30 09:45:00.191415 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.188817 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d60c68-29889225-nlbtt"] Oct 30 09:45:00.216907 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.216876 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-88cbc7-29889225-2h7ds"] Oct 30 09:45:00.219892 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.217202 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d60c68-29889225-nlbtt" Oct 30 09:45:00.245732 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.245218 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr6zz\" (UniqueName: \"kubernetes.io/projected/9b26e444-ab39-4a88-b18f-34c4846cb17b-kube-api-access-lr6zz\") pod \"maas-api-key-cleanup-e2e-extmod-a-d60c68-29889225-nlbtt\" (UID: \"9b26e444-ab39-4a88-b18f-34c4846cb17b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d60c68-29889225-nlbtt" Oct 30 09:45:00.245732 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.245499 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b26e444-ab39-4a88-b18f-34c4846cb17b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-d60c68-29889225-nlbtt\" (UID: \"9b26e444-ab39-4a88-b18f-34c4846cb17b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d60c68-29889225-nlbtt" Oct 30 09:45:00.250315 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.250294 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-88cbc7-29889225-2h7ds" Oct 30 09:45:00.269178 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.269151 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889225-7v6l2"] Oct 30 09:45:00.294428 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.278597 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889225-7v6l2" Oct 30 09:45:00.294820 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.291319 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d60c68-29889225-nlbtt"] Oct 30 09:45:00.300064 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.297963 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-88cbc7-29889225-2h7ds"] Oct 30 09:45:00.301938 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.301693 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:45:00.305312 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.305289 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-035d7d-29889225-7tnvm"] Oct 30 09:45:00.314341 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.314116 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f58f32-29889225-s94vs"] Oct 30 09:45:00.316601 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.316433 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-035d7d-29889225-7tnvm" Oct 30 09:45:00.323449 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.321635 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889225-7v6l2"] Oct 30 09:45:00.323449 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.321679 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889225-rkfrx"] Oct 30 09:45:00.323449 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.321879 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f58f32-29889225-s94vs" Oct 30 09:45:00.335210 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.335184 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-035d7d-29889225-7tnvm"] Oct 30 09:45:00.335373 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.335327 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889225-rkfrx" Oct 30 09:45:00.341279 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.341044 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fb1a2a-29889225-qq76k"] Oct 30 09:45:00.348878 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.346610 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq8mc\" (UniqueName: \"kubernetes.io/projected/802f9a81-779c-49ff-8e04-283b72c0a908-kube-api-access-fq8mc\") pod \"maas-api-key-cleanup-e2e-worker-w1-c5447d-29889225-rkfrx\" (UID: \"802f9a81-779c-49ff-8e04-283b72c0a908\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889225-rkfrx" Oct 30 09:45:00.348878 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.346673 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b26e444-ab39-4a88-b18f-34c4846cb17b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-d60c68-29889225-nlbtt\" (UID: \"9b26e444-ab39-4a88-b18f-34c4846cb17b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d60c68-29889225-nlbtt" Oct 30 09:45:00.348878 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.346702 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f25074f0-f8c9-4a6f-b899-4ae6a7ff4311-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-f58f32-29889225-s94vs\" (UID: \"f25074f0-f8c9-4a6f-b899-4ae6a7ff4311\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f58f32-29889225-s94vs" Oct 30 09:45:00.348878 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.346754 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt5ht\" (UniqueName: \"kubernetes.io/projected/db53ebe2-e3dd-4580-b79c-830feff939a1-kube-api-access-pt5ht\") pod \"maas-api-key-cleanup-e2e-extmod-b-035d7d-29889225-7tnvm\" (UID: \"db53ebe2-e3dd-4580-b79c-830feff939a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-035d7d-29889225-7tnvm" Oct 30 09:45:00.348878 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.346793 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/802f9a81-779c-49ff-8e04-283b72c0a908-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-c5447d-29889225-rkfrx\" (UID: \"802f9a81-779c-49ff-8e04-283b72c0a908\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889225-rkfrx" Oct 30 09:45:00.348878 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.346872 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr6zz\" (UniqueName: \"kubernetes.io/projected/9b26e444-ab39-4a88-b18f-34c4846cb17b-kube-api-access-lr6zz\") pod \"maas-api-key-cleanup-e2e-extmod-a-d60c68-29889225-nlbtt\" (UID: \"9b26e444-ab39-4a88-b18f-34c4846cb17b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d60c68-29889225-nlbtt" Oct 30 09:45:00.348878 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.346901 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqrhw\" (UniqueName: \"kubernetes.io/projected/6da36b71-6952-4a6c-a029-ebefe69addb2-kube-api-access-hqrhw\") pod \"maas-api-key-cleanup-e2e-extmod-a-88cbc7-29889225-2h7ds\" (UID: \"6da36b71-6952-4a6c-a029-ebefe69addb2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-88cbc7-29889225-2h7ds" Oct 30 09:45:00.348878 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.346932 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6da36b71-6952-4a6c-a029-ebefe69addb2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-88cbc7-29889225-2h7ds\" (UID: \"6da36b71-6952-4a6c-a029-ebefe69addb2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-88cbc7-29889225-2h7ds" Oct 30 09:45:00.348878 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.347038 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db53ebe2-e3dd-4580-b79c-830feff939a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-035d7d-29889225-7tnvm\" (UID: \"db53ebe2-e3dd-4580-b79c-830feff939a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-035d7d-29889225-7tnvm" Oct 30 09:45:00.348878 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.347090 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/362d480a-6622-40a3-ab4f-8a85c54b3f06-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889225-7v6l2\" (UID: \"362d480a-6622-40a3-ab4f-8a85c54b3f06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889225-7v6l2" Oct 30 09:45:00.348878 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.347140 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqv6c\" (UniqueName: \"kubernetes.io/projected/362d480a-6622-40a3-ab4f-8a85c54b3f06-kube-api-access-gqv6c\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889225-7v6l2\" (UID: \"362d480a-6622-40a3-ab4f-8a85c54b3f06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889225-7v6l2" Oct 30 09:45:00.348878 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.347184 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgl48\" (UniqueName: \"kubernetes.io/projected/f25074f0-f8c9-4a6f-b899-4ae6a7ff4311-kube-api-access-mgl48\") pod \"maas-api-key-cleanup-e2e-extmod-b-f58f32-29889225-s94vs\" (UID: \"f25074f0-f8c9-4a6f-b899-4ae6a7ff4311\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f58f32-29889225-s94vs" Oct 30 09:45:00.348878 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.348034 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b26e444-ab39-4a88-b18f-34c4846cb17b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-d60c68-29889225-nlbtt\" (UID: \"9b26e444-ab39-4a88-b18f-34c4846cb17b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d60c68-29889225-nlbtt" Oct 30 09:45:00.349560 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.349541 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fb1a2a-29889225-qq76k" Oct 30 09:45:00.352973 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.352957 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f58f32-29889225-s94vs"] Oct 30 09:45:00.359392 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.359281 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889225-rkfrx"] Oct 30 09:45:00.375648 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.372254 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr6zz\" (UniqueName: \"kubernetes.io/projected/9b26e444-ab39-4a88-b18f-34c4846cb17b-kube-api-access-lr6zz\") pod \"maas-api-key-cleanup-e2e-extmod-a-d60c68-29889225-nlbtt\" (UID: \"9b26e444-ab39-4a88-b18f-34c4846cb17b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d60c68-29889225-nlbtt" Oct 30 09:45:00.399404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.398540 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fb1a2a-29889225-qq76k"] Oct 30 09:45:00.460055 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.459516 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgl48\" (UniqueName: \"kubernetes.io/projected/f25074f0-f8c9-4a6f-b899-4ae6a7ff4311-kube-api-access-mgl48\") pod \"maas-api-key-cleanup-e2e-extmod-b-f58f32-29889225-s94vs\" (UID: \"f25074f0-f8c9-4a6f-b899-4ae6a7ff4311\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f58f32-29889225-s94vs" Oct 30 09:45:00.460055 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.459647 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq8mc\" (UniqueName: \"kubernetes.io/projected/802f9a81-779c-49ff-8e04-283b72c0a908-kube-api-access-fq8mc\") pod \"maas-api-key-cleanup-e2e-worker-w1-c5447d-29889225-rkfrx\" (UID: \"802f9a81-779c-49ff-8e04-283b72c0a908\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889225-rkfrx" Oct 30 09:45:00.460055 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.459756 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f25074f0-f8c9-4a6f-b899-4ae6a7ff4311-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-f58f32-29889225-s94vs\" (UID: \"f25074f0-f8c9-4a6f-b899-4ae6a7ff4311\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f58f32-29889225-s94vs" Oct 30 09:45:00.460055 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.459813 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt5ht\" (UniqueName: \"kubernetes.io/projected/db53ebe2-e3dd-4580-b79c-830feff939a1-kube-api-access-pt5ht\") pod \"maas-api-key-cleanup-e2e-extmod-b-035d7d-29889225-7tnvm\" (UID: \"db53ebe2-e3dd-4580-b79c-830feff939a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-035d7d-29889225-7tnvm" Oct 30 09:45:00.460055 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.459862 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/802f9a81-779c-49ff-8e04-283b72c0a908-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-c5447d-29889225-rkfrx\" (UID: \"802f9a81-779c-49ff-8e04-283b72c0a908\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889225-rkfrx" Oct 30 09:45:00.460055 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.459996 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqrhw\" (UniqueName: \"kubernetes.io/projected/6da36b71-6952-4a6c-a029-ebefe69addb2-kube-api-access-hqrhw\") pod \"maas-api-key-cleanup-e2e-extmod-a-88cbc7-29889225-2h7ds\" (UID: \"6da36b71-6952-4a6c-a029-ebefe69addb2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-88cbc7-29889225-2h7ds" Oct 30 09:45:00.461778 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.461750 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/802f9a81-779c-49ff-8e04-283b72c0a908-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-c5447d-29889225-rkfrx\" (UID: \"802f9a81-779c-49ff-8e04-283b72c0a908\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889225-rkfrx" Oct 30 09:45:00.473008 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.463736 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6da36b71-6952-4a6c-a029-ebefe69addb2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-88cbc7-29889225-2h7ds\" (UID: \"6da36b71-6952-4a6c-a029-ebefe69addb2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-88cbc7-29889225-2h7ds" Oct 30 09:45:00.475616 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.464889 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6da36b71-6952-4a6c-a029-ebefe69addb2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-88cbc7-29889225-2h7ds\" (UID: \"6da36b71-6952-4a6c-a029-ebefe69addb2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-88cbc7-29889225-2h7ds" Oct 30 09:45:00.475806 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.466293 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f25074f0-f8c9-4a6f-b899-4ae6a7ff4311-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-f58f32-29889225-s94vs\" (UID: \"f25074f0-f8c9-4a6f-b899-4ae6a7ff4311\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f58f32-29889225-s94vs" Oct 30 09:45:00.475917 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.474654 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db53ebe2-e3dd-4580-b79c-830feff939a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-035d7d-29889225-7tnvm\" (UID: \"db53ebe2-e3dd-4580-b79c-830feff939a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-035d7d-29889225-7tnvm" Oct 30 09:45:00.476091 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.476073 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/362d480a-6622-40a3-ab4f-8a85c54b3f06-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889225-7v6l2\" (UID: \"362d480a-6622-40a3-ab4f-8a85c54b3f06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889225-7v6l2" Oct 30 09:45:00.476258 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.476242 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqv6c\" (UniqueName: \"kubernetes.io/projected/362d480a-6622-40a3-ab4f-8a85c54b3f06-kube-api-access-gqv6c\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889225-7v6l2\" (UID: \"362d480a-6622-40a3-ab4f-8a85c54b3f06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889225-7v6l2" Oct 30 09:45:00.476540 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.475579 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db53ebe2-e3dd-4580-b79c-830feff939a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-035d7d-29889225-7tnvm\" (UID: \"db53ebe2-e3dd-4580-b79c-830feff939a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-035d7d-29889225-7tnvm" Oct 30 09:45:00.477473 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.477375 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/362d480a-6622-40a3-ab4f-8a85c54b3f06-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889225-7v6l2\" (UID: \"362d480a-6622-40a3-ab4f-8a85c54b3f06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889225-7v6l2" Oct 30 09:45:00.488660 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.488624 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqrhw\" (UniqueName: \"kubernetes.io/projected/6da36b71-6952-4a6c-a029-ebefe69addb2-kube-api-access-hqrhw\") pod \"maas-api-key-cleanup-e2e-extmod-a-88cbc7-29889225-2h7ds\" (UID: \"6da36b71-6952-4a6c-a029-ebefe69addb2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-88cbc7-29889225-2h7ds" Oct 30 09:45:00.491749 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.491680 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq8mc\" (UniqueName: \"kubernetes.io/projected/802f9a81-779c-49ff-8e04-283b72c0a908-kube-api-access-fq8mc\") pod \"maas-api-key-cleanup-e2e-worker-w1-c5447d-29889225-rkfrx\" (UID: \"802f9a81-779c-49ff-8e04-283b72c0a908\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889225-rkfrx" Oct 30 09:45:00.492567 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.492541 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgl48\" (UniqueName: \"kubernetes.io/projected/f25074f0-f8c9-4a6f-b899-4ae6a7ff4311-kube-api-access-mgl48\") pod \"maas-api-key-cleanup-e2e-extmod-b-f58f32-29889225-s94vs\" (UID: \"f25074f0-f8c9-4a6f-b899-4ae6a7ff4311\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f58f32-29889225-s94vs" Oct 30 09:45:00.493910 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.493879 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt5ht\" (UniqueName: \"kubernetes.io/projected/db53ebe2-e3dd-4580-b79c-830feff939a1-kube-api-access-pt5ht\") pod \"maas-api-key-cleanup-e2e-extmod-b-035d7d-29889225-7tnvm\" (UID: \"db53ebe2-e3dd-4580-b79c-830feff939a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-035d7d-29889225-7tnvm" Oct 30 09:45:00.497340 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.497311 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqv6c\" (UniqueName: \"kubernetes.io/projected/362d480a-6622-40a3-ab4f-8a85c54b3f06-kube-api-access-gqv6c\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889225-7v6l2\" (UID: \"362d480a-6622-40a3-ab4f-8a85c54b3f06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889225-7v6l2" Oct 30 09:45:00.550791 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.550761 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d60c68-29889225-nlbtt" Oct 30 09:45:00.583350 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.582902 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-88cbc7-29889225-2h7ds" Oct 30 09:45:00.590399 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.588766 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d8hq\" (UniqueName: \"kubernetes.io/projected/83484f50-c325-4c2c-beac-16f69e38eb66-kube-api-access-9d8hq\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-fb1a2a-29889225-qq76k\" (UID: \"83484f50-c325-4c2c-beac-16f69e38eb66\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fb1a2a-29889225-qq76k" Oct 30 09:45:00.590399 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.588825 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83484f50-c325-4c2c-beac-16f69e38eb66-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-fb1a2a-29889225-qq76k\" (UID: \"83484f50-c325-4c2c-beac-16f69e38eb66\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fb1a2a-29889225-qq76k" Oct 30 09:45:00.638406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.637689 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889225-7v6l2" Oct 30 09:45:00.655533 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.653869 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-035d7d-29889225-7tnvm" Oct 30 09:45:00.675761 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.675716 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f58f32-29889225-s94vs" Oct 30 09:45:00.692449 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.690598 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d8hq\" (UniqueName: \"kubernetes.io/projected/83484f50-c325-4c2c-beac-16f69e38eb66-kube-api-access-9d8hq\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-fb1a2a-29889225-qq76k\" (UID: \"83484f50-c325-4c2c-beac-16f69e38eb66\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fb1a2a-29889225-qq76k" Oct 30 09:45:00.692449 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.690658 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83484f50-c325-4c2c-beac-16f69e38eb66-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-fb1a2a-29889225-qq76k\" (UID: \"83484f50-c325-4c2c-beac-16f69e38eb66\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fb1a2a-29889225-qq76k" Oct 30 09:45:00.692449 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.691629 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83484f50-c325-4c2c-beac-16f69e38eb66-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-fb1a2a-29889225-qq76k\" (UID: \"83484f50-c325-4c2c-beac-16f69e38eb66\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fb1a2a-29889225-qq76k" Oct 30 09:45:00.714662 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.711973 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d8hq\" (UniqueName: \"kubernetes.io/projected/83484f50-c325-4c2c-beac-16f69e38eb66-kube-api-access-9d8hq\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-fb1a2a-29889225-qq76k\" (UID: \"83484f50-c325-4c2c-beac-16f69e38eb66\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fb1a2a-29889225-qq76k" Oct 30 09:45:00.766752 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.762986 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889225-rkfrx" Oct 30 09:45:00.794038 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.789296 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fb1a2a-29889225-qq76k" Oct 30 09:45:00.942938 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:00.942907 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d60c68-29889225-nlbtt"] Oct 30 09:45:00.943950 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:45:00.943920 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b26e444_ab39_4a88_b18f_34c4846cb17b.slice/crio-7d3fd435f2d3d94d43d64a64838da06f642d1859eae63d5edc0fcc2814e1c099 WatchSource:0}: Error finding container 7d3fd435f2d3d94d43d64a64838da06f642d1859eae63d5edc0fcc2814e1c099: Status 404 returned error can't find the container with id 7d3fd435f2d3d94d43d64a64838da06f642d1859eae63d5edc0fcc2814e1c099 Oct 30 09:45:01.156272 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:45:01.156237 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6da36b71_6952_4a6c_a029_ebefe69addb2.slice/crio-c8211c866d956744a979295dcb19497fe462b5c9595fc12d1480bb44e2be069e WatchSource:0}: Error finding container c8211c866d956744a979295dcb19497fe462b5c9595fc12d1480bb44e2be069e: Status 404 returned error can't find the container with id c8211c866d956744a979295dcb19497fe462b5c9595fc12d1480bb44e2be069e Oct 30 09:45:01.157270 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:45:01.157242 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83484f50_c325_4c2c_beac_16f69e38eb66.slice/crio-9e3def99b6fc66bf35d6e83b10a3a3e0edec3151a51bc11d88a80b42e410abe8 WatchSource:0}: Error finding container 9e3def99b6fc66bf35d6e83b10a3a3e0edec3151a51bc11d88a80b42e410abe8: Status 404 returned error can't find the container with id 9e3def99b6fc66bf35d6e83b10a3a3e0edec3151a51bc11d88a80b42e410abe8 Oct 30 09:45:01.158316 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:01.158297 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-88cbc7-29889225-2h7ds"] Oct 30 09:45:01.163224 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:01.163202 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fb1a2a-29889225-qq76k"] Oct 30 09:45:01.180192 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:01.180162 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889225-rkfrx"] Oct 30 09:45:01.219836 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:45:01.219804 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb53ebe2_e3dd_4580_b79c_830feff939a1.slice/crio-3b11c79805a72b296d63b25772d97a8de4a333d1eaac6e7823d6cefa99e8e66b WatchSource:0}: Error finding container 3b11c79805a72b296d63b25772d97a8de4a333d1eaac6e7823d6cefa99e8e66b: Status 404 returned error can't find the container with id 3b11c79805a72b296d63b25772d97a8de4a333d1eaac6e7823d6cefa99e8e66b Oct 30 09:45:01.221861 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:01.221839 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-035d7d-29889225-7tnvm"] Oct 30 09:45:01.243522 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:01.243493 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889225-7v6l2"] Oct 30 09:45:01.244518 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:45:01.244489 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod362d480a_6622_40a3_ab4f_8a85c54b3f06.slice/crio-a492198831eaaa1e8a0b00e6e4c74acef1213192bb79672c59ad89df4d926c4f WatchSource:0}: Error finding container a492198831eaaa1e8a0b00e6e4c74acef1213192bb79672c59ad89df4d926c4f: Status 404 returned error can't find the container with id a492198831eaaa1e8a0b00e6e4c74acef1213192bb79672c59ad89df4d926c4f Oct 30 09:45:01.294836 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:01.294704 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f58f32-29889225-s94vs"] Oct 30 09:45:01.346588 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:45:01.346552 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf25074f0_f8c9_4a6f_b899_4ae6a7ff4311.slice/crio-aa707bc94e7d16194f66cfea34f459b412f617912f426c904822ee26b3478e9e WatchSource:0}: Error finding container aa707bc94e7d16194f66cfea34f459b412f617912f426c904822ee26b3478e9e: Status 404 returned error can't find the container with id aa707bc94e7d16194f66cfea34f459b412f617912f426c904822ee26b3478e9e Oct 30 09:45:01.496615 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:01.496519 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889225-7v6l2" event={"ID":"362d480a-6622-40a3-ab4f-8a85c54b3f06","Type":"ContainerStarted","Data":"a492198831eaaa1e8a0b00e6e4c74acef1213192bb79672c59ad89df4d926c4f"} Oct 30 09:45:01.497850 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:01.497825 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889225-rkfrx" event={"ID":"802f9a81-779c-49ff-8e04-283b72c0a908","Type":"ContainerStarted","Data":"57c9edcaa5d59473678233f2629dc8163854cef6f807a4bae715245e0fc5829d"} Oct 30 09:45:01.499084 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:01.499006 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-88cbc7-29889225-2h7ds" event={"ID":"6da36b71-6952-4a6c-a029-ebefe69addb2","Type":"ContainerStarted","Data":"c8211c866d956744a979295dcb19497fe462b5c9595fc12d1480bb44e2be069e"} Oct 30 09:45:01.500604 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:01.500577 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-035d7d-29889225-7tnvm" event={"ID":"db53ebe2-e3dd-4580-b79c-830feff939a1","Type":"ContainerStarted","Data":"3b11c79805a72b296d63b25772d97a8de4a333d1eaac6e7823d6cefa99e8e66b"} Oct 30 09:45:01.501829 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:01.501805 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f58f32-29889225-s94vs" event={"ID":"f25074f0-f8c9-4a6f-b899-4ae6a7ff4311","Type":"ContainerStarted","Data":"aa707bc94e7d16194f66cfea34f459b412f617912f426c904822ee26b3478e9e"} Oct 30 09:45:01.503180 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:01.503154 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fb1a2a-29889225-qq76k" event={"ID":"83484f50-c325-4c2c-beac-16f69e38eb66","Type":"ContainerStarted","Data":"9e3def99b6fc66bf35d6e83b10a3a3e0edec3151a51bc11d88a80b42e410abe8"} Oct 30 09:45:01.504775 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:01.504753 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d60c68-29889225-nlbtt" event={"ID":"9b26e444-ab39-4a88-b18f-34c4846cb17b","Type":"ContainerStarted","Data":"84a00a342e0eeb7d6403f05e2dda408b57537f882dcca620814a6cf3fad50699"} Oct 30 09:45:01.504775 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:01.504787 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d60c68-29889225-nlbtt" event={"ID":"9b26e444-ab39-4a88-b18f-34c4846cb17b","Type":"ContainerStarted","Data":"7d3fd435f2d3d94d43d64a64838da06f642d1859eae63d5edc0fcc2814e1c099"} Oct 30 09:45:01.549851 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:01.549800 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d60c68-29889225-nlbtt" podStartSLOduration=1.54978601 podStartE2EDuration="1.54978601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:45:01.545160354 +0000 UTC m=+3061.806721353" watchObservedRunningTime="2026-10-30 09:45:01.54978601 +0000 UTC m=+3061.811346999" Oct 30 09:45:02.514994 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:02.514956 4562 generic.go:356] "Generic (PLEG): container finished" podID="362d480a-6622-40a3-ab4f-8a85c54b3f06" containerID="528727cc825a78e2c4d35f0aff0c7c03081723b6b733c3c20fa75d40202c13aa" exitCode=60 Oct 30 09:45:02.515436 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:02.515040 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889225-7v6l2" event={"ID":"362d480a-6622-40a3-ab4f-8a85c54b3f06","Type":"ContainerDied","Data":"528727cc825a78e2c4d35f0aff0c7c03081723b6b733c3c20fa75d40202c13aa"} Oct 30 09:45:02.515436 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:02.515391 4562 scope.go:122] "RemoveContainer" containerID="528727cc825a78e2c4d35f0aff0c7c03081723b6b733c3c20fa75d40202c13aa" Oct 30 09:45:02.516834 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:02.516809 4562 generic.go:356] "Generic (PLEG): container finished" podID="802f9a81-779c-49ff-8e04-283b72c0a908" containerID="eb11ec0c09974333dbd8c2bfa581922f9f91583c55e7e011c5b45196b5340a59" exitCode=60 Oct 30 09:45:02.516933 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:02.516895 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889225-rkfrx" event={"ID":"802f9a81-779c-49ff-8e04-283b72c0a908","Type":"ContainerDied","Data":"eb11ec0c09974333dbd8c2bfa581922f9f91583c55e7e011c5b45196b5340a59"} Oct 30 09:45:02.517157 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:02.517138 4562 scope.go:122] "RemoveContainer" containerID="eb11ec0c09974333dbd8c2bfa581922f9f91583c55e7e011c5b45196b5340a59" Oct 30 09:45:02.518986 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:02.518911 4562 generic.go:356] "Generic (PLEG): container finished" podID="6da36b71-6952-4a6c-a029-ebefe69addb2" containerID="418f86f91a5ca34c1e694e2ab37ed65f31ba0d4c3b643464d26a9333e6e6af4d" exitCode=60 Oct 30 09:45:02.519065 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:02.519017 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-88cbc7-29889225-2h7ds" event={"ID":"6da36b71-6952-4a6c-a029-ebefe69addb2","Type":"ContainerDied","Data":"418f86f91a5ca34c1e694e2ab37ed65f31ba0d4c3b643464d26a9333e6e6af4d"} Oct 30 09:45:02.519252 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:02.519236 4562 scope.go:122] "RemoveContainer" containerID="418f86f91a5ca34c1e694e2ab37ed65f31ba0d4c3b643464d26a9333e6e6af4d" Oct 30 09:45:02.522096 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:02.522073 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-035d7d-29889225-7tnvm" event={"ID":"db53ebe2-e3dd-4580-b79c-830feff939a1","Type":"ContainerStarted","Data":"ca159e931d4ea1f6525eb35c83817901e0ee957a0c6b153592b5b4db184fcc7c"} Oct 30 09:45:02.523776 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:02.523754 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f58f32-29889225-s94vs" event={"ID":"f25074f0-f8c9-4a6f-b899-4ae6a7ff4311","Type":"ContainerStarted","Data":"e81a3cc6d94378fb4f3c8f29fdacee4847865923e5b6a84c3ee428b16d35ab1e"} Oct 30 09:45:02.525311 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:02.525290 4562 generic.go:356] "Generic (PLEG): container finished" podID="83484f50-c325-4c2c-beac-16f69e38eb66" containerID="2bf43f1f4c23477249f72de82d4fea8ff3b178a7c0c05badc7a03b4f88701f37" exitCode=60 Oct 30 09:45:02.525453 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:02.525428 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fb1a2a-29889225-qq76k" event={"ID":"83484f50-c325-4c2c-beac-16f69e38eb66","Type":"ContainerDied","Data":"2bf43f1f4c23477249f72de82d4fea8ff3b178a7c0c05badc7a03b4f88701f37"} Oct 30 09:45:02.525587 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:02.525571 4562 scope.go:122] "RemoveContainer" containerID="2bf43f1f4c23477249f72de82d4fea8ff3b178a7c0c05badc7a03b4f88701f37" Oct 30 09:45:02.527000 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:02.526975 4562 generic.go:356] "Generic (PLEG): container finished" podID="9b26e444-ab39-4a88-b18f-34c4846cb17b" containerID="84a00a342e0eeb7d6403f05e2dda408b57537f882dcca620814a6cf3fad50699" exitCode=60 Oct 30 09:45:02.527109 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:02.527011 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d60c68-29889225-nlbtt" event={"ID":"9b26e444-ab39-4a88-b18f-34c4846cb17b","Type":"ContainerDied","Data":"84a00a342e0eeb7d6403f05e2dda408b57537f882dcca620814a6cf3fad50699"} Oct 30 09:45:02.527687 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:02.527668 4562 scope.go:122] "RemoveContainer" containerID="84a00a342e0eeb7d6403f05e2dda408b57537f882dcca620814a6cf3fad50699" Oct 30 09:45:02.675867 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:02.675819 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f58f32-29889225-s94vs" podStartSLOduration=2.675806233 podStartE2EDuration="2.675806233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:45:02.674555453 +0000 UTC m=+3062.936116440" watchObservedRunningTime="2026-10-30 09:45:02.675806233 +0000 UTC m=+3062.937367219" Oct 30 09:45:02.792181 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:02.792118 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-035d7d-29889225-7tnvm" podStartSLOduration=2.792103079 podStartE2EDuration="2.792103079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:45:02.791038056 +0000 UTC m=+3063.052599045" watchObservedRunningTime="2026-10-30 09:45:02.792103079 +0000 UTC m=+3063.053664061" Oct 30 09:45:03.537658 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:03.537621 4562 generic.go:356] "Generic (PLEG): container finished" podID="f25074f0-f8c9-4a6f-b899-4ae6a7ff4311" containerID="e81a3cc6d94378fb4f3c8f29fdacee4847865923e5b6a84c3ee428b16d35ab1e" exitCode=60 Oct 30 09:45:03.538134 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:03.537710 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f58f32-29889225-s94vs" event={"ID":"f25074f0-f8c9-4a6f-b899-4ae6a7ff4311","Type":"ContainerDied","Data":"e81a3cc6d94378fb4f3c8f29fdacee4847865923e5b6a84c3ee428b16d35ab1e"} Oct 30 09:45:03.538134 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:03.538028 4562 scope.go:122] "RemoveContainer" containerID="e81a3cc6d94378fb4f3c8f29fdacee4847865923e5b6a84c3ee428b16d35ab1e" Oct 30 09:45:03.546757 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:03.546729 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fb1a2a-29889225-qq76k" event={"ID":"83484f50-c325-4c2c-beac-16f69e38eb66","Type":"ContainerStarted","Data":"67683240ccc55d0f8cd0edb7438964f79cc043ce86fe759a9103b19fd6f0f62d"} Oct 30 09:45:03.548714 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:03.548687 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d60c68-29889225-nlbtt" event={"ID":"9b26e444-ab39-4a88-b18f-34c4846cb17b","Type":"ContainerStarted","Data":"b71577ba908a980479c3cff6daff5b1f46667f2c0fd735b4175d2f69c33d0bca"} Oct 30 09:45:03.550845 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:03.550820 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889225-7v6l2" event={"ID":"362d480a-6622-40a3-ab4f-8a85c54b3f06","Type":"ContainerStarted","Data":"f8ff722396fada580ae44b23fc7dc013b39da7b2b5fd97050ec2f160fd55c9ff"} Oct 30 09:45:03.552894 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:03.552868 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889225-rkfrx" event={"ID":"802f9a81-779c-49ff-8e04-283b72c0a908","Type":"ContainerStarted","Data":"55b608d455aac6581ab6489036004bb54948a28232628d83fed52bb9a2076deb"} Oct 30 09:45:03.554820 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:03.554785 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-88cbc7-29889225-2h7ds" event={"ID":"6da36b71-6952-4a6c-a029-ebefe69addb2","Type":"ContainerStarted","Data":"cf548f78240be07a48968824706b3428618a936ea2ac03dc91ae73017dc54309"} Oct 30 09:45:03.556434 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:03.556405 4562 generic.go:356] "Generic (PLEG): container finished" podID="db53ebe2-e3dd-4580-b79c-830feff939a1" containerID="ca159e931d4ea1f6525eb35c83817901e0ee957a0c6b153592b5b4db184fcc7c" exitCode=60 Oct 30 09:45:03.556522 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:03.556447 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-035d7d-29889225-7tnvm" event={"ID":"db53ebe2-e3dd-4580-b79c-830feff939a1","Type":"ContainerDied","Data":"ca159e931d4ea1f6525eb35c83817901e0ee957a0c6b153592b5b4db184fcc7c"} Oct 30 09:45:03.556688 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:03.556674 4562 scope.go:122] "RemoveContainer" containerID="ca159e931d4ea1f6525eb35c83817901e0ee957a0c6b153592b5b4db184fcc7c" Oct 30 09:45:03.663179 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:03.663126 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-88cbc7-29889225-2h7ds" podStartSLOduration=3.663111522 podStartE2EDuration="3.663111522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:45:03.61954212 +0000 UTC m=+3063.881103105" watchObservedRunningTime="2026-10-30 09:45:03.663111522 +0000 UTC m=+3063.924672565" Oct 30 09:45:03.664829 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:03.664796 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889225-7v6l2" podStartSLOduration=3.664785647 podStartE2EDuration="3.664785647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:45:03.660536954 +0000 UTC m=+3063.922097942" watchObservedRunningTime="2026-10-30 09:45:03.664785647 +0000 UTC m=+3063.926346635" Oct 30 09:45:03.701089 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:03.701033 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889225-rkfrx" podStartSLOduration=3.701015686 podStartE2EDuration="3.701015686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:45:03.698947389 +0000 UTC m=+3063.960508379" watchObservedRunningTime="2026-10-30 09:45:03.701015686 +0000 UTC m=+3063.962576674" Oct 30 09:45:03.738506 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:03.738450 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fb1a2a-29889225-qq76k" podStartSLOduration=3.738437036 podStartE2EDuration="3.738437036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:45:03.737699537 +0000 UTC m=+3063.999260526" watchObservedRunningTime="2026-10-30 09:45:03.738437036 +0000 UTC m=+3063.999998023" Oct 30 09:45:04.565959 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:04.565928 4562 generic.go:356] "Generic (PLEG): container finished" podID="db53ebe2-e3dd-4580-b79c-830feff939a1" containerID="0074c834a1983f3f695585ab9fd14bafef306bad489c9676bd46d76fa145097e" exitCode=60 Oct 30 09:45:04.566334 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:04.566011 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-035d7d-29889225-7tnvm" event={"ID":"db53ebe2-e3dd-4580-b79c-830feff939a1","Type":"ContainerDied","Data":"0074c834a1983f3f695585ab9fd14bafef306bad489c9676bd46d76fa145097e"} Oct 30 09:45:04.566334 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:04.566078 4562 scope.go:122] "RemoveContainer" containerID="ca159e931d4ea1f6525eb35c83817901e0ee957a0c6b153592b5b4db184fcc7c" Oct 30 09:45:04.566485 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:04.566367 4562 scope.go:122] "RemoveContainer" containerID="0074c834a1983f3f695585ab9fd14bafef306bad489c9676bd46d76fa145097e" Oct 30 09:45:04.566761 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:45:04.566735 4562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-035d7d-29889225-7tnvm_odh-ai-gateway-infra(db53ebe2-e3dd-4580-b79c-830feff939a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-035d7d-29889225-7tnvm" podUID="db53ebe2-e3dd-4580-b79c-830feff939a1" Oct 30 09:45:04.568047 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:04.568026 4562 generic.go:356] "Generic (PLEG): container finished" podID="f25074f0-f8c9-4a6f-b899-4ae6a7ff4311" containerID="2ba54bfb111bad3b18dabf7a84638fd1db2ff9a60391b16f95435f3dbf1f848f" exitCode=60 Oct 30 09:45:04.568129 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:04.568086 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f58f32-29889225-s94vs" event={"ID":"f25074f0-f8c9-4a6f-b899-4ae6a7ff4311","Type":"ContainerDied","Data":"2ba54bfb111bad3b18dabf7a84638fd1db2ff9a60391b16f95435f3dbf1f848f"} Oct 30 09:45:04.568359 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:04.568345 4562 scope.go:122] "RemoveContainer" containerID="2ba54bfb111bad3b18dabf7a84638fd1db2ff9a60391b16f95435f3dbf1f848f" Oct 30 09:45:04.568687 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:45:04.568664 4562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-f58f32-29889225-s94vs_odh-ai-gateway-infra(f25074f0-f8c9-4a6f-b899-4ae6a7ff4311)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f58f32-29889225-s94vs" podUID="f25074f0-f8c9-4a6f-b899-4ae6a7ff4311" Oct 30 09:45:04.569938 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:04.569917 4562 generic.go:356] "Generic (PLEG): container finished" podID="83484f50-c325-4c2c-beac-16f69e38eb66" containerID="67683240ccc55d0f8cd0edb7438964f79cc043ce86fe759a9103b19fd6f0f62d" exitCode=60 Oct 30 09:45:04.570020 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:04.570003 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fb1a2a-29889225-qq76k" event={"ID":"83484f50-c325-4c2c-beac-16f69e38eb66","Type":"ContainerDied","Data":"67683240ccc55d0f8cd0edb7438964f79cc043ce86fe759a9103b19fd6f0f62d"} Oct 30 09:45:04.570190 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:04.570172 4562 scope.go:122] "RemoveContainer" containerID="67683240ccc55d0f8cd0edb7438964f79cc043ce86fe759a9103b19fd6f0f62d" Oct 30 09:45:04.570503 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:45:04.570484 4562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-fb1a2a-29889225-qq76k_odh-ai-gateway-infra(83484f50-c325-4c2c-beac-16f69e38eb66)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fb1a2a-29889225-qq76k" podUID="83484f50-c325-4c2c-beac-16f69e38eb66" Oct 30 09:45:04.571991 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:04.571972 4562 generic.go:356] "Generic (PLEG): container finished" podID="9b26e444-ab39-4a88-b18f-34c4846cb17b" containerID="b71577ba908a980479c3cff6daff5b1f46667f2c0fd735b4175d2f69c33d0bca" exitCode=60 Oct 30 09:45:04.572082 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:04.572050 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d60c68-29889225-nlbtt" event={"ID":"9b26e444-ab39-4a88-b18f-34c4846cb17b","Type":"ContainerDied","Data":"b71577ba908a980479c3cff6daff5b1f46667f2c0fd735b4175d2f69c33d0bca"} Oct 30 09:45:04.572261 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:04.572241 4562 scope.go:122] "RemoveContainer" containerID="b71577ba908a980479c3cff6daff5b1f46667f2c0fd735b4175d2f69c33d0bca" Oct 30 09:45:04.572562 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:45:04.572539 4562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-d60c68-29889225-nlbtt_odh-ai-gateway-infra(9b26e444-ab39-4a88-b18f-34c4846cb17b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d60c68-29889225-nlbtt" podUID="9b26e444-ab39-4a88-b18f-34c4846cb17b" Oct 30 09:45:04.573996 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:04.573976 4562 generic.go:356] "Generic (PLEG): container finished" podID="362d480a-6622-40a3-ab4f-8a85c54b3f06" containerID="f8ff722396fada580ae44b23fc7dc013b39da7b2b5fd97050ec2f160fd55c9ff" exitCode=60 Oct 30 09:45:04.574103 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:04.574067 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889225-7v6l2" event={"ID":"362d480a-6622-40a3-ab4f-8a85c54b3f06","Type":"ContainerDied","Data":"f8ff722396fada580ae44b23fc7dc013b39da7b2b5fd97050ec2f160fd55c9ff"} Oct 30 09:45:04.574270 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:04.574255 4562 scope.go:122] "RemoveContainer" containerID="f8ff722396fada580ae44b23fc7dc013b39da7b2b5fd97050ec2f160fd55c9ff" Oct 30 09:45:04.574536 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:45:04.574517 4562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889225-7v6l2_odh-ai-gateway-infra(362d480a-6622-40a3-ab4f-8a85c54b3f06)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889225-7v6l2" podUID="362d480a-6622-40a3-ab4f-8a85c54b3f06" Oct 30 09:45:04.575907 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:04.575890 4562 generic.go:356] "Generic (PLEG): container finished" podID="802f9a81-779c-49ff-8e04-283b72c0a908" containerID="55b608d455aac6581ab6489036004bb54948a28232628d83fed52bb9a2076deb" exitCode=60 Oct 30 09:45:04.576003 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:04.575973 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889225-rkfrx" event={"ID":"802f9a81-779c-49ff-8e04-283b72c0a908","Type":"ContainerDied","Data":"55b608d455aac6581ab6489036004bb54948a28232628d83fed52bb9a2076deb"} Oct 30 09:45:04.576161 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:04.576145 4562 scope.go:122] "RemoveContainer" containerID="55b608d455aac6581ab6489036004bb54948a28232628d83fed52bb9a2076deb" Oct 30 09:45:04.576432 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:45:04.576409 4562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-c5447d-29889225-rkfrx_odh-ai-gateway-infra(802f9a81-779c-49ff-8e04-283b72c0a908)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889225-rkfrx" podUID="802f9a81-779c-49ff-8e04-283b72c0a908" Oct 30 09:45:04.577648 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:04.577627 4562 generic.go:356] "Generic (PLEG): container finished" podID="6da36b71-6952-4a6c-a029-ebefe69addb2" containerID="cf548f78240be07a48968824706b3428618a936ea2ac03dc91ae73017dc54309" exitCode=60 Oct 30 09:45:04.577717 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:04.577703 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-88cbc7-29889225-2h7ds" event={"ID":"6da36b71-6952-4a6c-a029-ebefe69addb2","Type":"ContainerDied","Data":"cf548f78240be07a48968824706b3428618a936ea2ac03dc91ae73017dc54309"} Oct 30 09:45:04.577869 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:04.577856 4562 scope.go:122] "RemoveContainer" containerID="cf548f78240be07a48968824706b3428618a936ea2ac03dc91ae73017dc54309" Oct 30 09:45:04.578093 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:45:04.578073 4562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-88cbc7-29889225-2h7ds_odh-ai-gateway-infra(6da36b71-6952-4a6c-a029-ebefe69addb2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-88cbc7-29889225-2h7ds" podUID="6da36b71-6952-4a6c-a029-ebefe69addb2" Oct 30 09:45:04.649202 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:04.649174 4562 scope.go:122] "RemoveContainer" containerID="e81a3cc6d94378fb4f3c8f29fdacee4847865923e5b6a84c3ee428b16d35ab1e" Oct 30 09:45:04.663051 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:04.663028 4562 scope.go:122] "RemoveContainer" containerID="2bf43f1f4c23477249f72de82d4fea8ff3b178a7c0c05badc7a03b4f88701f37" Oct 30 09:45:04.676055 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:04.676036 4562 scope.go:122] "RemoveContainer" containerID="84a00a342e0eeb7d6403f05e2dda408b57537f882dcca620814a6cf3fad50699" Oct 30 09:45:04.684437 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:04.684400 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5d5cqnr4z" podUID="370afa2f-9b22-491c-8574-985a715e84c2" containerName="payload-processing" containerID="cri-o://197ab4a0dd214b995e5eb8d1347a96d9e77224f92e2886689e05dac668f1e1d0" gracePeriod=25 Oct 30 09:45:04.691101 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:04.691077 4562 scope.go:122] "RemoveContainer" containerID="528727cc825a78e2c4d35f0aff0c7c03081723b6b733c3c20fa75d40202c13aa" Oct 30 09:45:04.697697 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:04.697667 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-6c845vbkc" podUID="52ee1a5d-2e5c-460c-a8c8-92eeb570bb5b" containerName="payload-processing" containerID="cri-o://5ffb5f2b789e421a7c2852caa33f092737b85ada4bac293f824740e9bbf38532" gracePeriod=25 Oct 30 09:45:04.831833 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:04.831808 4562 scope.go:122] "RemoveContainer" containerID="eb11ec0c09974333dbd8c2bfa581922f9f91583c55e7e011c5b45196b5340a59" Oct 30 09:45:04.850348 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:04.850321 4562 scope.go:122] "RemoveContainer" containerID="418f86f91a5ca34c1e694e2ab37ed65f31ba0d4c3b643464d26a9333e6e6af4d" Oct 30 09:45:04.993233 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:04.993196 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5d5cqnr4z" Oct 30 09:45:05.011539 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:05.011510 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-6c845vbkc" Oct 30 09:45:05.066441 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:05.066344 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/370afa2f-9b22-491c-8574-985a715e84c2-provider-credentials\" (UniqueName: \"kubernetes.io/projected/370afa2f-9b22-491c-8574-985a715e84c2-provider-credentials\") pod \"370afa2f-9b22-491c-8574-985a715e84c2\" (UID: \"370afa2f-9b22-491c-8574-985a715e84c2\") " Oct 30 09:45:05.066596 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:05.066463 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/370afa2f-9b22-491c-8574-985a715e84c2-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/370afa2f-9b22-491c-8574-985a715e84c2-routing-overlay-volume\") pod \"370afa2f-9b22-491c-8574-985a715e84c2\" (UID: \"370afa2f-9b22-491c-8574-985a715e84c2\") " Oct 30 09:45:05.066596 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:05.066505 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52ee1a5d-2e5c-460c-a8c8-92eeb570bb5b-kube-api-access-bmz2s\" (UniqueName: \"kubernetes.io/projected/52ee1a5d-2e5c-460c-a8c8-92eeb570bb5b-kube-api-access-bmz2s\") pod \"52ee1a5d-2e5c-460c-a8c8-92eeb570bb5b\" (UID: \"52ee1a5d-2e5c-460c-a8c8-92eeb570bb5b\") " Oct 30 09:45:05.066707 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:05.066634 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/370afa2f-9b22-491c-8574-985a715e84c2-kube-api-access-pp94g\" (UniqueName: \"kubernetes.io/projected/370afa2f-9b22-491c-8574-985a715e84c2-kube-api-access-pp94g\") pod \"370afa2f-9b22-491c-8574-985a715e84c2\" (UID: \"370afa2f-9b22-491c-8574-985a715e84c2\") " Oct 30 09:45:05.066707 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:05.066691 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/370afa2f-9b22-491c-8574-985a715e84c2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/370afa2f-9b22-491c-8574-985a715e84c2-plugins-config-volume\") pod \"370afa2f-9b22-491c-8574-985a715e84c2\" (UID: \"370afa2f-9b22-491c-8574-985a715e84c2\") " Oct 30 09:45:05.066828 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:05.066745 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52ee1a5d-2e5c-460c-a8c8-92eeb570bb5b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/52ee1a5d-2e5c-460c-a8c8-92eeb570bb5b-provider-credentials\") pod \"52ee1a5d-2e5c-460c-a8c8-92eeb570bb5b\" (UID: \"52ee1a5d-2e5c-460c-a8c8-92eeb570bb5b\") " Oct 30 09:45:05.067011 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:05.066987 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52ee1a5d-2e5c-460c-a8c8-92eeb570bb5b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52ee1a5d-2e5c-460c-a8c8-92eeb570bb5b-plugins-config-volume\") pod \"52ee1a5d-2e5c-460c-a8c8-92eeb570bb5b\" (UID: \"52ee1a5d-2e5c-460c-a8c8-92eeb570bb5b\") " Oct 30 09:45:05.067100 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:05.067053 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52ee1a5d-2e5c-460c-a8c8-92eeb570bb5b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/52ee1a5d-2e5c-460c-a8c8-92eeb570bb5b-routing-overlay-volume\") pod \"52ee1a5d-2e5c-460c-a8c8-92eeb570bb5b\" (UID: \"52ee1a5d-2e5c-460c-a8c8-92eeb570bb5b\") " Oct 30 09:45:05.068580 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:05.068548 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52ee1a5d-2e5c-460c-a8c8-92eeb570bb5b-routing-overlay-volume" pod "52ee1a5d-2e5c-460c-a8c8-92eeb570bb5b" (UID: "52ee1a5d-2e5c-460c-a8c8-92eeb570bb5b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:45:05.071446 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:05.071415 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/370afa2f-9b22-491c-8574-985a715e84c2-provider-credentials" pod "370afa2f-9b22-491c-8574-985a715e84c2" (UID: "370afa2f-9b22-491c-8574-985a715e84c2"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:45:05.071859 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:05.071830 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/370afa2f-9b22-491c-8574-985a715e84c2-plugins-config-volume" pod "370afa2f-9b22-491c-8574-985a715e84c2" (UID: "370afa2f-9b22-491c-8574-985a715e84c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:45:05.073364 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:05.073339 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/370afa2f-9b22-491c-8574-985a715e84c2-routing-overlay-volume" pod "370afa2f-9b22-491c-8574-985a715e84c2" (UID: "370afa2f-9b22-491c-8574-985a715e84c2"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:45:05.073514 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:05.073469 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52ee1a5d-2e5c-460c-a8c8-92eeb570bb5b-plugins-config-volume" pod "52ee1a5d-2e5c-460c-a8c8-92eeb570bb5b" (UID: "52ee1a5d-2e5c-460c-a8c8-92eeb570bb5b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:45:05.074440 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:05.074411 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52ee1a5d-2e5c-460c-a8c8-92eeb570bb5b-provider-credentials" pod "52ee1a5d-2e5c-460c-a8c8-92eeb570bb5b" (UID: "52ee1a5d-2e5c-460c-a8c8-92eeb570bb5b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:45:05.079539 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:05.079504 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52ee1a5d-2e5c-460c-a8c8-92eeb570bb5b-kube-api-access-bmz2s" pod "52ee1a5d-2e5c-460c-a8c8-92eeb570bb5b" (UID: "52ee1a5d-2e5c-460c-a8c8-92eeb570bb5b"). InnerVolumeSpecName "kube-api-access-bmz2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:45:05.086485 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:05.086441 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/370afa2f-9b22-491c-8574-985a715e84c2-kube-api-access-pp94g" pod "370afa2f-9b22-491c-8574-985a715e84c2" (UID: "370afa2f-9b22-491c-8574-985a715e84c2"). InnerVolumeSpecName "kube-api-access-pp94g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:45:05.169571 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:05.169529 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp94g\" (UniqueName: \"kubernetes.io/projected/370afa2f-9b22-491c-8574-985a715e84c2-kube-api-access-pp94g\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:45:05.169571 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:05.169564 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/370afa2f-9b22-491c-8574-985a715e84c2-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:45:05.169571 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:05.169577 4562 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/52ee1a5d-2e5c-460c-a8c8-92eeb570bb5b-provider-credentials\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:45:05.169820 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:05.169590 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52ee1a5d-2e5c-460c-a8c8-92eeb570bb5b-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:45:05.169820 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:05.169602 4562 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/52ee1a5d-2e5c-460c-a8c8-92eeb570bb5b-routing-overlay-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:45:05.169820 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:05.169614 4562 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/370afa2f-9b22-491c-8574-985a715e84c2-provider-credentials\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:45:05.169820 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:05.169625 4562 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/370afa2f-9b22-491c-8574-985a715e84c2-routing-overlay-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:45:05.169820 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:05.169637 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmz2s\" (UniqueName: \"kubernetes.io/projected/52ee1a5d-2e5c-460c-a8c8-92eeb570bb5b-kube-api-access-bmz2s\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:45:05.600617 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:05.598588 4562 scope.go:122] "RemoveContainer" containerID="b71577ba908a980479c3cff6daff5b1f46667f2c0fd735b4175d2f69c33d0bca" Oct 30 09:45:05.600617 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:45:05.599079 4562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-d60c68-29889225-nlbtt_odh-ai-gateway-infra(9b26e444-ab39-4a88-b18f-34c4846cb17b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d60c68-29889225-nlbtt" podUID="9b26e444-ab39-4a88-b18f-34c4846cb17b" Oct 30 09:45:05.601658 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:05.601557 4562 generic.go:356] "Generic (PLEG): container finished" podID="370afa2f-9b22-491c-8574-985a715e84c2" containerID="197ab4a0dd214b995e5eb8d1347a96d9e77224f92e2886689e05dac668f1e1d0" exitCode=0 Oct 30 09:45:05.601658 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:05.601638 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5d5cqnr4z" event={"ID":"370afa2f-9b22-491c-8574-985a715e84c2","Type":"ContainerDied","Data":"197ab4a0dd214b995e5eb8d1347a96d9e77224f92e2886689e05dac668f1e1d0"} Oct 30 09:45:05.601846 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:05.601718 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5d5cqnr4z" event={"ID":"370afa2f-9b22-491c-8574-985a715e84c2","Type":"ContainerDied","Data":"99497a6857512676025efaff542167092501198d5cb7265b7800113ac0926cf1"} Oct 30 09:45:05.601846 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:05.601821 4562 scope.go:122] "RemoveContainer" containerID="197ab4a0dd214b995e5eb8d1347a96d9e77224f92e2886689e05dac668f1e1d0" Oct 30 09:45:05.616441 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:05.602066 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5d5cqnr4z" Oct 30 09:45:05.621156 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:05.619479 4562 scope.go:122] "RemoveContainer" containerID="f8ff722396fada580ae44b23fc7dc013b39da7b2b5fd97050ec2f160fd55c9ff" Oct 30 09:45:05.621156 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:45:05.619897 4562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889225-7v6l2_odh-ai-gateway-infra(362d480a-6622-40a3-ab4f-8a85c54b3f06)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889225-7v6l2" podUID="362d480a-6622-40a3-ab4f-8a85c54b3f06" Oct 30 09:45:05.626426 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:05.626002 4562 scope.go:122] "RemoveContainer" containerID="55b608d455aac6581ab6489036004bb54948a28232628d83fed52bb9a2076deb" Oct 30 09:45:05.626569 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:45:05.626468 4562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-c5447d-29889225-rkfrx_odh-ai-gateway-infra(802f9a81-779c-49ff-8e04-283b72c0a908)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889225-rkfrx" podUID="802f9a81-779c-49ff-8e04-283b72c0a908" Oct 30 09:45:05.630212 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:05.630188 4562 scope.go:122] "RemoveContainer" containerID="cf548f78240be07a48968824706b3428618a936ea2ac03dc91ae73017dc54309" Oct 30 09:45:05.630591 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:45:05.630567 4562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-88cbc7-29889225-2h7ds_odh-ai-gateway-infra(6da36b71-6952-4a6c-a029-ebefe69addb2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-88cbc7-29889225-2h7ds" podUID="6da36b71-6952-4a6c-a029-ebefe69addb2" Oct 30 09:45:05.632586 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:05.632561 4562 generic.go:356] "Generic (PLEG): container finished" podID="52ee1a5d-2e5c-460c-a8c8-92eeb570bb5b" containerID="5ffb5f2b789e421a7c2852caa33f092737b85ada4bac293f824740e9bbf38532" exitCode=0 Oct 30 09:45:05.632693 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:05.632631 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-6c845vbkc" Oct 30 09:45:05.632693 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:05.632646 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-6c845vbkc" event={"ID":"52ee1a5d-2e5c-460c-a8c8-92eeb570bb5b","Type":"ContainerDied","Data":"5ffb5f2b789e421a7c2852caa33f092737b85ada4bac293f824740e9bbf38532"} Oct 30 09:45:05.632794 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:05.632706 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-6c845vbkc" event={"ID":"52ee1a5d-2e5c-460c-a8c8-92eeb570bb5b","Type":"ContainerDied","Data":"3cf1d4ee7fbc9000a3fddea2b5b1dc16470d606058e357e5e413786f62fd3eb7"} Oct 30 09:45:05.633241 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:05.633223 4562 scope.go:122] "RemoveContainer" containerID="197ab4a0dd214b995e5eb8d1347a96d9e77224f92e2886689e05dac668f1e1d0" Oct 30 09:45:05.633639 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:45:05.633612 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"197ab4a0dd214b995e5eb8d1347a96d9e77224f92e2886689e05dac668f1e1d0\": container with ID starting with 197ab4a0dd214b995e5eb8d1347a96d9e77224f92e2886689e05dac668f1e1d0 not found: ID does not exist" containerID="197ab4a0dd214b995e5eb8d1347a96d9e77224f92e2886689e05dac668f1e1d0" Oct 30 09:45:05.633726 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:05.633645 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"197ab4a0dd214b995e5eb8d1347a96d9e77224f92e2886689e05dac668f1e1d0"} err="failed to get container status \"197ab4a0dd214b995e5eb8d1347a96d9e77224f92e2886689e05dac668f1e1d0\": rpc error: code = NotFound desc = could not find container \"197ab4a0dd214b995e5eb8d1347a96d9e77224f92e2886689e05dac668f1e1d0\": container with ID starting with 197ab4a0dd214b995e5eb8d1347a96d9e77224f92e2886689e05dac668f1e1d0 not found: ID does not exist" Oct 30 09:45:05.633726 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:05.633663 4562 scope.go:122] "RemoveContainer" containerID="5ffb5f2b789e421a7c2852caa33f092737b85ada4bac293f824740e9bbf38532" Oct 30 09:45:05.638354 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:05.638338 4562 scope.go:122] "RemoveContainer" containerID="0074c834a1983f3f695585ab9fd14bafef306bad489c9676bd46d76fa145097e" Oct 30 09:45:05.638728 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:45:05.638701 4562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-035d7d-29889225-7tnvm_odh-ai-gateway-infra(db53ebe2-e3dd-4580-b79c-830feff939a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-035d7d-29889225-7tnvm" podUID="db53ebe2-e3dd-4580-b79c-830feff939a1" Oct 30 09:45:05.640842 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:05.640823 4562 scope.go:122] "RemoveContainer" containerID="2ba54bfb111bad3b18dabf7a84638fd1db2ff9a60391b16f95435f3dbf1f848f" Oct 30 09:45:05.641149 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:45:05.641126 4562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-f58f32-29889225-s94vs_odh-ai-gateway-infra(f25074f0-f8c9-4a6f-b899-4ae6a7ff4311)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f58f32-29889225-s94vs" podUID="f25074f0-f8c9-4a6f-b899-4ae6a7ff4311" Oct 30 09:45:05.642558 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:05.642541 4562 scope.go:122] "RemoveContainer" containerID="67683240ccc55d0f8cd0edb7438964f79cc043ce86fe759a9103b19fd6f0f62d" Oct 30 09:45:05.642839 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:45:05.642800 4562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-fb1a2a-29889225-qq76k_odh-ai-gateway-infra(83484f50-c325-4c2c-beac-16f69e38eb66)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fb1a2a-29889225-qq76k" podUID="83484f50-c325-4c2c-beac-16f69e38eb66" Oct 30 09:45:05.648846 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:05.648831 4562 scope.go:122] "RemoveContainer" containerID="5ffb5f2b789e421a7c2852caa33f092737b85ada4bac293f824740e9bbf38532" Oct 30 09:45:05.649111 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:45:05.649091 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ffb5f2b789e421a7c2852caa33f092737b85ada4bac293f824740e9bbf38532\": container with ID starting with 5ffb5f2b789e421a7c2852caa33f092737b85ada4bac293f824740e9bbf38532 not found: ID does not exist" containerID="5ffb5f2b789e421a7c2852caa33f092737b85ada4bac293f824740e9bbf38532" Oct 30 09:45:05.649155 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:05.649118 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ffb5f2b789e421a7c2852caa33f092737b85ada4bac293f824740e9bbf38532"} err="failed to get container status \"5ffb5f2b789e421a7c2852caa33f092737b85ada4bac293f824740e9bbf38532\": rpc error: code = NotFound desc = could not find container \"5ffb5f2b789e421a7c2852caa33f092737b85ada4bac293f824740e9bbf38532\": container with ID starting with 5ffb5f2b789e421a7c2852caa33f092737b85ada4bac293f824740e9bbf38532 not found: ID does not exist" Oct 30 09:45:05.757259 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:05.757226 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5d5cqnr4z"] Oct 30 09:45:05.761239 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:05.761213 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5d5cqnr4z"] Oct 30 09:45:05.877982 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:05.877891 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="370afa2f-9b22-491c-8574-985a715e84c2" path="/var/lib/kubelet/pods/370afa2f-9b22-491c-8574-985a715e84c2/volumes" Oct 30 09:45:05.889359 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:05.889331 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-6c845vbkc"] Oct 30 09:45:05.893275 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:05.893249 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-6c845vbkc"] Oct 30 09:45:07.877155 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:07.877122 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52ee1a5d-2e5c-460c-a8c8-92eeb570bb5b" path="/var/lib/kubelet/pods/52ee1a5d-2e5c-460c-a8c8-92eeb570bb5b/volumes" Oct 30 09:45:15.867549 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:15.867515 4562 scope.go:122] "RemoveContainer" containerID="cf548f78240be07a48968824706b3428618a936ea2ac03dc91ae73017dc54309" Oct 30 09:45:15.868024 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:15.867835 4562 scope.go:122] "RemoveContainer" containerID="2ba54bfb111bad3b18dabf7a84638fd1db2ff9a60391b16f95435f3dbf1f848f" Oct 30 09:45:15.868024 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:15.867903 4562 scope.go:122] "RemoveContainer" containerID="67683240ccc55d0f8cd0edb7438964f79cc043ce86fe759a9103b19fd6f0f62d" Oct 30 09:45:16.321490 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:16.321349 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-58cc4wdfp"] Oct 30 09:45:16.329570 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:16.329509 4562 cpu_manager.go:414] "containerMap: removing container" podUID="370afa2f-9b22-491c-8574-985a715e84c2" containerName="payload-processing" Oct 30 09:45:16.329570 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:16.329540 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="370afa2f-9b22-491c-8574-985a715e84c2" containerName="payload-processing" Oct 30 09:45:16.329926 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:16.329676 4562 cpu_manager.go:414] "containerMap: removing container" podUID="52ee1a5d-2e5c-460c-a8c8-92eeb570bb5b" containerName="payload-processing" Oct 30 09:45:16.329926 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:16.329687 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52ee1a5d-2e5c-460c-a8c8-92eeb570bb5b" containerName="payload-processing" Oct 30 09:45:16.330104 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:16.329983 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="370afa2f-9b22-491c-8574-985a715e84c2" containerName="payload-processing" Oct 30 09:45:16.330104 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:16.330008 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="52ee1a5d-2e5c-460c-a8c8-92eeb570bb5b" containerName="payload-processing" Oct 30 09:45:16.336897 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:16.336872 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-58cc4wdfp" Oct 30 09:45:16.342537 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:16.342509 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-58cc4wdfp"] Oct 30 09:45:16.375159 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:16.375128 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d60c68\"/\"payload-processing-external-model-e2e--2110ea34-dockercfg-d56mf\"" Oct 30 09:45:16.375314 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:16.375195 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d60c68\"/\"agc-cred-b17ddae8de95\"" Oct 30 09:45:16.375448 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:16.375425 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d60c68\"/\"payload-processing-external-model-plugins-e2e-extmod-a-d60c68\"" Oct 30 09:45:16.375623 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:16.375601 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d60c68\"/\"openshift-service-ca.crt\"" Oct 30 09:45:16.375719 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:16.375704 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d60c68\"/\"kube-root-ca.crt\"" Oct 30 09:45:16.375815 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:16.375799 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d60c68\"/\"routing-overlay\"" Oct 30 09:45:16.418046 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:16.417966 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49m98\" (UniqueName: \"kubernetes.io/projected/10c51f33-a21c-4a58-9c39-4c7eaa52e50a-kube-api-access-49m98\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-58cc4wdfp\" (UID: \"10c51f33-a21c-4a58-9c39-4c7eaa52e50a\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-58cc4wdfp" Oct 30 09:45:16.418175 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:16.418106 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/10c51f33-a21c-4a58-9c39-4c7eaa52e50a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-58cc4wdfp\" (UID: \"10c51f33-a21c-4a58-9c39-4c7eaa52e50a\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-58cc4wdfp" Oct 30 09:45:16.418275 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:16.418258 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10c51f33-a21c-4a58-9c39-4c7eaa52e50a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-58cc4wdfp\" (UID: \"10c51f33-a21c-4a58-9c39-4c7eaa52e50a\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-58cc4wdfp" Oct 30 09:45:16.418330 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:16.418318 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/10c51f33-a21c-4a58-9c39-4c7eaa52e50a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-58cc4wdfp\" (UID: \"10c51f33-a21c-4a58-9c39-4c7eaa52e50a\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-58cc4wdfp" Oct 30 09:45:16.542404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:16.524426 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49m98\" (UniqueName: \"kubernetes.io/projected/10c51f33-a21c-4a58-9c39-4c7eaa52e50a-kube-api-access-49m98\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-58cc4wdfp\" (UID: \"10c51f33-a21c-4a58-9c39-4c7eaa52e50a\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-58cc4wdfp" Oct 30 09:45:16.542404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:16.524550 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/10c51f33-a21c-4a58-9c39-4c7eaa52e50a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-58cc4wdfp\" (UID: \"10c51f33-a21c-4a58-9c39-4c7eaa52e50a\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-58cc4wdfp" Oct 30 09:45:16.542404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:16.524666 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10c51f33-a21c-4a58-9c39-4c7eaa52e50a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-58cc4wdfp\" (UID: \"10c51f33-a21c-4a58-9c39-4c7eaa52e50a\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-58cc4wdfp" Oct 30 09:45:16.542404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:16.524734 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/10c51f33-a21c-4a58-9c39-4c7eaa52e50a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-58cc4wdfp\" (UID: \"10c51f33-a21c-4a58-9c39-4c7eaa52e50a\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-58cc4wdfp" Oct 30 09:45:16.542404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:16.525866 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/10c51f33-a21c-4a58-9c39-4c7eaa52e50a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-58cc4wdfp\" (UID: \"10c51f33-a21c-4a58-9c39-4c7eaa52e50a\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-58cc4wdfp" Oct 30 09:45:16.542404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:16.531706 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10c51f33-a21c-4a58-9c39-4c7eaa52e50a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-58cc4wdfp\" (UID: \"10c51f33-a21c-4a58-9c39-4c7eaa52e50a\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-58cc4wdfp" Oct 30 09:45:16.598403 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:16.597107 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49m98\" (UniqueName: \"kubernetes.io/projected/10c51f33-a21c-4a58-9c39-4c7eaa52e50a-kube-api-access-49m98\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-58cc4wdfp\" (UID: \"10c51f33-a21c-4a58-9c39-4c7eaa52e50a\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-58cc4wdfp" Oct 30 09:45:16.607402 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:16.601948 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/10c51f33-a21c-4a58-9c39-4c7eaa52e50a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-58cc4wdfp\" (UID: \"10c51f33-a21c-4a58-9c39-4c7eaa52e50a\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-58cc4wdfp" Oct 30 09:45:16.678287 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:16.678202 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-58cc4wdfp" Oct 30 09:45:16.780560 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:16.779673 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-88cbc7-29889225-2h7ds" event={"ID":"6da36b71-6952-4a6c-a029-ebefe69addb2","Type":"ContainerStarted","Data":"9525ddf0a7419d6def999a7066464626f81fd1a727ac59bd545737b0aab81448"} Oct 30 09:45:16.782590 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:16.781867 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f58f32-29889225-s94vs" event={"ID":"f25074f0-f8c9-4a6f-b899-4ae6a7ff4311","Type":"ContainerStarted","Data":"0e5a795d619989a02582990ccd820ffce78e716b0ac066ebd99309342baca004"} Oct 30 09:45:16.784058 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:16.784032 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fb1a2a-29889225-qq76k" event={"ID":"83484f50-c325-4c2c-beac-16f69e38eb66","Type":"ContainerStarted","Data":"452e586c7f4a2a08252f41db3e82d373dcc0b08e0195cbf12b517fa8b29498db"} Oct 30 09:45:16.867872 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:16.867845 4562 scope.go:122] "RemoveContainer" containerID="55b608d455aac6581ab6489036004bb54948a28232628d83fed52bb9a2076deb" Oct 30 09:45:16.983786 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:16.983758 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-58cc4wdfp"] Oct 30 09:45:17.793449 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:17.793412 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889225-rkfrx" event={"ID":"802f9a81-779c-49ff-8e04-283b72c0a908","Type":"ContainerStarted","Data":"89faef974ae40d555342a98af7a73178dfc68a838d232972f498b450f56fd34e"} Oct 30 09:45:17.795362 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:17.795336 4562 generic.go:356] "Generic (PLEG): container finished" podID="6da36b71-6952-4a6c-a029-ebefe69addb2" containerID="9525ddf0a7419d6def999a7066464626f81fd1a727ac59bd545737b0aab81448" exitCode=60 Oct 30 09:45:17.795508 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:17.795415 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-88cbc7-29889225-2h7ds" event={"ID":"6da36b71-6952-4a6c-a029-ebefe69addb2","Type":"ContainerDied","Data":"9525ddf0a7419d6def999a7066464626f81fd1a727ac59bd545737b0aab81448"} Oct 30 09:45:17.795508 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:17.795477 4562 scope.go:122] "RemoveContainer" containerID="cf548f78240be07a48968824706b3428618a936ea2ac03dc91ae73017dc54309" Oct 30 09:45:17.795777 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:17.795753 4562 scope.go:122] "RemoveContainer" containerID="9525ddf0a7419d6def999a7066464626f81fd1a727ac59bd545737b0aab81448" Oct 30 09:45:17.796093 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:45:17.796068 4562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-88cbc7-29889225-2h7ds_odh-ai-gateway-infra(6da36b71-6952-4a6c-a029-ebefe69addb2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-88cbc7-29889225-2h7ds" podUID="6da36b71-6952-4a6c-a029-ebefe69addb2" Oct 30 09:45:17.797449 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:17.797431 4562 generic.go:356] "Generic (PLEG): container finished" podID="f25074f0-f8c9-4a6f-b899-4ae6a7ff4311" containerID="0e5a795d619989a02582990ccd820ffce78e716b0ac066ebd99309342baca004" exitCode=60 Oct 30 09:45:17.797540 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:17.797497 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f58f32-29889225-s94vs" event={"ID":"f25074f0-f8c9-4a6f-b899-4ae6a7ff4311","Type":"ContainerDied","Data":"0e5a795d619989a02582990ccd820ffce78e716b0ac066ebd99309342baca004"} Oct 30 09:45:17.797827 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:17.797812 4562 scope.go:122] "RemoveContainer" containerID="0e5a795d619989a02582990ccd820ffce78e716b0ac066ebd99309342baca004" Oct 30 09:45:17.798099 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:45:17.798083 4562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-f58f32-29889225-s94vs_odh-ai-gateway-infra(f25074f0-f8c9-4a6f-b899-4ae6a7ff4311)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f58f32-29889225-s94vs" podUID="f25074f0-f8c9-4a6f-b899-4ae6a7ff4311" Oct 30 09:45:17.799309 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:17.799291 4562 generic.go:356] "Generic (PLEG): container finished" podID="83484f50-c325-4c2c-beac-16f69e38eb66" containerID="452e586c7f4a2a08252f41db3e82d373dcc0b08e0195cbf12b517fa8b29498db" exitCode=60 Oct 30 09:45:17.799392 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:17.799361 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fb1a2a-29889225-qq76k" event={"ID":"83484f50-c325-4c2c-beac-16f69e38eb66","Type":"ContainerDied","Data":"452e586c7f4a2a08252f41db3e82d373dcc0b08e0195cbf12b517fa8b29498db"} Oct 30 09:45:17.799723 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:17.799705 4562 scope.go:122] "RemoveContainer" containerID="452e586c7f4a2a08252f41db3e82d373dcc0b08e0195cbf12b517fa8b29498db" Oct 30 09:45:17.799963 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:45:17.799941 4562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-fb1a2a-29889225-qq76k_odh-ai-gateway-infra(83484f50-c325-4c2c-beac-16f69e38eb66)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fb1a2a-29889225-qq76k" podUID="83484f50-c325-4c2c-beac-16f69e38eb66" Oct 30 09:45:17.801003 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:17.800978 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-58cc4wdfp" event={"ID":"10c51f33-a21c-4a58-9c39-4c7eaa52e50a","Type":"ContainerStarted","Data":"e8bf2fe9c9eb3080692ef2cc971bd9a97a33b7b6198fb8bbbb71b03f1b120770"} Oct 30 09:45:17.801109 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:17.801022 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-58cc4wdfp" event={"ID":"10c51f33-a21c-4a58-9c39-4c7eaa52e50a","Type":"ContainerStarted","Data":"2ba09fe452aa83680dafdf2aed41d9705cf3d0c45806869926d16323edcfcd35"} Oct 30 09:45:17.801109 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:17.801055 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-58cc4wdfp" Oct 30 09:45:17.808732 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:17.808714 4562 scope.go:122] "RemoveContainer" containerID="2ba54bfb111bad3b18dabf7a84638fd1db2ff9a60391b16f95435f3dbf1f848f" Oct 30 09:45:17.823840 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:17.823814 4562 scope.go:122] "RemoveContainer" containerID="67683240ccc55d0f8cd0edb7438964f79cc043ce86fe759a9103b19fd6f0f62d" Oct 30 09:45:17.834730 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:17.834699 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-88cbc7-29889225-2h7ds"] Oct 30 09:45:17.867699 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:17.867678 4562 scope.go:122] "RemoveContainer" containerID="0074c834a1983f3f695585ab9fd14bafef306bad489c9676bd46d76fa145097e" Oct 30 09:45:17.867877 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:17.867859 4562 scope.go:122] "RemoveContainer" containerID="f8ff722396fada580ae44b23fc7dc013b39da7b2b5fd97050ec2f160fd55c9ff" Oct 30 09:45:17.868110 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:17.868094 4562 scope.go:122] "RemoveContainer" containerID="b71577ba908a980479c3cff6daff5b1f46667f2c0fd735b4175d2f69c33d0bca" Oct 30 09:45:17.891983 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:17.891954 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fb1a2a-29889225-qq76k"] Oct 30 09:45:17.924219 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:17.924184 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f58f32-29889225-s94vs"] Oct 30 09:45:17.946619 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:17.946563 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-58cc4wdfp" podStartSLOduration=1.9465447139999998 podStartE2EDuration="1.946544714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:45:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:45:17.940825252 +0000 UTC m=+3078.202386243" watchObservedRunningTime="2026-10-30 09:45:17.946544714 +0000 UTC m=+3078.208105704" Oct 30 09:45:18.810192 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:18.810109 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-035d7d-29889225-7tnvm" event={"ID":"db53ebe2-e3dd-4580-b79c-830feff939a1","Type":"ContainerStarted","Data":"033e4449202bfacf0480d7f100c34567689f828ca2cf5ac1079c9bf31d0a360d"} Oct 30 09:45:18.816991 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:18.816952 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d60c68-29889225-nlbtt" event={"ID":"9b26e444-ab39-4a88-b18f-34c4846cb17b","Type":"ContainerStarted","Data":"457226a849d8295c6549bc9e8279e63305bcf10d2ee1b813bd452a7a43b74bda"} Oct 30 09:45:18.819653 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:18.819621 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889225-7v6l2" event={"ID":"362d480a-6622-40a3-ab4f-8a85c54b3f06","Type":"ContainerStarted","Data":"306b2ca8e157d1cd6cb99750e574e50b598193177823fe3c94d080294e7279b6"} Oct 30 09:45:18.822711 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:18.822687 4562 generic.go:356] "Generic (PLEG): container finished" podID="802f9a81-779c-49ff-8e04-283b72c0a908" containerID="89faef974ae40d555342a98af7a73178dfc68a838d232972f498b450f56fd34e" exitCode=60 Oct 30 09:45:18.822825 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:18.822739 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889225-rkfrx" event={"ID":"802f9a81-779c-49ff-8e04-283b72c0a908","Type":"ContainerDied","Data":"89faef974ae40d555342a98af7a73178dfc68a838d232972f498b450f56fd34e"} Oct 30 09:45:18.822825 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:18.822799 4562 scope.go:122] "RemoveContainer" containerID="55b608d455aac6581ab6489036004bb54948a28232628d83fed52bb9a2076deb" Oct 30 09:45:18.823135 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:18.823101 4562 scope.go:122] "RemoveContainer" containerID="89faef974ae40d555342a98af7a73178dfc68a838d232972f498b450f56fd34e" Oct 30 09:45:18.823482 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:45:18.823444 4562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-c5447d-29889225-rkfrx_odh-ai-gateway-infra(802f9a81-779c-49ff-8e04-283b72c0a908)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889225-rkfrx" podUID="802f9a81-779c-49ff-8e04-283b72c0a908" Oct 30 09:45:18.852877 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:18.852838 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889225-rkfrx"] Oct 30 09:45:19.229776 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:19.229719 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f58f32-29889225-s94vs" Oct 30 09:45:19.234907 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:19.234510 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-88cbc7-29889225-2h7ds" Oct 30 09:45:19.237271 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:19.237250 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fb1a2a-29889225-qq76k" Oct 30 09:45:19.403485 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:19.403406 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6da36b71-6952-4a6c-a029-ebefe69addb2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6da36b71-6952-4a6c-a029-ebefe69addb2-maas-api-service-ca\") pod \"6da36b71-6952-4a6c-a029-ebefe69addb2\" (UID: \"6da36b71-6952-4a6c-a029-ebefe69addb2\") " Oct 30 09:45:19.403485 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:19.403477 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f25074f0-f8c9-4a6f-b899-4ae6a7ff4311-kube-api-access-mgl48\" (UniqueName: \"kubernetes.io/projected/f25074f0-f8c9-4a6f-b899-4ae6a7ff4311-kube-api-access-mgl48\") pod \"f25074f0-f8c9-4a6f-b899-4ae6a7ff4311\" (UID: \"f25074f0-f8c9-4a6f-b899-4ae6a7ff4311\") " Oct 30 09:45:19.403729 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:19.403503 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6da36b71-6952-4a6c-a029-ebefe69addb2-kube-api-access-hqrhw\" (UniqueName: \"kubernetes.io/projected/6da36b71-6952-4a6c-a029-ebefe69addb2-kube-api-access-hqrhw\") pod \"6da36b71-6952-4a6c-a029-ebefe69addb2\" (UID: \"6da36b71-6952-4a6c-a029-ebefe69addb2\") " Oct 30 09:45:19.403729 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:19.403621 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83484f50-c325-4c2c-beac-16f69e38eb66-kube-api-access-9d8hq\" (UniqueName: \"kubernetes.io/projected/83484f50-c325-4c2c-beac-16f69e38eb66-kube-api-access-9d8hq\") pod \"83484f50-c325-4c2c-beac-16f69e38eb66\" (UID: \"83484f50-c325-4c2c-beac-16f69e38eb66\") " Oct 30 09:45:19.403729 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:19.403658 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83484f50-c325-4c2c-beac-16f69e38eb66-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83484f50-c325-4c2c-beac-16f69e38eb66-maas-api-service-ca\") pod \"83484f50-c325-4c2c-beac-16f69e38eb66\" (UID: \"83484f50-c325-4c2c-beac-16f69e38eb66\") " Oct 30 09:45:19.403904 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:19.403815 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6da36b71-6952-4a6c-a029-ebefe69addb2-maas-api-service-ca" pod "6da36b71-6952-4a6c-a029-ebefe69addb2" (UID: "6da36b71-6952-4a6c-a029-ebefe69addb2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:45:19.403904 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:19.403829 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f25074f0-f8c9-4a6f-b899-4ae6a7ff4311-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f25074f0-f8c9-4a6f-b899-4ae6a7ff4311-maas-api-service-ca\") pod \"f25074f0-f8c9-4a6f-b899-4ae6a7ff4311\" (UID: \"f25074f0-f8c9-4a6f-b899-4ae6a7ff4311\") " Oct 30 09:45:19.404243 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:19.404207 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83484f50-c325-4c2c-beac-16f69e38eb66-maas-api-service-ca" pod "83484f50-c325-4c2c-beac-16f69e38eb66" (UID: "83484f50-c325-4c2c-beac-16f69e38eb66"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:45:19.404499 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:19.404447 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f25074f0-f8c9-4a6f-b899-4ae6a7ff4311-maas-api-service-ca" pod "f25074f0-f8c9-4a6f-b899-4ae6a7ff4311" (UID: "f25074f0-f8c9-4a6f-b899-4ae6a7ff4311"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:45:19.404730 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:19.404706 4562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83484f50-c325-4c2c-beac-16f69e38eb66-maas-api-service-ca\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:45:19.404799 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:19.404739 4562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f25074f0-f8c9-4a6f-b899-4ae6a7ff4311-maas-api-service-ca\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:45:19.404799 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:19.404755 4562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6da36b71-6952-4a6c-a029-ebefe69addb2-maas-api-service-ca\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:45:19.405971 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:19.405944 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83484f50-c325-4c2c-beac-16f69e38eb66-kube-api-access-9d8hq" pod "83484f50-c325-4c2c-beac-16f69e38eb66" (UID: "83484f50-c325-4c2c-beac-16f69e38eb66"). InnerVolumeSpecName "kube-api-access-9d8hq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:45:19.406091 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:19.406046 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f25074f0-f8c9-4a6f-b899-4ae6a7ff4311-kube-api-access-mgl48" pod "f25074f0-f8c9-4a6f-b899-4ae6a7ff4311" (UID: "f25074f0-f8c9-4a6f-b899-4ae6a7ff4311"). InnerVolumeSpecName "kube-api-access-mgl48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:45:19.406215 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:19.406194 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6da36b71-6952-4a6c-a029-ebefe69addb2-kube-api-access-hqrhw" pod "6da36b71-6952-4a6c-a029-ebefe69addb2" (UID: "6da36b71-6952-4a6c-a029-ebefe69addb2"). InnerVolumeSpecName "kube-api-access-hqrhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:45:19.506568 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:19.506537 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgl48\" (UniqueName: \"kubernetes.io/projected/f25074f0-f8c9-4a6f-b899-4ae6a7ff4311-kube-api-access-mgl48\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:45:19.506568 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:19.506564 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqrhw\" (UniqueName: \"kubernetes.io/projected/6da36b71-6952-4a6c-a029-ebefe69addb2-kube-api-access-hqrhw\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:45:19.506568 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:19.506574 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d8hq\" (UniqueName: \"kubernetes.io/projected/83484f50-c325-4c2c-beac-16f69e38eb66-kube-api-access-9d8hq\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:45:19.837266 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:19.837234 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-88cbc7-29889225-2h7ds" Oct 30 09:45:19.837403 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:19.837267 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-88cbc7-29889225-2h7ds" event={"ID":"6da36b71-6952-4a6c-a029-ebefe69addb2","Type":"ContainerDied","Data":"c8211c866d956744a979295dcb19497fe462b5c9595fc12d1480bb44e2be069e"} Oct 30 09:45:19.837403 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:19.837327 4562 scope.go:122] "RemoveContainer" containerID="9525ddf0a7419d6def999a7066464626f81fd1a727ac59bd545737b0aab81448" Oct 30 09:45:19.839454 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:19.839354 4562 generic.go:356] "Generic (PLEG): container finished" podID="db53ebe2-e3dd-4580-b79c-830feff939a1" containerID="033e4449202bfacf0480d7f100c34567689f828ca2cf5ac1079c9bf31d0a360d" exitCode=60 Oct 30 09:45:19.839454 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:19.839424 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-035d7d-29889225-7tnvm" event={"ID":"db53ebe2-e3dd-4580-b79c-830feff939a1","Type":"ContainerDied","Data":"033e4449202bfacf0480d7f100c34567689f828ca2cf5ac1079c9bf31d0a360d"} Oct 30 09:45:19.839903 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:19.839881 4562 scope.go:122] "RemoveContainer" containerID="033e4449202bfacf0480d7f100c34567689f828ca2cf5ac1079c9bf31d0a360d" Oct 30 09:45:19.840250 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:45:19.840215 4562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-035d7d-29889225-7tnvm_odh-ai-gateway-infra(db53ebe2-e3dd-4580-b79c-830feff939a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-035d7d-29889225-7tnvm" podUID="db53ebe2-e3dd-4580-b79c-830feff939a1" Oct 30 09:45:19.841260 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:19.841240 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f58f32-29889225-s94vs" Oct 30 09:45:19.841352 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:19.841242 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f58f32-29889225-s94vs" event={"ID":"f25074f0-f8c9-4a6f-b899-4ae6a7ff4311","Type":"ContainerDied","Data":"aa707bc94e7d16194f66cfea34f459b412f617912f426c904822ee26b3478e9e"} Oct 30 09:45:19.842638 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:19.842614 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fb1a2a-29889225-qq76k" Oct 30 09:45:19.842739 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:19.842659 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fb1a2a-29889225-qq76k" event={"ID":"83484f50-c325-4c2c-beac-16f69e38eb66","Type":"ContainerDied","Data":"9e3def99b6fc66bf35d6e83b10a3a3e0edec3151a51bc11d88a80b42e410abe8"} Oct 30 09:45:19.852472 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:19.852447 4562 generic.go:356] "Generic (PLEG): container finished" podID="9b26e444-ab39-4a88-b18f-34c4846cb17b" containerID="457226a849d8295c6549bc9e8279e63305bcf10d2ee1b813bd452a7a43b74bda" exitCode=60 Oct 30 09:45:19.852700 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:19.852680 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d60c68-29889225-nlbtt" event={"ID":"9b26e444-ab39-4a88-b18f-34c4846cb17b","Type":"ContainerDied","Data":"457226a849d8295c6549bc9e8279e63305bcf10d2ee1b813bd452a7a43b74bda"} Oct 30 09:45:19.853274 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:19.853114 4562 scope.go:122] "RemoveContainer" containerID="457226a849d8295c6549bc9e8279e63305bcf10d2ee1b813bd452a7a43b74bda" Oct 30 09:45:19.853459 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:45:19.853433 4562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-d60c68-29889225-nlbtt_odh-ai-gateway-infra(9b26e444-ab39-4a88-b18f-34c4846cb17b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d60c68-29889225-nlbtt" podUID="9b26e444-ab39-4a88-b18f-34c4846cb17b" Oct 30 09:45:19.857247 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:19.857220 4562 generic.go:356] "Generic (PLEG): container finished" podID="362d480a-6622-40a3-ab4f-8a85c54b3f06" containerID="306b2ca8e157d1cd6cb99750e574e50b598193177823fe3c94d080294e7279b6" exitCode=60 Oct 30 09:45:19.857353 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:19.857253 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889225-7v6l2" event={"ID":"362d480a-6622-40a3-ab4f-8a85c54b3f06","Type":"ContainerDied","Data":"306b2ca8e157d1cd6cb99750e574e50b598193177823fe3c94d080294e7279b6"} Oct 30 09:45:19.857736 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:19.857718 4562 scope.go:122] "RemoveContainer" containerID="306b2ca8e157d1cd6cb99750e574e50b598193177823fe3c94d080294e7279b6" Oct 30 09:45:19.858034 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:45:19.858010 4562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889225-7v6l2_odh-ai-gateway-infra(362d480a-6622-40a3-ab4f-8a85c54b3f06)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889225-7v6l2" podUID="362d480a-6622-40a3-ab4f-8a85c54b3f06" Oct 30 09:45:19.858851 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:19.858829 4562 scope.go:122] "RemoveContainer" containerID="0074c834a1983f3f695585ab9fd14bafef306bad489c9676bd46d76fa145097e" Oct 30 09:45:19.886794 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:19.883075 4562 scope.go:122] "RemoveContainer" containerID="0e5a795d619989a02582990ccd820ffce78e716b0ac066ebd99309342baca004" Oct 30 09:45:19.886794 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:19.884671 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-035d7d-29889225-7tnvm"] Oct 30 09:45:19.907513 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:19.907489 4562 scope.go:122] "RemoveContainer" containerID="452e586c7f4a2a08252f41db3e82d373dcc0b08e0195cbf12b517fa8b29498db" Oct 30 09:45:19.929510 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:19.929483 4562 scope.go:122] "RemoveContainer" containerID="b71577ba908a980479c3cff6daff5b1f46667f2c0fd735b4175d2f69c33d0bca" Oct 30 09:45:19.946586 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:19.946564 4562 scope.go:122] "RemoveContainer" containerID="f8ff722396fada580ae44b23fc7dc013b39da7b2b5fd97050ec2f160fd55c9ff" Oct 30 09:45:20.022314 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:20.022284 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fb1a2a-29889225-qq76k"] Oct 30 09:45:20.037414 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:20.037376 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889225-rkfrx" Oct 30 09:45:20.037758 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:20.037735 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d60c68-29889225-nlbtt"] Oct 30 09:45:20.050175 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:20.050147 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fb1a2a-29889225-qq76k"] Oct 30 09:45:20.115634 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:20.115456 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/802f9a81-779c-49ff-8e04-283b72c0a908-kube-api-access-fq8mc\" (UniqueName: \"kubernetes.io/projected/802f9a81-779c-49ff-8e04-283b72c0a908-kube-api-access-fq8mc\") pod \"802f9a81-779c-49ff-8e04-283b72c0a908\" (UID: \"802f9a81-779c-49ff-8e04-283b72c0a908\") " Oct 30 09:45:20.115789 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:20.115649 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/802f9a81-779c-49ff-8e04-283b72c0a908-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/802f9a81-779c-49ff-8e04-283b72c0a908-maas-api-service-ca\") pod \"802f9a81-779c-49ff-8e04-283b72c0a908\" (UID: \"802f9a81-779c-49ff-8e04-283b72c0a908\") " Oct 30 09:45:20.116223 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:20.116194 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/802f9a81-779c-49ff-8e04-283b72c0a908-maas-api-service-ca" pod "802f9a81-779c-49ff-8e04-283b72c0a908" (UID: "802f9a81-779c-49ff-8e04-283b72c0a908"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:45:20.116419 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:20.116402 4562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/802f9a81-779c-49ff-8e04-283b72c0a908-maas-api-service-ca\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:45:20.117814 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:20.117792 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/802f9a81-779c-49ff-8e04-283b72c0a908-kube-api-access-fq8mc" pod "802f9a81-779c-49ff-8e04-283b72c0a908" (UID: "802f9a81-779c-49ff-8e04-283b72c0a908"). InnerVolumeSpecName "kube-api-access-fq8mc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:45:20.117947 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:20.117930 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889225-7v6l2"] Oct 30 09:45:20.200308 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:20.200275 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-88cbc7-29889225-2h7ds"] Oct 30 09:45:20.205343 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:20.205308 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-88cbc7-29889225-2h7ds"] Oct 30 09:45:20.217245 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:20.217220 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq8mc\" (UniqueName: \"kubernetes.io/projected/802f9a81-779c-49ff-8e04-283b72c0a908-kube-api-access-fq8mc\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:45:20.280535 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:20.280503 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f58f32-29889225-s94vs"] Oct 30 09:45:20.288698 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:20.288669 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f58f32-29889225-s94vs"] Oct 30 09:45:20.878841 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:20.878802 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889225-rkfrx" Oct 30 09:45:20.879104 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:20.878806 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889225-rkfrx" event={"ID":"802f9a81-779c-49ff-8e04-283b72c0a908","Type":"ContainerDied","Data":"57c9edcaa5d59473678233f2629dc8163854cef6f807a4bae715245e0fc5829d"} Oct 30 09:45:20.879270 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:20.879144 4562 scope.go:122] "RemoveContainer" containerID="89faef974ae40d555342a98af7a73178dfc68a838d232972f498b450f56fd34e" Oct 30 09:45:20.956397 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:20.956352 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889225-rkfrx"] Oct 30 09:45:20.968046 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:20.968008 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c5447d-29889225-rkfrx"] Oct 30 09:45:21.212912 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:21.212884 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-035d7d-29889225-7tnvm" Oct 30 09:45:21.269307 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:21.269283 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889225-7v6l2" Oct 30 09:45:21.273051 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:21.273025 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d60c68-29889225-nlbtt" Oct 30 09:45:21.312891 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:21.312855 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/362d480a-6622-40a3-ab4f-8a85c54b3f06-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/362d480a-6622-40a3-ab4f-8a85c54b3f06-maas-api-service-ca\") pod \"362d480a-6622-40a3-ab4f-8a85c54b3f06\" (UID: \"362d480a-6622-40a3-ab4f-8a85c54b3f06\") " Oct 30 09:45:21.313308 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:21.312940 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b26e444-ab39-4a88-b18f-34c4846cb17b-kube-api-access-lr6zz\" (UniqueName: \"kubernetes.io/projected/9b26e444-ab39-4a88-b18f-34c4846cb17b-kube-api-access-lr6zz\") pod \"9b26e444-ab39-4a88-b18f-34c4846cb17b\" (UID: \"9b26e444-ab39-4a88-b18f-34c4846cb17b\") " Oct 30 09:45:21.313308 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:21.313122 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/362d480a-6622-40a3-ab4f-8a85c54b3f06-kube-api-access-gqv6c\" (UniqueName: \"kubernetes.io/projected/362d480a-6622-40a3-ab4f-8a85c54b3f06-kube-api-access-gqv6c\") pod \"362d480a-6622-40a3-ab4f-8a85c54b3f06\" (UID: \"362d480a-6622-40a3-ab4f-8a85c54b3f06\") " Oct 30 09:45:21.313308 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:21.313145 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b26e444-ab39-4a88-b18f-34c4846cb17b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b26e444-ab39-4a88-b18f-34c4846cb17b-maas-api-service-ca\") pod \"9b26e444-ab39-4a88-b18f-34c4846cb17b\" (UID: \"9b26e444-ab39-4a88-b18f-34c4846cb17b\") " Oct 30 09:45:21.313308 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:21.313222 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db53ebe2-e3dd-4580-b79c-830feff939a1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db53ebe2-e3dd-4580-b79c-830feff939a1-maas-api-service-ca\") pod \"db53ebe2-e3dd-4580-b79c-830feff939a1\" (UID: \"db53ebe2-e3dd-4580-b79c-830feff939a1\") " Oct 30 09:45:21.313308 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:21.313272 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db53ebe2-e3dd-4580-b79c-830feff939a1-kube-api-access-pt5ht\" (UniqueName: \"kubernetes.io/projected/db53ebe2-e3dd-4580-b79c-830feff939a1-kube-api-access-pt5ht\") pod \"db53ebe2-e3dd-4580-b79c-830feff939a1\" (UID: \"db53ebe2-e3dd-4580-b79c-830feff939a1\") " Oct 30 09:45:21.315358 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:21.315328 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db53ebe2-e3dd-4580-b79c-830feff939a1-maas-api-service-ca" pod "db53ebe2-e3dd-4580-b79c-830feff939a1" (UID: "db53ebe2-e3dd-4580-b79c-830feff939a1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:45:21.316322 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:21.316294 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b26e444-ab39-4a88-b18f-34c4846cb17b-maas-api-service-ca" pod "9b26e444-ab39-4a88-b18f-34c4846cb17b" (UID: "9b26e444-ab39-4a88-b18f-34c4846cb17b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:45:21.316756 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:21.316732 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/362d480a-6622-40a3-ab4f-8a85c54b3f06-maas-api-service-ca" pod "362d480a-6622-40a3-ab4f-8a85c54b3f06" (UID: "362d480a-6622-40a3-ab4f-8a85c54b3f06"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:45:21.318537 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:21.318498 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/362d480a-6622-40a3-ab4f-8a85c54b3f06-kube-api-access-gqv6c" pod "362d480a-6622-40a3-ab4f-8a85c54b3f06" (UID: "362d480a-6622-40a3-ab4f-8a85c54b3f06"). InnerVolumeSpecName "kube-api-access-gqv6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:45:21.319151 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:21.319125 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db53ebe2-e3dd-4580-b79c-830feff939a1-kube-api-access-pt5ht" pod "db53ebe2-e3dd-4580-b79c-830feff939a1" (UID: "db53ebe2-e3dd-4580-b79c-830feff939a1"). InnerVolumeSpecName "kube-api-access-pt5ht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:45:21.319253 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:21.319227 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b26e444-ab39-4a88-b18f-34c4846cb17b-kube-api-access-lr6zz" pod "9b26e444-ab39-4a88-b18f-34c4846cb17b" (UID: "9b26e444-ab39-4a88-b18f-34c4846cb17b"). InnerVolumeSpecName "kube-api-access-lr6zz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:45:21.416066 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:21.415988 4562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/362d480a-6622-40a3-ab4f-8a85c54b3f06-maas-api-service-ca\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:45:21.416066 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:21.416028 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lr6zz\" (UniqueName: \"kubernetes.io/projected/9b26e444-ab39-4a88-b18f-34c4846cb17b-kube-api-access-lr6zz\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:45:21.416066 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:21.416045 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqv6c\" (UniqueName: \"kubernetes.io/projected/362d480a-6622-40a3-ab4f-8a85c54b3f06-kube-api-access-gqv6c\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:45:21.416066 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:21.416059 4562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b26e444-ab39-4a88-b18f-34c4846cb17b-maas-api-service-ca\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:45:21.416290 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:21.416072 4562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db53ebe2-e3dd-4580-b79c-830feff939a1-maas-api-service-ca\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:45:21.416290 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:21.416086 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pt5ht\" (UniqueName: \"kubernetes.io/projected/db53ebe2-e3dd-4580-b79c-830feff939a1-kube-api-access-pt5ht\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:45:21.883174 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:21.883139 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6da36b71-6952-4a6c-a029-ebefe69addb2" path="/var/lib/kubelet/pods/6da36b71-6952-4a6c-a029-ebefe69addb2/volumes" Oct 30 09:45:21.883763 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:21.883740 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="802f9a81-779c-49ff-8e04-283b72c0a908" path="/var/lib/kubelet/pods/802f9a81-779c-49ff-8e04-283b72c0a908/volumes" Oct 30 09:45:21.884281 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:21.884261 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83484f50-c325-4c2c-beac-16f69e38eb66" path="/var/lib/kubelet/pods/83484f50-c325-4c2c-beac-16f69e38eb66/volumes" Oct 30 09:45:21.885119 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:21.885099 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f25074f0-f8c9-4a6f-b899-4ae6a7ff4311" path="/var/lib/kubelet/pods/f25074f0-f8c9-4a6f-b899-4ae6a7ff4311/volumes" Oct 30 09:45:21.904958 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:21.904932 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-035d7d-29889225-7tnvm" Oct 30 09:45:21.905071 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:21.904967 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-035d7d-29889225-7tnvm" event={"ID":"db53ebe2-e3dd-4580-b79c-830feff939a1","Type":"ContainerDied","Data":"3b11c79805a72b296d63b25772d97a8de4a333d1eaac6e7823d6cefa99e8e66b"} Oct 30 09:45:21.905071 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:21.905035 4562 scope.go:122] "RemoveContainer" containerID="033e4449202bfacf0480d7f100c34567689f828ca2cf5ac1079c9bf31d0a360d" Oct 30 09:45:21.906988 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:21.906968 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d60c68-29889225-nlbtt" event={"ID":"9b26e444-ab39-4a88-b18f-34c4846cb17b","Type":"ContainerDied","Data":"7d3fd435f2d3d94d43d64a64838da06f642d1859eae63d5edc0fcc2814e1c099"} Oct 30 09:45:21.907072 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:21.907043 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d60c68-29889225-nlbtt" Oct 30 09:45:21.910750 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:21.910424 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889225-7v6l2" event={"ID":"362d480a-6622-40a3-ab4f-8a85c54b3f06","Type":"ContainerDied","Data":"a492198831eaaa1e8a0b00e6e4c74acef1213192bb79672c59ad89df4d926c4f"} Oct 30 09:45:21.910995 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:21.910966 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889225-7v6l2" Oct 30 09:45:21.919454 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:21.919437 4562 scope.go:122] "RemoveContainer" containerID="457226a849d8295c6549bc9e8279e63305bcf10d2ee1b813bd452a7a43b74bda" Oct 30 09:45:21.928965 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:21.928944 4562 scope.go:122] "RemoveContainer" containerID="306b2ca8e157d1cd6cb99750e574e50b598193177823fe3c94d080294e7279b6" Oct 30 09:45:21.962578 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:21.962554 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889225-7v6l2"] Oct 30 09:45:21.969273 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:21.969251 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ebd65-29889225-7v6l2"] Oct 30 09:45:22.015150 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:22.015123 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-035d7d-29889225-7tnvm"] Oct 30 09:45:22.018645 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:22.018623 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-035d7d-29889225-7tnvm"] Oct 30 09:45:22.054333 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:22.054280 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d60c68-29889225-nlbtt"] Oct 30 09:45:22.058001 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:22.057978 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d60c68-29889225-nlbtt"] Oct 30 09:45:23.876803 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:23.876770 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="362d480a-6622-40a3-ab4f-8a85c54b3f06" path="/var/lib/kubelet/pods/362d480a-6622-40a3-ab4f-8a85c54b3f06/volumes" Oct 30 09:45:23.877259 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:23.877245 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b26e444-ab39-4a88-b18f-34c4846cb17b" path="/var/lib/kubelet/pods/9b26e444-ab39-4a88-b18f-34c4846cb17b/volumes" Oct 30 09:45:23.877689 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:23.877675 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db53ebe2-e3dd-4580-b79c-830feff939a1" path="/var/lib/kubelet/pods/db53ebe2-e3dd-4580-b79c-830feff939a1/volumes" Oct 30 09:45:28.598127 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:45:28.598092 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:45:28.830924 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:28.830850 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-58cc4wdfp" Oct 30 09:45:29.538992 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:29.538947 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-58cc4wdfp"] Oct 30 09:45:34.539692 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:34.539648 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-58cc4wdfp" podUID="10c51f33-a21c-4a58-9c39-4c7eaa52e50a" containerName="payload-processing" containerID="cri-o://e8bf2fe9c9eb3080692ef2cc971bd9a97a33b7b6198fb8bbbb71b03f1b120770" gracePeriod=25 Oct 30 09:45:34.826012 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:34.825986 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-58cc4wdfp" Oct 30 09:45:34.927307 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:34.927270 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10c51f33-a21c-4a58-9c39-4c7eaa52e50a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10c51f33-a21c-4a58-9c39-4c7eaa52e50a-plugins-config-volume\") pod \"10c51f33-a21c-4a58-9c39-4c7eaa52e50a\" (UID: \"10c51f33-a21c-4a58-9c39-4c7eaa52e50a\") " Oct 30 09:45:34.927496 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:34.927429 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10c51f33-a21c-4a58-9c39-4c7eaa52e50a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/10c51f33-a21c-4a58-9c39-4c7eaa52e50a-routing-overlay-volume\") pod \"10c51f33-a21c-4a58-9c39-4c7eaa52e50a\" (UID: \"10c51f33-a21c-4a58-9c39-4c7eaa52e50a\") " Oct 30 09:45:34.927496 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:34.927477 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10c51f33-a21c-4a58-9c39-4c7eaa52e50a-kube-api-access-49m98\" (UniqueName: \"kubernetes.io/projected/10c51f33-a21c-4a58-9c39-4c7eaa52e50a-kube-api-access-49m98\") pod \"10c51f33-a21c-4a58-9c39-4c7eaa52e50a\" (UID: \"10c51f33-a21c-4a58-9c39-4c7eaa52e50a\") " Oct 30 09:45:34.927595 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:34.927571 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10c51f33-a21c-4a58-9c39-4c7eaa52e50a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/10c51f33-a21c-4a58-9c39-4c7eaa52e50a-provider-credentials\") pod \"10c51f33-a21c-4a58-9c39-4c7eaa52e50a\" (UID: \"10c51f33-a21c-4a58-9c39-4c7eaa52e50a\") " Oct 30 09:45:34.927689 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:34.927663 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10c51f33-a21c-4a58-9c39-4c7eaa52e50a-plugins-config-volume" pod "10c51f33-a21c-4a58-9c39-4c7eaa52e50a" (UID: "10c51f33-a21c-4a58-9c39-4c7eaa52e50a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:45:34.927837 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:34.927779 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10c51f33-a21c-4a58-9c39-4c7eaa52e50a-routing-overlay-volume" pod "10c51f33-a21c-4a58-9c39-4c7eaa52e50a" (UID: "10c51f33-a21c-4a58-9c39-4c7eaa52e50a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:45:34.928202 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:34.928185 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10c51f33-a21c-4a58-9c39-4c7eaa52e50a-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:45:34.928255 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:34.928205 4562 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/10c51f33-a21c-4a58-9c39-4c7eaa52e50a-routing-overlay-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:45:34.929787 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:34.929765 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10c51f33-a21c-4a58-9c39-4c7eaa52e50a-provider-credentials" pod "10c51f33-a21c-4a58-9c39-4c7eaa52e50a" (UID: "10c51f33-a21c-4a58-9c39-4c7eaa52e50a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:45:34.929855 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:34.929808 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10c51f33-a21c-4a58-9c39-4c7eaa52e50a-kube-api-access-49m98" pod "10c51f33-a21c-4a58-9c39-4c7eaa52e50a" (UID: "10c51f33-a21c-4a58-9c39-4c7eaa52e50a"). InnerVolumeSpecName "kube-api-access-49m98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:45:35.010622 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:35.010587 4562 generic.go:356] "Generic (PLEG): container finished" podID="10c51f33-a21c-4a58-9c39-4c7eaa52e50a" containerID="e8bf2fe9c9eb3080692ef2cc971bd9a97a33b7b6198fb8bbbb71b03f1b120770" exitCode=0 Oct 30 09:45:35.010787 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:35.010635 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-58cc4wdfp" Oct 30 09:45:35.010787 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:35.010676 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-58cc4wdfp" event={"ID":"10c51f33-a21c-4a58-9c39-4c7eaa52e50a","Type":"ContainerDied","Data":"e8bf2fe9c9eb3080692ef2cc971bd9a97a33b7b6198fb8bbbb71b03f1b120770"} Oct 30 09:45:35.010787 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:35.010733 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-58cc4wdfp" event={"ID":"10c51f33-a21c-4a58-9c39-4c7eaa52e50a","Type":"ContainerDied","Data":"2ba09fe452aa83680dafdf2aed41d9705cf3d0c45806869926d16323edcfcd35"} Oct 30 09:45:35.010787 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:35.010762 4562 scope.go:122] "RemoveContainer" containerID="e8bf2fe9c9eb3080692ef2cc971bd9a97a33b7b6198fb8bbbb71b03f1b120770" Oct 30 09:45:35.023807 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:35.023785 4562 scope.go:122] "RemoveContainer" containerID="e8bf2fe9c9eb3080692ef2cc971bd9a97a33b7b6198fb8bbbb71b03f1b120770" Oct 30 09:45:35.024096 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:45:35.024078 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8bf2fe9c9eb3080692ef2cc971bd9a97a33b7b6198fb8bbbb71b03f1b120770\": container with ID starting with e8bf2fe9c9eb3080692ef2cc971bd9a97a33b7b6198fb8bbbb71b03f1b120770 not found: ID does not exist" containerID="e8bf2fe9c9eb3080692ef2cc971bd9a97a33b7b6198fb8bbbb71b03f1b120770" Oct 30 09:45:35.024150 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:35.024103 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8bf2fe9c9eb3080692ef2cc971bd9a97a33b7b6198fb8bbbb71b03f1b120770"} err="failed to get container status \"e8bf2fe9c9eb3080692ef2cc971bd9a97a33b7b6198fb8bbbb71b03f1b120770\": rpc error: code = NotFound desc = could not find container \"e8bf2fe9c9eb3080692ef2cc971bd9a97a33b7b6198fb8bbbb71b03f1b120770\": container with ID starting with e8bf2fe9c9eb3080692ef2cc971bd9a97a33b7b6198fb8bbbb71b03f1b120770 not found: ID does not exist" Oct 30 09:45:35.029893 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:35.029874 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49m98\" (UniqueName: \"kubernetes.io/projected/10c51f33-a21c-4a58-9c39-4c7eaa52e50a-kube-api-access-49m98\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:45:35.029985 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:35.029894 4562 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/10c51f33-a21c-4a58-9c39-4c7eaa52e50a-provider-credentials\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:45:35.069553 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:35.069474 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-58cc4wdfp"] Oct 30 09:45:35.074796 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:35.074768 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-58cc4wdfp"] Oct 30 09:45:35.876619 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:35.876582 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10c51f33-a21c-4a58-9c39-4c7eaa52e50a" path="/var/lib/kubelet/pods/10c51f33-a21c-4a58-9c39-4c7eaa52e50a/volumes" Oct 30 09:45:52.269333 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.269246 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-657985w6h"] Oct 30 09:45:52.270215 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.270195 4562 cpu_manager.go:414] "containerMap: removing container" podUID="362d480a-6622-40a3-ab4f-8a85c54b3f06" containerName="cleanup" Oct 30 09:45:52.270282 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.270216 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="362d480a-6622-40a3-ab4f-8a85c54b3f06" containerName="cleanup" Oct 30 09:45:52.270282 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.270230 4562 cpu_manager.go:414] "containerMap: removing container" podUID="802f9a81-779c-49ff-8e04-283b72c0a908" containerName="cleanup" Oct 30 09:45:52.270282 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.270235 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="802f9a81-779c-49ff-8e04-283b72c0a908" containerName="cleanup" Oct 30 09:45:52.270282 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.270245 4562 cpu_manager.go:414] "containerMap: removing container" podUID="6da36b71-6952-4a6c-a029-ebefe69addb2" containerName="cleanup" Oct 30 09:45:52.270282 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.270250 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6da36b71-6952-4a6c-a029-ebefe69addb2" containerName="cleanup" Oct 30 09:45:52.270282 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.270259 4562 cpu_manager.go:414] "containerMap: removing container" podUID="802f9a81-779c-49ff-8e04-283b72c0a908" containerName="cleanup" Oct 30 09:45:52.270282 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.270266 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="802f9a81-779c-49ff-8e04-283b72c0a908" containerName="cleanup" Oct 30 09:45:52.270616 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.270284 4562 cpu_manager.go:414] "containerMap: removing container" podUID="802f9a81-779c-49ff-8e04-283b72c0a908" containerName="cleanup" Oct 30 09:45:52.270616 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.270292 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="802f9a81-779c-49ff-8e04-283b72c0a908" containerName="cleanup" Oct 30 09:45:52.270616 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.270310 4562 cpu_manager.go:414] "containerMap: removing container" podUID="db53ebe2-e3dd-4580-b79c-830feff939a1" containerName="cleanup" Oct 30 09:45:52.270616 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.270317 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db53ebe2-e3dd-4580-b79c-830feff939a1" containerName="cleanup" Oct 30 09:45:52.270616 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.270348 4562 cpu_manager.go:414] "containerMap: removing container" podUID="f25074f0-f8c9-4a6f-b899-4ae6a7ff4311" containerName="cleanup" Oct 30 09:45:52.270616 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.270355 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f25074f0-f8c9-4a6f-b899-4ae6a7ff4311" containerName="cleanup" Oct 30 09:45:52.270616 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.270366 4562 cpu_manager.go:414] "containerMap: removing container" podUID="10c51f33-a21c-4a58-9c39-4c7eaa52e50a" containerName="payload-processing" Oct 30 09:45:52.270616 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.270374 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10c51f33-a21c-4a58-9c39-4c7eaa52e50a" containerName="payload-processing" Oct 30 09:45:52.270616 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.270403 4562 cpu_manager.go:414] "containerMap: removing container" podUID="6da36b71-6952-4a6c-a029-ebefe69addb2" containerName="cleanup" Oct 30 09:45:52.270616 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.270410 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6da36b71-6952-4a6c-a029-ebefe69addb2" containerName="cleanup" Oct 30 09:45:52.270616 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.270422 4562 cpu_manager.go:414] "containerMap: removing container" podUID="83484f50-c325-4c2c-beac-16f69e38eb66" containerName="cleanup" Oct 30 09:45:52.270616 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.270429 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83484f50-c325-4c2c-beac-16f69e38eb66" containerName="cleanup" Oct 30 09:45:52.270616 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.270439 4562 cpu_manager.go:414] "containerMap: removing container" podUID="db53ebe2-e3dd-4580-b79c-830feff939a1" containerName="cleanup" Oct 30 09:45:52.270616 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.270446 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db53ebe2-e3dd-4580-b79c-830feff939a1" containerName="cleanup" Oct 30 09:45:52.270616 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.270457 4562 cpu_manager.go:414] "containerMap: removing container" podUID="db53ebe2-e3dd-4580-b79c-830feff939a1" containerName="cleanup" Oct 30 09:45:52.270616 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.270462 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db53ebe2-e3dd-4580-b79c-830feff939a1" containerName="cleanup" Oct 30 09:45:52.270616 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.270470 4562 cpu_manager.go:414] "containerMap: removing container" podUID="9b26e444-ab39-4a88-b18f-34c4846cb17b" containerName="cleanup" Oct 30 09:45:52.270616 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.270474 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b26e444-ab39-4a88-b18f-34c4846cb17b" containerName="cleanup" Oct 30 09:45:52.270616 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.270490 4562 cpu_manager.go:414] "containerMap: removing container" podUID="9b26e444-ab39-4a88-b18f-34c4846cb17b" containerName="cleanup" Oct 30 09:45:52.270616 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.270494 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b26e444-ab39-4a88-b18f-34c4846cb17b" containerName="cleanup" Oct 30 09:45:52.270616 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.270505 4562 cpu_manager.go:414] "containerMap: removing container" podUID="362d480a-6622-40a3-ab4f-8a85c54b3f06" containerName="cleanup" Oct 30 09:45:52.270616 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.270511 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="362d480a-6622-40a3-ab4f-8a85c54b3f06" containerName="cleanup" Oct 30 09:45:52.270616 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.270517 4562 cpu_manager.go:414] "containerMap: removing container" podUID="f25074f0-f8c9-4a6f-b899-4ae6a7ff4311" containerName="cleanup" Oct 30 09:45:52.270616 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.270521 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f25074f0-f8c9-4a6f-b899-4ae6a7ff4311" containerName="cleanup" Oct 30 09:45:52.270616 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.270535 4562 cpu_manager.go:414] "containerMap: removing container" podUID="83484f50-c325-4c2c-beac-16f69e38eb66" containerName="cleanup" Oct 30 09:45:52.270616 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.270539 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83484f50-c325-4c2c-beac-16f69e38eb66" containerName="cleanup" Oct 30 09:45:52.272269 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.270654 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="6da36b71-6952-4a6c-a029-ebefe69addb2" containerName="cleanup" Oct 30 09:45:52.272269 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.270662 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="f25074f0-f8c9-4a6f-b899-4ae6a7ff4311" containerName="cleanup" Oct 30 09:45:52.272269 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.270667 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="83484f50-c325-4c2c-beac-16f69e38eb66" containerName="cleanup" Oct 30 09:45:52.272269 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.270673 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="83484f50-c325-4c2c-beac-16f69e38eb66" containerName="cleanup" Oct 30 09:45:52.272269 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.270681 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="362d480a-6622-40a3-ab4f-8a85c54b3f06" containerName="cleanup" Oct 30 09:45:52.272269 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.270686 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="802f9a81-779c-49ff-8e04-283b72c0a908" containerName="cleanup" Oct 30 09:45:52.272269 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.270691 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="6da36b71-6952-4a6c-a029-ebefe69addb2" containerName="cleanup" Oct 30 09:45:52.272269 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.270697 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="802f9a81-779c-49ff-8e04-283b72c0a908" containerName="cleanup" Oct 30 09:45:52.272269 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.270708 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="802f9a81-779c-49ff-8e04-283b72c0a908" containerName="cleanup" Oct 30 09:45:52.272269 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.270717 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="db53ebe2-e3dd-4580-b79c-830feff939a1" containerName="cleanup" Oct 30 09:45:52.272269 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.270727 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="f25074f0-f8c9-4a6f-b899-4ae6a7ff4311" containerName="cleanup" Oct 30 09:45:52.272269 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.270735 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="10c51f33-a21c-4a58-9c39-4c7eaa52e50a" containerName="payload-processing" Oct 30 09:45:52.272269 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.270744 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="db53ebe2-e3dd-4580-b79c-830feff939a1" containerName="cleanup" Oct 30 09:45:52.272269 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.270749 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="db53ebe2-e3dd-4580-b79c-830feff939a1" containerName="cleanup" Oct 30 09:45:52.272269 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.270754 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b26e444-ab39-4a88-b18f-34c4846cb17b" containerName="cleanup" Oct 30 09:45:52.272269 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.270763 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b26e444-ab39-4a88-b18f-34c4846cb17b" containerName="cleanup" Oct 30 09:45:52.272269 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.270770 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="362d480a-6622-40a3-ab4f-8a85c54b3f06" containerName="cleanup" Oct 30 09:45:52.272269 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.270778 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b26e444-ab39-4a88-b18f-34c4846cb17b" containerName="cleanup" Oct 30 09:45:52.276687 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.276662 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-657985w6h" Oct 30 09:45:52.292671 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.292645 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-657985w6h"] Oct 30 09:45:52.300634 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.300609 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-88cbc7\"/\"routing-overlay\"" Oct 30 09:45:52.300764 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.300674 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-88cbc7\"/\"payload-processing-external-model-plugins-e2e-extmod-a-88cbc7\"" Oct 30 09:45:52.300764 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.300737 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-88cbc7\"/\"agc-cred-6d0ba7c4e69e\"" Oct 30 09:45:52.300764 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.300624 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-88cbc7\"/\"openshift-service-ca.crt\"" Oct 30 09:45:52.300764 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.300749 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-88cbc7\"/\"payload-processing-external-model-e2e--185456a2-dockercfg-2j9zz\"" Oct 30 09:45:52.304980 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.304959 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-88cbc7\"/\"kube-root-ca.crt\"" Oct 30 09:45:52.367833 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.367799 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e96cc238-9807-4fe4-8d22-eec180158515-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-657985w6h\" (UID: \"e96cc238-9807-4fe4-8d22-eec180158515\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-657985w6h" Oct 30 09:45:52.367982 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.367915 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e96cc238-9807-4fe4-8d22-eec180158515-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-657985w6h\" (UID: \"e96cc238-9807-4fe4-8d22-eec180158515\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-657985w6h" Oct 30 09:45:52.367982 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.367953 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7xhm\" (UniqueName: \"kubernetes.io/projected/e96cc238-9807-4fe4-8d22-eec180158515-kube-api-access-t7xhm\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-657985w6h\" (UID: \"e96cc238-9807-4fe4-8d22-eec180158515\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-657985w6h" Oct 30 09:45:52.368156 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.368141 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e96cc238-9807-4fe4-8d22-eec180158515-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-657985w6h\" (UID: \"e96cc238-9807-4fe4-8d22-eec180158515\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-657985w6h" Oct 30 09:45:52.480407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.476405 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7xhm\" (UniqueName: \"kubernetes.io/projected/e96cc238-9807-4fe4-8d22-eec180158515-kube-api-access-t7xhm\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-657985w6h\" (UID: \"e96cc238-9807-4fe4-8d22-eec180158515\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-657985w6h" Oct 30 09:45:52.480407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.476650 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e96cc238-9807-4fe4-8d22-eec180158515-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-657985w6h\" (UID: \"e96cc238-9807-4fe4-8d22-eec180158515\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-657985w6h" Oct 30 09:45:52.480407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.476772 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e96cc238-9807-4fe4-8d22-eec180158515-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-657985w6h\" (UID: \"e96cc238-9807-4fe4-8d22-eec180158515\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-657985w6h" Oct 30 09:45:52.480407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.476849 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e96cc238-9807-4fe4-8d22-eec180158515-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-657985w6h\" (UID: \"e96cc238-9807-4fe4-8d22-eec180158515\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-657985w6h" Oct 30 09:45:52.480407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.478036 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e96cc238-9807-4fe4-8d22-eec180158515-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-657985w6h\" (UID: \"e96cc238-9807-4fe4-8d22-eec180158515\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-657985w6h" Oct 30 09:45:52.480407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.478935 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e96cc238-9807-4fe4-8d22-eec180158515-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-657985w6h\" (UID: \"e96cc238-9807-4fe4-8d22-eec180158515\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-657985w6h" Oct 30 09:45:52.488400 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.483112 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e96cc238-9807-4fe4-8d22-eec180158515-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-657985w6h\" (UID: \"e96cc238-9807-4fe4-8d22-eec180158515\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-657985w6h" Oct 30 09:45:52.516933 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.514022 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7xhm\" (UniqueName: \"kubernetes.io/projected/e96cc238-9807-4fe4-8d22-eec180158515-kube-api-access-t7xhm\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-657985w6h\" (UID: \"e96cc238-9807-4fe4-8d22-eec180158515\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-657985w6h" Oct 30 09:45:52.595039 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.594949 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-657985w6h" Oct 30 09:45:52.901241 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:45:52.901209 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode96cc238_9807_4fe4_8d22_eec180158515.slice/crio-f461cdfff824f3ba3ac57894c20de2fd4d8169bb30e45459df2ef63941764a7e WatchSource:0}: Error finding container f461cdfff824f3ba3ac57894c20de2fd4d8169bb30e45459df2ef63941764a7e: Status 404 returned error can't find the container with id f461cdfff824f3ba3ac57894c20de2fd4d8169bb30e45459df2ef63941764a7e Oct 30 09:45:52.903131 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:52.903108 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-657985w6h"] Oct 30 09:45:53.281632 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:53.281593 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-657985w6h" event={"ID":"e96cc238-9807-4fe4-8d22-eec180158515","Type":"ContainerStarted","Data":"d02ab09f3aef996db5b0ff9f4e7f8c83bbb44dc8abe1ef49efc8f8b71c9092f3"} Oct 30 09:45:53.282293 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:53.282192 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-657985w6h" event={"ID":"e96cc238-9807-4fe4-8d22-eec180158515","Type":"ContainerStarted","Data":"f461cdfff824f3ba3ac57894c20de2fd4d8169bb30e45459df2ef63941764a7e"} Oct 30 09:45:53.283564 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:53.283533 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-657985w6h" Oct 30 09:45:53.334674 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:45:53.334610 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-657985w6h" podStartSLOduration=1.334590534 podStartE2EDuration="1.334590534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:45:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:45:53.333817308 +0000 UTC m=+3113.595378316" watchObservedRunningTime="2026-10-30 09:45:53.334590534 +0000 UTC m=+3113.596151524" Oct 30 09:45:54.430699 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:45:54.430660 4562 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-88cbc7/agc-cred-6d0ba7c4e69e: secret "agc-cred-6d0ba7c4e69e" not found Oct 30 09:45:54.430699 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:45:54.430693 4562 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-657985w6h: secret "agc-cred-6d0ba7c4e69e" not found Oct 30 09:45:54.431326 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:45:54.430761 4562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e96cc238-9807-4fe4-8d22-eec180158515-provider-credentials podName:e96cc238-9807-4fe4-8d22-eec180158515 nodeName:}" failed. No retries permitted until 2026-10-30 09:45:54.930744684 +0000 UTC m=+3115.192305667 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e96cc238-9807-4fe4-8d22-eec180158515-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-88cbc7-657985w6h" (UID: "e96cc238-9807-4fe4-8d22-eec180158515") : secret "agc-cred-6d0ba7c4e69e" not found Oct 30 09:45:54.937969 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:45:54.937930 4562 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-88cbc7/agc-cred-6d0ba7c4e69e: secret "agc-cred-6d0ba7c4e69e" not found Oct 30 09:45:54.937969 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:45:54.937964 4562 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-657985w6h: secret "agc-cred-6d0ba7c4e69e" not found Oct 30 09:45:54.938192 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:45:54.938037 4562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e96cc238-9807-4fe4-8d22-eec180158515-provider-credentials podName:e96cc238-9807-4fe4-8d22-eec180158515 nodeName:}" failed. No retries permitted until 2026-10-30 09:45:55.938014732 +0000 UTC m=+3116.199575721 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e96cc238-9807-4fe4-8d22-eec180158515-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-88cbc7-657985w6h" (UID: "e96cc238-9807-4fe4-8d22-eec180158515") : secret "agc-cred-6d0ba7c4e69e" not found Oct 30 09:45:55.953277 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:45:55.953246 4562 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-88cbc7/agc-cred-6d0ba7c4e69e: secret "agc-cred-6d0ba7c4e69e" not found Oct 30 09:45:55.953277 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:45:55.953276 4562 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-657985w6h: secret "agc-cred-6d0ba7c4e69e" not found Oct 30 09:45:55.953831 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:45:55.953361 4562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e96cc238-9807-4fe4-8d22-eec180158515-provider-credentials podName:e96cc238-9807-4fe4-8d22-eec180158515 nodeName:}" failed. No retries permitted until 2026-10-30 09:45:57.953338931 +0000 UTC m=+3118.214899910 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e96cc238-9807-4fe4-8d22-eec180158515-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-88cbc7-657985w6h" (UID: "e96cc238-9807-4fe4-8d22-eec180158515") : secret "agc-cred-6d0ba7c4e69e" not found Oct 30 09:45:57.984057 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:45:57.984023 4562 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-88cbc7/agc-cred-6d0ba7c4e69e: secret "agc-cred-6d0ba7c4e69e" not found Oct 30 09:45:57.984057 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:45:57.984054 4562 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-657985w6h: secret "agc-cred-6d0ba7c4e69e" not found Oct 30 09:45:57.984649 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:45:57.984120 4562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e96cc238-9807-4fe4-8d22-eec180158515-provider-credentials podName:e96cc238-9807-4fe4-8d22-eec180158515 nodeName:}" failed. No retries permitted until 2026-10-30 09:46:01.984100379 +0000 UTC m=+3122.245661347 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e96cc238-9807-4fe4-8d22-eec180158515-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-88cbc7-657985w6h" (UID: "e96cc238-9807-4fe4-8d22-eec180158515") : secret "agc-cred-6d0ba7c4e69e" not found Oct 30 09:45:58.600950 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:45:58.600912 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:46:02.048840 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:46:02.048807 4562 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-88cbc7/agc-cred-6d0ba7c4e69e: secret "agc-cred-6d0ba7c4e69e" not found Oct 30 09:46:02.048840 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:46:02.048835 4562 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-657985w6h: secret "agc-cred-6d0ba7c4e69e" not found Oct 30 09:46:02.049261 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:46:02.048897 4562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e96cc238-9807-4fe4-8d22-eec180158515-provider-credentials podName:e96cc238-9807-4fe4-8d22-eec180158515 nodeName:}" failed. No retries permitted until 2026-10-30 09:46:10.048881604 +0000 UTC m=+3130.310442570 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e96cc238-9807-4fe4-8d22-eec180158515-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-88cbc7-657985w6h" (UID: "e96cc238-9807-4fe4-8d22-eec180158515") : secret "agc-cred-6d0ba7c4e69e" not found Oct 30 09:46:04.293271 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:04.293237 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-657985w6h" Oct 30 09:46:10.143331 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:46:10.143296 4562 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-88cbc7/agc-cred-6d0ba7c4e69e: secret "agc-cred-6d0ba7c4e69e" not found Oct 30 09:46:10.143331 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:46:10.143324 4562 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-657985w6h: secret "agc-cred-6d0ba7c4e69e" not found Oct 30 09:46:10.143928 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:46:10.143399 4562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e96cc238-9807-4fe4-8d22-eec180158515-provider-credentials podName:e96cc238-9807-4fe4-8d22-eec180158515 nodeName:}" failed. No retries permitted until 2026-10-30 09:46:26.143369138 +0000 UTC m=+3146.404930104 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e96cc238-9807-4fe4-8d22-eec180158515-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-88cbc7-657985w6h" (UID: "e96cc238-9807-4fe4-8d22-eec180158515") : secret "agc-cred-6d0ba7c4e69e" not found Oct 30 09:46:10.758498 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:10.758463 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b8mzp"] Oct 30 09:46:10.759256 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:10.759241 4562 cpu_manager.go:414] "containerMap: removing container" podUID="83484f50-c325-4c2c-beac-16f69e38eb66" containerName="cleanup" Oct 30 09:46:10.759306 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:10.759258 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83484f50-c325-4c2c-beac-16f69e38eb66" containerName="cleanup" Oct 30 09:46:10.759306 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:10.759278 4562 cpu_manager.go:414] "containerMap: removing container" podUID="9b26e444-ab39-4a88-b18f-34c4846cb17b" containerName="cleanup" Oct 30 09:46:10.759306 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:10.759284 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b26e444-ab39-4a88-b18f-34c4846cb17b" containerName="cleanup" Oct 30 09:46:10.759306 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:10.759294 4562 cpu_manager.go:414] "containerMap: removing container" podUID="f25074f0-f8c9-4a6f-b899-4ae6a7ff4311" containerName="cleanup" Oct 30 09:46:10.759306 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:10.759299 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f25074f0-f8c9-4a6f-b899-4ae6a7ff4311" containerName="cleanup" Oct 30 09:46:10.759488 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:10.759320 4562 cpu_manager.go:414] "containerMap: removing container" podUID="362d480a-6622-40a3-ab4f-8a85c54b3f06" containerName="cleanup" Oct 30 09:46:10.759488 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:10.759325 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="362d480a-6622-40a3-ab4f-8a85c54b3f06" containerName="cleanup" Oct 30 09:46:10.759488 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:10.759334 4562 cpu_manager.go:414] "containerMap: removing container" podUID="6da36b71-6952-4a6c-a029-ebefe69addb2" containerName="cleanup" Oct 30 09:46:10.759488 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:10.759339 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6da36b71-6952-4a6c-a029-ebefe69addb2" containerName="cleanup" Oct 30 09:46:10.759488 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:10.759479 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="f25074f0-f8c9-4a6f-b899-4ae6a7ff4311" containerName="cleanup" Oct 30 09:46:10.759488 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:10.759487 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="83484f50-c325-4c2c-beac-16f69e38eb66" containerName="cleanup" Oct 30 09:46:10.759678 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:10.759494 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="362d480a-6622-40a3-ab4f-8a85c54b3f06" containerName="cleanup" Oct 30 09:46:10.759678 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:10.759500 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="6da36b71-6952-4a6c-a029-ebefe69addb2" containerName="cleanup" Oct 30 09:46:10.765565 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:10.765544 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b8mzp" Oct 30 09:46:10.786895 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:10.786860 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b8mzp"] Oct 30 09:46:10.855068 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:10.855024 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ef70489-cb47-4bd5-844b-72e81331b648-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-b8mzp\" (UID: \"5ef70489-cb47-4bd5-844b-72e81331b648\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b8mzp" Oct 30 09:46:10.855257 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:10.855091 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ef70489-cb47-4bd5-844b-72e81331b648-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-b8mzp\" (UID: \"5ef70489-cb47-4bd5-844b-72e81331b648\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b8mzp" Oct 30 09:46:10.855257 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:10.855180 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ef70489-cb47-4bd5-844b-72e81331b648-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-b8mzp\" (UID: \"5ef70489-cb47-4bd5-844b-72e81331b648\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b8mzp" Oct 30 09:46:10.855257 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:10.855254 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfmt2\" (UniqueName: \"kubernetes.io/projected/5ef70489-cb47-4bd5-844b-72e81331b648-kube-api-access-wfmt2\") pod \"maas-default-gateway-openshift-default-77f556d647-b8mzp\" (UID: \"5ef70489-cb47-4bd5-844b-72e81331b648\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b8mzp" Oct 30 09:46:10.855426 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:10.855375 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef70489-cb47-4bd5-844b-72e81331b648-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-b8mzp\" (UID: \"5ef70489-cb47-4bd5-844b-72e81331b648\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b8mzp" Oct 30 09:46:10.855502 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:10.855487 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ef70489-cb47-4bd5-844b-72e81331b648-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-b8mzp\" (UID: \"5ef70489-cb47-4bd5-844b-72e81331b648\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b8mzp" Oct 30 09:46:10.855551 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:10.855516 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef70489-cb47-4bd5-844b-72e81331b648-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-b8mzp\" (UID: \"5ef70489-cb47-4bd5-844b-72e81331b648\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b8mzp" Oct 30 09:46:10.855603 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:10.855590 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ef70489-cb47-4bd5-844b-72e81331b648-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-b8mzp\" (UID: \"5ef70489-cb47-4bd5-844b-72e81331b648\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b8mzp" Oct 30 09:46:10.855658 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:10.855646 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ef70489-cb47-4bd5-844b-72e81331b648-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-b8mzp\" (UID: \"5ef70489-cb47-4bd5-844b-72e81331b648\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b8mzp" Oct 30 09:46:10.956774 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:10.956738 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef70489-cb47-4bd5-844b-72e81331b648-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-b8mzp\" (UID: \"5ef70489-cb47-4bd5-844b-72e81331b648\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b8mzp" Oct 30 09:46:10.956976 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:10.956800 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ef70489-cb47-4bd5-844b-72e81331b648-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-b8mzp\" (UID: \"5ef70489-cb47-4bd5-844b-72e81331b648\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b8mzp" Oct 30 09:46:10.956976 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:10.956820 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef70489-cb47-4bd5-844b-72e81331b648-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-b8mzp\" (UID: \"5ef70489-cb47-4bd5-844b-72e81331b648\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b8mzp" Oct 30 09:46:10.956976 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:10.956869 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ef70489-cb47-4bd5-844b-72e81331b648-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-b8mzp\" (UID: \"5ef70489-cb47-4bd5-844b-72e81331b648\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b8mzp" Oct 30 09:46:10.956976 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:10.956918 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ef70489-cb47-4bd5-844b-72e81331b648-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-b8mzp\" (UID: \"5ef70489-cb47-4bd5-844b-72e81331b648\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b8mzp" Oct 30 09:46:10.956976 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:10.956952 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ef70489-cb47-4bd5-844b-72e81331b648-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-b8mzp\" (UID: \"5ef70489-cb47-4bd5-844b-72e81331b648\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b8mzp" Oct 30 09:46:10.957666 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:10.956987 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ef70489-cb47-4bd5-844b-72e81331b648-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-b8mzp\" (UID: \"5ef70489-cb47-4bd5-844b-72e81331b648\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b8mzp" Oct 30 09:46:10.957666 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:10.957014 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ef70489-cb47-4bd5-844b-72e81331b648-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-b8mzp\" (UID: \"5ef70489-cb47-4bd5-844b-72e81331b648\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b8mzp" Oct 30 09:46:10.957666 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:10.957082 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfmt2\" (UniqueName: \"kubernetes.io/projected/5ef70489-cb47-4bd5-844b-72e81331b648-kube-api-access-wfmt2\") pod \"maas-default-gateway-openshift-default-77f556d647-b8mzp\" (UID: \"5ef70489-cb47-4bd5-844b-72e81331b648\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b8mzp" Oct 30 09:46:10.957666 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:10.957418 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ef70489-cb47-4bd5-844b-72e81331b648-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-b8mzp\" (UID: \"5ef70489-cb47-4bd5-844b-72e81331b648\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b8mzp" Oct 30 09:46:10.957666 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:10.957418 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef70489-cb47-4bd5-844b-72e81331b648-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-b8mzp\" (UID: \"5ef70489-cb47-4bd5-844b-72e81331b648\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b8mzp" Oct 30 09:46:10.957666 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:10.957548 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ef70489-cb47-4bd5-844b-72e81331b648-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-b8mzp\" (UID: \"5ef70489-cb47-4bd5-844b-72e81331b648\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b8mzp" Oct 30 09:46:10.957666 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:10.957575 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef70489-cb47-4bd5-844b-72e81331b648-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-b8mzp\" (UID: \"5ef70489-cb47-4bd5-844b-72e81331b648\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b8mzp" Oct 30 09:46:10.958032 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:10.957768 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ef70489-cb47-4bd5-844b-72e81331b648-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-b8mzp\" (UID: \"5ef70489-cb47-4bd5-844b-72e81331b648\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b8mzp" Oct 30 09:46:10.959676 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:10.959646 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ef70489-cb47-4bd5-844b-72e81331b648-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-b8mzp\" (UID: \"5ef70489-cb47-4bd5-844b-72e81331b648\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b8mzp" Oct 30 09:46:10.959827 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:10.959808 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ef70489-cb47-4bd5-844b-72e81331b648-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-b8mzp\" (UID: \"5ef70489-cb47-4bd5-844b-72e81331b648\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b8mzp" Oct 30 09:46:10.976874 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:10.976847 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ef70489-cb47-4bd5-844b-72e81331b648-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-b8mzp\" (UID: \"5ef70489-cb47-4bd5-844b-72e81331b648\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b8mzp" Oct 30 09:46:10.976993 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:10.976915 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfmt2\" (UniqueName: \"kubernetes.io/projected/5ef70489-cb47-4bd5-844b-72e81331b648-kube-api-access-wfmt2\") pod \"maas-default-gateway-openshift-default-77f556d647-b8mzp\" (UID: \"5ef70489-cb47-4bd5-844b-72e81331b648\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b8mzp" Oct 30 09:46:11.080869 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:11.080781 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b8mzp" Oct 30 09:46:11.403977 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:46:11.403948 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ef70489_cb47_4bd5_844b_72e81331b648.slice/crio-2d342b1c8fdce455ead6ff1ca1f714a6ae561b20a287b7dc3520f540b6313622 WatchSource:0}: Error finding container 2d342b1c8fdce455ead6ff1ca1f714a6ae561b20a287b7dc3520f540b6313622: Status 404 returned error can't find the container with id 2d342b1c8fdce455ead6ff1ca1f714a6ae561b20a287b7dc3520f540b6313622 Oct 30 09:46:11.405973 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:11.405944 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b8mzp"] Oct 30 09:46:11.406878 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:11.406850 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:46:11.406949 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:11.406911 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:46:11.406949 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:11.406942 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:46:11.494588 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:11.491939 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b8mzp" event={"ID":"5ef70489-cb47-4bd5-844b-72e81331b648","Type":"ContainerStarted","Data":"2d342b1c8fdce455ead6ff1ca1f714a6ae561b20a287b7dc3520f540b6313622"} Oct 30 09:46:12.503527 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:12.503479 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b8mzp" event={"ID":"5ef70489-cb47-4bd5-844b-72e81331b648","Type":"ContainerStarted","Data":"fd08ce487619a41ccfa624e059fe2da212bf1be0fe10b9dec9f9941716b2ef8c"} Oct 30 09:46:12.571001 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:12.570938 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b8mzp" podStartSLOduration=2.570921554 podStartE2EDuration="2.570921554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:46:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:46:12.564712172 +0000 UTC m=+3132.826273161" watchObservedRunningTime="2026-10-30 09:46:12.570921554 +0000 UTC m=+3132.832482542" Oct 30 09:46:13.081770 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:13.081731 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b8mzp" Oct 30 09:46:13.083953 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:13.083923 4562 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-b8mzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body= Oct 30 09:46:13.084101 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:13.083975 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b8mzp" podUID="5ef70489-cb47-4bd5-844b-72e81331b648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" Oct 30 09:46:14.081408 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:14.081068 4562 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-b8mzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body= Oct 30 09:46:14.081408 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:14.081128 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b8mzp" podUID="5ef70489-cb47-4bd5-844b-72e81331b648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" Oct 30 09:46:15.081586 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:15.081543 4562 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-b8mzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body= Oct 30 09:46:15.082072 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:15.081597 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b8mzp" podUID="5ef70489-cb47-4bd5-844b-72e81331b648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" Oct 30 09:46:16.081442 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:16.081404 4562 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-b8mzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body= Oct 30 09:46:16.081643 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:16.081467 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b8mzp" podUID="5ef70489-cb47-4bd5-844b-72e81331b648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" Oct 30 09:46:17.082232 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:17.082193 4562 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-b8mzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body= Oct 30 09:46:17.082692 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:17.082276 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b8mzp" podUID="5ef70489-cb47-4bd5-844b-72e81331b648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" Oct 30 09:46:18.081280 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:18.081243 4562 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-b8mzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body= Oct 30 09:46:18.081489 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:18.081316 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b8mzp" podUID="5ef70489-cb47-4bd5-844b-72e81331b648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" Oct 30 09:46:19.081333 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:19.081298 4562 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-b8mzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body= Oct 30 09:46:19.081850 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:19.081435 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b8mzp" podUID="5ef70489-cb47-4bd5-844b-72e81331b648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" Oct 30 09:46:20.301650 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:20.301618 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b8mzp" Oct 30 09:46:20.302800 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:20.302677 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b8mzp" Oct 30 09:46:20.304395 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:20.304351 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b8mzp" Oct 30 09:46:26.173934 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:46:26.173897 4562 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-88cbc7/agc-cred-6d0ba7c4e69e: secret "agc-cred-6d0ba7c4e69e" not found Oct 30 09:46:26.173934 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:46:26.173926 4562 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-657985w6h: secret "agc-cred-6d0ba7c4e69e" not found Oct 30 09:46:26.174369 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:46:26.173988 4562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e96cc238-9807-4fe4-8d22-eec180158515-provider-credentials podName:e96cc238-9807-4fe4-8d22-eec180158515 nodeName:}" failed. No retries permitted until 2026-10-30 09:46:58.17397366 +0000 UTC m=+3178.435534628 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e96cc238-9807-4fe4-8d22-eec180158515-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-88cbc7-657985w6h" (UID: "e96cc238-9807-4fe4-8d22-eec180158515") : secret "agc-cred-6d0ba7c4e69e" not found Oct 30 09:46:28.597643 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:46:28.597603 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:46:40.652607 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:40.652570 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-f9bbfbmql"] Oct 30 09:46:40.656862 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:40.656829 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-f9bbfbmql" Oct 30 09:46:40.677052 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:40.677011 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/613d2758-5e8b-46a6-94a0-a81ee59f56a0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-f9bbfbmql\" (UID: \"613d2758-5e8b-46a6-94a0-a81ee59f56a0\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-f9bbfbmql" Oct 30 09:46:40.677217 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:40.677184 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd4nj\" (UniqueName: \"kubernetes.io/projected/613d2758-5e8b-46a6-94a0-a81ee59f56a0-kube-api-access-gd4nj\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-f9bbfbmql\" (UID: \"613d2758-5e8b-46a6-94a0-a81ee59f56a0\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-f9bbfbmql" Oct 30 09:46:40.677289 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:40.677263 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/613d2758-5e8b-46a6-94a0-a81ee59f56a0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-f9bbfbmql\" (UID: \"613d2758-5e8b-46a6-94a0-a81ee59f56a0\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-f9bbfbmql" Oct 30 09:46:40.677393 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:40.677360 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/613d2758-5e8b-46a6-94a0-a81ee59f56a0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-f9bbfbmql\" (UID: \"613d2758-5e8b-46a6-94a0-a81ee59f56a0\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-f9bbfbmql" Oct 30 09:46:40.680657 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:40.680625 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-f9bbfbmql"] Oct 30 09:46:40.685132 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:40.685104 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-881f83\"/\"payload-processing-external-model-e2e--d06f4d1b-dockercfg-nr74x\"" Oct 30 09:46:40.685316 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:40.685138 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-881f83\"/\"openshift-service-ca.crt\"" Oct 30 09:46:40.685316 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:40.685141 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-881f83\"/\"agc-cred-cf9bd9a0dc6b\"" Oct 30 09:46:40.685504 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:40.685156 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-881f83\"/\"kube-root-ca.crt\"" Oct 30 09:46:40.688411 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:40.688372 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-881f83\"/\"routing-overlay\"" Oct 30 09:46:40.688545 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:40.688378 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-881f83\"/\"payload-processing-external-model-plugins-e2e-extmod-a-881f83\"" Oct 30 09:46:40.778340 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:40.778310 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/613d2758-5e8b-46a6-94a0-a81ee59f56a0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-f9bbfbmql\" (UID: \"613d2758-5e8b-46a6-94a0-a81ee59f56a0\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-f9bbfbmql" Oct 30 09:46:40.778539 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:40.778450 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd4nj\" (UniqueName: \"kubernetes.io/projected/613d2758-5e8b-46a6-94a0-a81ee59f56a0-kube-api-access-gd4nj\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-f9bbfbmql\" (UID: \"613d2758-5e8b-46a6-94a0-a81ee59f56a0\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-f9bbfbmql" Oct 30 09:46:40.778539 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:40.778517 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/613d2758-5e8b-46a6-94a0-a81ee59f56a0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-f9bbfbmql\" (UID: \"613d2758-5e8b-46a6-94a0-a81ee59f56a0\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-f9bbfbmql" Oct 30 09:46:40.778637 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:40.778597 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/613d2758-5e8b-46a6-94a0-a81ee59f56a0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-f9bbfbmql\" (UID: \"613d2758-5e8b-46a6-94a0-a81ee59f56a0\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-f9bbfbmql" Oct 30 09:46:40.779138 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:40.779112 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/613d2758-5e8b-46a6-94a0-a81ee59f56a0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-f9bbfbmql\" (UID: \"613d2758-5e8b-46a6-94a0-a81ee59f56a0\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-f9bbfbmql" Oct 30 09:46:40.779262 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:40.779138 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/613d2758-5e8b-46a6-94a0-a81ee59f56a0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-f9bbfbmql\" (UID: \"613d2758-5e8b-46a6-94a0-a81ee59f56a0\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-f9bbfbmql" Oct 30 09:46:40.781196 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:40.781173 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/613d2758-5e8b-46a6-94a0-a81ee59f56a0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-f9bbfbmql\" (UID: \"613d2758-5e8b-46a6-94a0-a81ee59f56a0\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-f9bbfbmql" Oct 30 09:46:40.797840 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:40.797809 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd4nj\" (UniqueName: \"kubernetes.io/projected/613d2758-5e8b-46a6-94a0-a81ee59f56a0-kube-api-access-gd4nj\") pod \"payload-processing-external-model-e2e-extmod-a-881f83-f9bbfbmql\" (UID: \"613d2758-5e8b-46a6-94a0-a81ee59f56a0\") " pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-f9bbfbmql" Oct 30 09:46:40.979565 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:40.979534 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-f9bbfbmql" Oct 30 09:46:41.296020 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:46:41.295992 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod613d2758_5e8b_46a6_94a0_a81ee59f56a0.slice/crio-31dc86a34064db22128edc655f518b4668587428404d17234cdd3b19988bf6f6 WatchSource:0}: Error finding container 31dc86a34064db22128edc655f518b4668587428404d17234cdd3b19988bf6f6: Status 404 returned error can't find the container with id 31dc86a34064db22128edc655f518b4668587428404d17234cdd3b19988bf6f6 Oct 30 09:46:41.297493 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:41.297467 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-f9bbfbmql"] Oct 30 09:46:41.762582 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:41.762545 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-f9bbfbmql" event={"ID":"613d2758-5e8b-46a6-94a0-a81ee59f56a0","Type":"ContainerStarted","Data":"20ad003d3c89bb25631a899828f6c256003ccba1efb3da0b98d0286a05631ddf"} Oct 30 09:46:41.762992 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:41.762599 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-f9bbfbmql" event={"ID":"613d2758-5e8b-46a6-94a0-a81ee59f56a0","Type":"ContainerStarted","Data":"31dc86a34064db22128edc655f518b4668587428404d17234cdd3b19988bf6f6"} Oct 30 09:46:41.762992 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:41.762673 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-f9bbfbmql" Oct 30 09:46:41.808364 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:41.808301 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-f9bbfbmql" podStartSLOduration=1.808288055 podStartE2EDuration="1.808288055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:46:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:46:41.804731763 +0000 UTC m=+3162.066292751" watchObservedRunningTime="2026-10-30 09:46:41.808288055 +0000 UTC m=+3162.069849069" Oct 30 09:46:52.779546 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:52.779512 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-f9bbfbmql" Oct 30 09:46:52.886859 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:52.886823 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-f9bbfbmql"] Oct 30 09:46:52.887265 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:46:52.887237 4562 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-881f83/agc-cred-cf9bd9a0dc6b: secret "agc-cred-cf9bd9a0dc6b" not found Oct 30 09:46:52.887265 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:46:52.887265 4562 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-f9bbfbmql: secret "agc-cred-cf9bd9a0dc6b" not found Oct 30 09:46:52.887496 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:46:52.887318 4562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/613d2758-5e8b-46a6-94a0-a81ee59f56a0-provider-credentials podName:613d2758-5e8b-46a6-94a0-a81ee59f56a0 nodeName:}" failed. No retries permitted until 2026-10-30 09:46:53.387301679 +0000 UTC m=+3173.648862663 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/613d2758-5e8b-46a6-94a0-a81ee59f56a0-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-881f83-f9bbfbmql" (UID: "613d2758-5e8b-46a6-94a0-a81ee59f56a0") : secret "agc-cred-cf9bd9a0dc6b" not found Oct 30 09:46:53.395883 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:46:53.395843 4562 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-881f83/agc-cred-cf9bd9a0dc6b: secret "agc-cred-cf9bd9a0dc6b" not found Oct 30 09:46:53.395883 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:46:53.395881 4562 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-f9bbfbmql: secret "agc-cred-cf9bd9a0dc6b" not found Oct 30 09:46:53.396175 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:46:53.395980 4562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/613d2758-5e8b-46a6-94a0-a81ee59f56a0-provider-credentials podName:613d2758-5e8b-46a6-94a0-a81ee59f56a0 nodeName:}" failed. No retries permitted until 2026-10-30 09:46:54.395959918 +0000 UTC m=+3174.657520897 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/613d2758-5e8b-46a6-94a0-a81ee59f56a0-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-881f83-f9bbfbmql" (UID: "613d2758-5e8b-46a6-94a0-a81ee59f56a0") : secret "agc-cred-cf9bd9a0dc6b" not found Oct 30 09:46:54.415167 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:46:54.415133 4562 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-881f83/agc-cred-cf9bd9a0dc6b: secret "agc-cred-cf9bd9a0dc6b" not found Oct 30 09:46:54.415167 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:46:54.415161 4562 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-f9bbfbmql: secret "agc-cred-cf9bd9a0dc6b" not found Oct 30 09:46:54.415633 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:46:54.415213 4562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/613d2758-5e8b-46a6-94a0-a81ee59f56a0-provider-credentials podName:613d2758-5e8b-46a6-94a0-a81ee59f56a0 nodeName:}" failed. No retries permitted until 2026-10-30 09:46:56.415198546 +0000 UTC m=+3176.676759526 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/613d2758-5e8b-46a6-94a0-a81ee59f56a0-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-881f83-f9bbfbmql" (UID: "613d2758-5e8b-46a6-94a0-a81ee59f56a0") : secret "agc-cred-cf9bd9a0dc6b" not found Oct 30 09:46:56.455969 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:46:56.455933 4562 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-881f83/agc-cred-cf9bd9a0dc6b: secret "agc-cred-cf9bd9a0dc6b" not found Oct 30 09:46:56.455969 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:46:56.455967 4562 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-f9bbfbmql: secret "agc-cred-cf9bd9a0dc6b" not found Oct 30 09:46:56.456449 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:46:56.456031 4562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/613d2758-5e8b-46a6-94a0-a81ee59f56a0-provider-credentials podName:613d2758-5e8b-46a6-94a0-a81ee59f56a0 nodeName:}" failed. No retries permitted until 2026-10-30 09:47:00.45601 +0000 UTC m=+3180.717570981 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/613d2758-5e8b-46a6-94a0-a81ee59f56a0-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-881f83-f9bbfbmql" (UID: "613d2758-5e8b-46a6-94a0-a81ee59f56a0") : secret "agc-cred-cf9bd9a0dc6b" not found Oct 30 09:46:57.887972 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:57.887934 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-f9bbfbmql" podUID="613d2758-5e8b-46a6-94a0-a81ee59f56a0" containerName="payload-processing" containerID="cri-o://20ad003d3c89bb25631a899828f6c256003ccba1efb3da0b98d0286a05631ddf" gracePeriod=25 Oct 30 09:46:58.203314 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:46:58.200714 4562 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-88cbc7/agc-cred-6d0ba7c4e69e: secret "agc-cred-6d0ba7c4e69e" not found Oct 30 09:46:58.203314 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:46:58.200739 4562 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-657985w6h: secret "agc-cred-6d0ba7c4e69e" not found Oct 30 09:46:58.203314 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:46:58.200797 4562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e96cc238-9807-4fe4-8d22-eec180158515-provider-credentials podName:e96cc238-9807-4fe4-8d22-eec180158515 nodeName:}" failed. No retries permitted until 2026-10-30 09:48:02.20077736 +0000 UTC m=+3242.462338344 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e96cc238-9807-4fe4-8d22-eec180158515-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-88cbc7-657985w6h" (UID: "e96cc238-9807-4fe4-8d22-eec180158515") : secret "agc-cred-6d0ba7c4e69e" not found Oct 30 09:46:58.210432 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:58.210406 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-f9bbfbmql" Oct 30 09:46:58.301396 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:58.301355 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/613d2758-5e8b-46a6-94a0-a81ee59f56a0-provider-credentials\" (UniqueName: \"kubernetes.io/projected/613d2758-5e8b-46a6-94a0-a81ee59f56a0-provider-credentials\") pod \"613d2758-5e8b-46a6-94a0-a81ee59f56a0\" (UID: \"613d2758-5e8b-46a6-94a0-a81ee59f56a0\") " Oct 30 09:46:58.301596 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:58.301497 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/613d2758-5e8b-46a6-94a0-a81ee59f56a0-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/613d2758-5e8b-46a6-94a0-a81ee59f56a0-routing-overlay-volume\") pod \"613d2758-5e8b-46a6-94a0-a81ee59f56a0\" (UID: \"613d2758-5e8b-46a6-94a0-a81ee59f56a0\") " Oct 30 09:46:58.301596 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:58.301536 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/613d2758-5e8b-46a6-94a0-a81ee59f56a0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/613d2758-5e8b-46a6-94a0-a81ee59f56a0-plugins-config-volume\") pod \"613d2758-5e8b-46a6-94a0-a81ee59f56a0\" (UID: \"613d2758-5e8b-46a6-94a0-a81ee59f56a0\") " Oct 30 09:46:58.302214 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:58.301869 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/613d2758-5e8b-46a6-94a0-a81ee59f56a0-kube-api-access-gd4nj\" (UniqueName: \"kubernetes.io/projected/613d2758-5e8b-46a6-94a0-a81ee59f56a0-kube-api-access-gd4nj\") pod \"613d2758-5e8b-46a6-94a0-a81ee59f56a0\" (UID: \"613d2758-5e8b-46a6-94a0-a81ee59f56a0\") " Oct 30 09:46:58.302214 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:58.302096 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/613d2758-5e8b-46a6-94a0-a81ee59f56a0-routing-overlay-volume" pod "613d2758-5e8b-46a6-94a0-a81ee59f56a0" (UID: "613d2758-5e8b-46a6-94a0-a81ee59f56a0"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:46:58.302214 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:58.302205 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/613d2758-5e8b-46a6-94a0-a81ee59f56a0-plugins-config-volume" pod "613d2758-5e8b-46a6-94a0-a81ee59f56a0" (UID: "613d2758-5e8b-46a6-94a0-a81ee59f56a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:46:58.302711 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:58.302679 4562 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/613d2758-5e8b-46a6-94a0-a81ee59f56a0-routing-overlay-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:46:58.302711 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:58.302711 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/613d2758-5e8b-46a6-94a0-a81ee59f56a0-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:46:58.303831 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:58.303806 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/613d2758-5e8b-46a6-94a0-a81ee59f56a0-provider-credentials" pod "613d2758-5e8b-46a6-94a0-a81ee59f56a0" (UID: "613d2758-5e8b-46a6-94a0-a81ee59f56a0"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:46:58.304849 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:58.304821 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/613d2758-5e8b-46a6-94a0-a81ee59f56a0-kube-api-access-gd4nj" pod "613d2758-5e8b-46a6-94a0-a81ee59f56a0" (UID: "613d2758-5e8b-46a6-94a0-a81ee59f56a0"). InnerVolumeSpecName "kube-api-access-gd4nj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:46:58.428487 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:58.427437 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd4nj\" (UniqueName: \"kubernetes.io/projected/613d2758-5e8b-46a6-94a0-a81ee59f56a0-kube-api-access-gd4nj\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:46:58.428487 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:58.427475 4562 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/613d2758-5e8b-46a6-94a0-a81ee59f56a0-provider-credentials\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:46:58.596164 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:46:58.596123 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:46:58.912159 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:58.912124 4562 generic.go:356] "Generic (PLEG): container finished" podID="613d2758-5e8b-46a6-94a0-a81ee59f56a0" containerID="20ad003d3c89bb25631a899828f6c256003ccba1efb3da0b98d0286a05631ddf" exitCode=0 Oct 30 09:46:58.912661 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:58.912181 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-f9bbfbmql" Oct 30 09:46:58.912661 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:58.912185 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-f9bbfbmql" event={"ID":"613d2758-5e8b-46a6-94a0-a81ee59f56a0","Type":"ContainerDied","Data":"20ad003d3c89bb25631a899828f6c256003ccba1efb3da0b98d0286a05631ddf"} Oct 30 09:46:58.912661 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:58.912314 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-f9bbfbmql" event={"ID":"613d2758-5e8b-46a6-94a0-a81ee59f56a0","Type":"ContainerDied","Data":"31dc86a34064db22128edc655f518b4668587428404d17234cdd3b19988bf6f6"} Oct 30 09:46:58.912661 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:58.912347 4562 scope.go:122] "RemoveContainer" containerID="20ad003d3c89bb25631a899828f6c256003ccba1efb3da0b98d0286a05631ddf" Oct 30 09:46:58.927730 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:58.927699 4562 scope.go:122] "RemoveContainer" containerID="20ad003d3c89bb25631a899828f6c256003ccba1efb3da0b98d0286a05631ddf" Oct 30 09:46:58.928081 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:46:58.928057 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20ad003d3c89bb25631a899828f6c256003ccba1efb3da0b98d0286a05631ddf\": container with ID starting with 20ad003d3c89bb25631a899828f6c256003ccba1efb3da0b98d0286a05631ddf not found: ID does not exist" containerID="20ad003d3c89bb25631a899828f6c256003ccba1efb3da0b98d0286a05631ddf" Oct 30 09:46:58.928155 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:58.928091 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20ad003d3c89bb25631a899828f6c256003ccba1efb3da0b98d0286a05631ddf"} err="failed to get container status \"20ad003d3c89bb25631a899828f6c256003ccba1efb3da0b98d0286a05631ddf\": rpc error: code = NotFound desc = could not find container \"20ad003d3c89bb25631a899828f6c256003ccba1efb3da0b98d0286a05631ddf\": container with ID starting with 20ad003d3c89bb25631a899828f6c256003ccba1efb3da0b98d0286a05631ddf not found: ID does not exist" Oct 30 09:46:58.967205 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:58.967174 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-f9bbfbmql"] Oct 30 09:46:58.979811 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:58.979783 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-881f83/payload-processing-external-model-e2e-extmod-a-881f83-f9bbfbmql"] Oct 30 09:46:59.877152 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:46:59.877116 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="613d2758-5e8b-46a6-94a0-a81ee59f56a0" path="/var/lib/kubelet/pods/613d2758-5e8b-46a6-94a0-a81ee59f56a0/volumes" Oct 30 09:47:28.597931 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:47:28.597895 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:47:58.598250 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:47:58.598210 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:48:02.202833 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:48:02.202788 4562 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-88cbc7/agc-cred-6d0ba7c4e69e: secret "agc-cred-6d0ba7c4e69e" not found Oct 30 09:48:02.202833 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:48:02.202827 4562 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-657985w6h: secret "agc-cred-6d0ba7c4e69e" not found Oct 30 09:48:02.203272 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:48:02.202903 4562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e96cc238-9807-4fe4-8d22-eec180158515-provider-credentials podName:e96cc238-9807-4fe4-8d22-eec180158515 nodeName:}" failed. No retries permitted until 2026-10-30 09:50:04.202887831 +0000 UTC m=+3364.464448809 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e96cc238-9807-4fe4-8d22-eec180158515-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-88cbc7-657985w6h" (UID: "e96cc238-9807-4fe4-8d22-eec180158515") : secret "agc-cred-6d0ba7c4e69e" not found Oct 30 09:48:05.456628 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:05.456587 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-657985w6h"] Oct 30 09:48:10.457486 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:10.457423 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-657985w6h" podUID="e96cc238-9807-4fe4-8d22-eec180158515" containerName="payload-processing" containerID="cri-o://d02ab09f3aef996db5b0ff9f4e7f8c83bbb44dc8abe1ef49efc8f8b71c9092f3" gracePeriod=25 Oct 30 09:48:10.888305 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:10.888274 4562 generic.go:356] "Generic (PLEG): container finished" podID="e96cc238-9807-4fe4-8d22-eec180158515" containerID="d02ab09f3aef996db5b0ff9f4e7f8c83bbb44dc8abe1ef49efc8f8b71c9092f3" exitCode=0 Oct 30 09:48:10.888812 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:10.888791 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-657985w6h" event={"ID":"e96cc238-9807-4fe4-8d22-eec180158515","Type":"ContainerDied","Data":"d02ab09f3aef996db5b0ff9f4e7f8c83bbb44dc8abe1ef49efc8f8b71c9092f3"} Oct 30 09:48:10.928866 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:10.928836 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-657985w6h" Oct 30 09:48:11.051168 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:11.051136 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e96cc238-9807-4fe4-8d22-eec180158515-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e96cc238-9807-4fe4-8d22-eec180158515-provider-credentials\") pod \"e96cc238-9807-4fe4-8d22-eec180158515\" (UID: \"e96cc238-9807-4fe4-8d22-eec180158515\") " Oct 30 09:48:11.051349 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:11.051233 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e96cc238-9807-4fe4-8d22-eec180158515-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e96cc238-9807-4fe4-8d22-eec180158515-plugins-config-volume\") pod \"e96cc238-9807-4fe4-8d22-eec180158515\" (UID: \"e96cc238-9807-4fe4-8d22-eec180158515\") " Oct 30 09:48:11.051349 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:11.051277 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e96cc238-9807-4fe4-8d22-eec180158515-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e96cc238-9807-4fe4-8d22-eec180158515-routing-overlay-volume\") pod \"e96cc238-9807-4fe4-8d22-eec180158515\" (UID: \"e96cc238-9807-4fe4-8d22-eec180158515\") " Oct 30 09:48:11.051612 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:11.051584 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e96cc238-9807-4fe4-8d22-eec180158515-kube-api-access-t7xhm\" (UniqueName: \"kubernetes.io/projected/e96cc238-9807-4fe4-8d22-eec180158515-kube-api-access-t7xhm\") pod \"e96cc238-9807-4fe4-8d22-eec180158515\" (UID: \"e96cc238-9807-4fe4-8d22-eec180158515\") " Oct 30 09:48:11.051717 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:11.051686 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e96cc238-9807-4fe4-8d22-eec180158515-plugins-config-volume" pod "e96cc238-9807-4fe4-8d22-eec180158515" (UID: "e96cc238-9807-4fe4-8d22-eec180158515"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:48:11.051717 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:11.051696 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e96cc238-9807-4fe4-8d22-eec180158515-routing-overlay-volume" pod "e96cc238-9807-4fe4-8d22-eec180158515" (UID: "e96cc238-9807-4fe4-8d22-eec180158515"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:48:11.052168 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:11.052150 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e96cc238-9807-4fe4-8d22-eec180158515-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:48:11.052168 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:11.052170 4562 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e96cc238-9807-4fe4-8d22-eec180158515-routing-overlay-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:48:11.053807 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:11.053785 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e96cc238-9807-4fe4-8d22-eec180158515-kube-api-access-t7xhm" pod "e96cc238-9807-4fe4-8d22-eec180158515" (UID: "e96cc238-9807-4fe4-8d22-eec180158515"). InnerVolumeSpecName "kube-api-access-t7xhm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:48:11.053909 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:11.053812 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e96cc238-9807-4fe4-8d22-eec180158515-provider-credentials" pod "e96cc238-9807-4fe4-8d22-eec180158515" (UID: "e96cc238-9807-4fe4-8d22-eec180158515"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:48:11.153273 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:11.153191 4562 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e96cc238-9807-4fe4-8d22-eec180158515-provider-credentials\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:48:11.153273 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:11.153223 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7xhm\" (UniqueName: \"kubernetes.io/projected/e96cc238-9807-4fe4-8d22-eec180158515-kube-api-access-t7xhm\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:48:11.897108 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:11.897077 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-657985w6h" Oct 30 09:48:11.897546 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:11.897122 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-657985w6h" event={"ID":"e96cc238-9807-4fe4-8d22-eec180158515","Type":"ContainerDied","Data":"f461cdfff824f3ba3ac57894c20de2fd4d8169bb30e45459df2ef63941764a7e"} Oct 30 09:48:11.897546 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:11.897189 4562 scope.go:122] "RemoveContainer" containerID="d02ab09f3aef996db5b0ff9f4e7f8c83bbb44dc8abe1ef49efc8f8b71c9092f3" Oct 30 09:48:12.016420 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:12.007430 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-657985w6h"] Oct 30 09:48:12.055403 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:12.051940 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-657985w6h"] Oct 30 09:48:14.243402 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:14.239503 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e96cc238-9807-4fe4-8d22-eec180158515" path="/var/lib/kubelet/pods/e96cc238-9807-4fe4-8d22-eec180158515/volumes" Oct 30 09:48:14.741324 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:14.741289 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-59686ffd74-722wt"] Oct 30 09:48:14.742086 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:14.742072 4562 cpu_manager.go:414] "containerMap: removing container" podUID="613d2758-5e8b-46a6-94a0-a81ee59f56a0" containerName="payload-processing" Oct 30 09:48:14.742137 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:14.742088 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="613d2758-5e8b-46a6-94a0-a81ee59f56a0" containerName="payload-processing" Oct 30 09:48:14.742137 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:14.742106 4562 cpu_manager.go:414] "containerMap: removing container" podUID="e96cc238-9807-4fe4-8d22-eec180158515" containerName="payload-processing" Oct 30 09:48:14.742137 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:14.742111 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e96cc238-9807-4fe4-8d22-eec180158515" containerName="payload-processing" Oct 30 09:48:14.742239 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:14.742229 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="613d2758-5e8b-46a6-94a0-a81ee59f56a0" containerName="payload-processing" Oct 30 09:48:14.742274 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:14.742239 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="e96cc238-9807-4fe4-8d22-eec180158515" containerName="payload-processing" Oct 30 09:48:14.747255 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:14.747233 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-59686ffd74-722wt" Oct 30 09:48:14.762263 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:14.762230 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-59686ffd74-722wt"] Oct 30 09:48:14.770574 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:14.770551 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 30 09:48:14.770797 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:14.770772 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 30 09:48:14.770940 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:14.770786 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 30 09:48:14.771004 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:14.770844 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 30 09:48:14.778515 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:14.778484 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-ff81f0cb94c6\"" Oct 30 09:48:14.778607 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:14.778535 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-047b9f328507\"" Oct 30 09:48:14.778607 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:14.778535 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-dbd224b134a2\"" Oct 30 09:48:14.778712 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:14.778615 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-pwntl\"" Oct 30 09:48:14.778819 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:14.778791 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-908e572dbf63\"" Oct 30 09:48:14.797941 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:14.797903 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c130167-0158-4c0e-a55a-4aaa9a1a70e4-plugins-config-volume\") pod \"payload-processing-external-model-59686ffd74-722wt\" (UID: \"7c130167-0158-4c0e-a55a-4aaa9a1a70e4\") " pod="models-as-a-service/payload-processing-external-model-59686ffd74-722wt" Oct 30 09:48:14.798030 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:14.797961 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7c130167-0158-4c0e-a55a-4aaa9a1a70e4-provider-credentials\") pod \"payload-processing-external-model-59686ffd74-722wt\" (UID: \"7c130167-0158-4c0e-a55a-4aaa9a1a70e4\") " pod="models-as-a-service/payload-processing-external-model-59686ffd74-722wt" Oct 30 09:48:14.798262 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:14.798244 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7c130167-0158-4c0e-a55a-4aaa9a1a70e4-routing-overlay-volume\") pod \"payload-processing-external-model-59686ffd74-722wt\" (UID: \"7c130167-0158-4c0e-a55a-4aaa9a1a70e4\") " pod="models-as-a-service/payload-processing-external-model-59686ffd74-722wt" Oct 30 09:48:14.798338 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:14.798321 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hszx\" (UniqueName: \"kubernetes.io/projected/7c130167-0158-4c0e-a55a-4aaa9a1a70e4-kube-api-access-7hszx\") pod \"payload-processing-external-model-59686ffd74-722wt\" (UID: \"7c130167-0158-4c0e-a55a-4aaa9a1a70e4\") " pod="models-as-a-service/payload-processing-external-model-59686ffd74-722wt" Oct 30 09:48:14.899440 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:14.899406 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7c130167-0158-4c0e-a55a-4aaa9a1a70e4-routing-overlay-volume\") pod \"payload-processing-external-model-59686ffd74-722wt\" (UID: \"7c130167-0158-4c0e-a55a-4aaa9a1a70e4\") " pod="models-as-a-service/payload-processing-external-model-59686ffd74-722wt" Oct 30 09:48:14.899590 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:14.899475 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hszx\" (UniqueName: \"kubernetes.io/projected/7c130167-0158-4c0e-a55a-4aaa9a1a70e4-kube-api-access-7hszx\") pod \"payload-processing-external-model-59686ffd74-722wt\" (UID: \"7c130167-0158-4c0e-a55a-4aaa9a1a70e4\") " pod="models-as-a-service/payload-processing-external-model-59686ffd74-722wt" Oct 30 09:48:14.899590 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:14.899524 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c130167-0158-4c0e-a55a-4aaa9a1a70e4-plugins-config-volume\") pod \"payload-processing-external-model-59686ffd74-722wt\" (UID: \"7c130167-0158-4c0e-a55a-4aaa9a1a70e4\") " pod="models-as-a-service/payload-processing-external-model-59686ffd74-722wt" Oct 30 09:48:14.899590 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:14.899545 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7c130167-0158-4c0e-a55a-4aaa9a1a70e4-provider-credentials\") pod \"payload-processing-external-model-59686ffd74-722wt\" (UID: \"7c130167-0158-4c0e-a55a-4aaa9a1a70e4\") " pod="models-as-a-service/payload-processing-external-model-59686ffd74-722wt" Oct 30 09:48:14.900136 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:14.900111 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7c130167-0158-4c0e-a55a-4aaa9a1a70e4-routing-overlay-volume\") pod \"payload-processing-external-model-59686ffd74-722wt\" (UID: \"7c130167-0158-4c0e-a55a-4aaa9a1a70e4\") " pod="models-as-a-service/payload-processing-external-model-59686ffd74-722wt" Oct 30 09:48:14.900203 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:14.900185 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c130167-0158-4c0e-a55a-4aaa9a1a70e4-plugins-config-volume\") pod \"payload-processing-external-model-59686ffd74-722wt\" (UID: \"7c130167-0158-4c0e-a55a-4aaa9a1a70e4\") " pod="models-as-a-service/payload-processing-external-model-59686ffd74-722wt" Oct 30 09:48:14.902447 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:14.902424 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7c130167-0158-4c0e-a55a-4aaa9a1a70e4-provider-credentials\") pod \"payload-processing-external-model-59686ffd74-722wt\" (UID: \"7c130167-0158-4c0e-a55a-4aaa9a1a70e4\") " pod="models-as-a-service/payload-processing-external-model-59686ffd74-722wt" Oct 30 09:48:14.919578 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:14.919549 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hszx\" (UniqueName: \"kubernetes.io/projected/7c130167-0158-4c0e-a55a-4aaa9a1a70e4-kube-api-access-7hszx\") pod \"payload-processing-external-model-59686ffd74-722wt\" (UID: \"7c130167-0158-4c0e-a55a-4aaa9a1a70e4\") " pod="models-as-a-service/payload-processing-external-model-59686ffd74-722wt" Oct 30 09:48:15.063824 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:15.063745 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-59686ffd74-722wt" Oct 30 09:48:15.373854 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:48:15.373823 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c130167_0158_4c0e_a55a_4aaa9a1a70e4.slice/crio-3db1ad2171c598459fe614116c9f7ba627ed16aaf6b59ecaae93809cec864cbc WatchSource:0}: Error finding container 3db1ad2171c598459fe614116c9f7ba627ed16aaf6b59ecaae93809cec864cbc: Status 404 returned error can't find the container with id 3db1ad2171c598459fe614116c9f7ba627ed16aaf6b59ecaae93809cec864cbc Oct 30 09:48:15.375212 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:15.375191 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-59686ffd74-722wt"] Oct 30 09:48:16.176784 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:16.176745 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-59686ffd74-722wt" event={"ID":"7c130167-0158-4c0e-a55a-4aaa9a1a70e4","Type":"ContainerStarted","Data":"713673d3111aaad9cb1f45532af699081b23b80e699f2e325873e8d82eaa41f6"} Oct 30 09:48:16.176991 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:16.176813 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-59686ffd74-722wt" event={"ID":"7c130167-0158-4c0e-a55a-4aaa9a1a70e4","Type":"ContainerStarted","Data":"3db1ad2171c598459fe614116c9f7ba627ed16aaf6b59ecaae93809cec864cbc"} Oct 30 09:48:16.176991 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:16.176843 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-59686ffd74-722wt" Oct 30 09:48:16.241362 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:16.241308 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-59686ffd74-722wt" podStartSLOduration=2.241293238 podStartE2EDuration="2.241293238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:48:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:48:16.239783808 +0000 UTC m=+3256.501344803" watchObservedRunningTime="2026-10-30 09:48:16.241293238 +0000 UTC m=+3256.502854226" Oct 30 09:48:19.899163 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:19.899125 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-59686ffd74-722wt"] Oct 30 09:48:24.899933 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:24.899896 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-59686ffd74-722wt" podUID="7c130167-0158-4c0e-a55a-4aaa9a1a70e4" containerName="payload-processing" containerID="cri-o://713673d3111aaad9cb1f45532af699081b23b80e699f2e325873e8d82eaa41f6" gracePeriod=25 Oct 30 09:48:25.240645 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:25.240619 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-59686ffd74-722wt" Oct 30 09:48:25.402121 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:25.402087 4562 generic.go:356] "Generic (PLEG): container finished" podID="7c130167-0158-4c0e-a55a-4aaa9a1a70e4" containerID="713673d3111aaad9cb1f45532af699081b23b80e699f2e325873e8d82eaa41f6" exitCode=0 Oct 30 09:48:25.402300 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:25.402137 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-59686ffd74-722wt" Oct 30 09:48:25.402300 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:25.402166 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-59686ffd74-722wt" event={"ID":"7c130167-0158-4c0e-a55a-4aaa9a1a70e4","Type":"ContainerDied","Data":"713673d3111aaad9cb1f45532af699081b23b80e699f2e325873e8d82eaa41f6"} Oct 30 09:48:25.402300 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:25.402223 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-59686ffd74-722wt" event={"ID":"7c130167-0158-4c0e-a55a-4aaa9a1a70e4","Type":"ContainerDied","Data":"3db1ad2171c598459fe614116c9f7ba627ed16aaf6b59ecaae93809cec864cbc"} Oct 30 09:48:25.402300 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:25.402253 4562 scope.go:122] "RemoveContainer" containerID="713673d3111aaad9cb1f45532af699081b23b80e699f2e325873e8d82eaa41f6" Oct 30 09:48:25.412607 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:25.412582 4562 scope.go:122] "RemoveContainer" containerID="713673d3111aaad9cb1f45532af699081b23b80e699f2e325873e8d82eaa41f6" Oct 30 09:48:25.412875 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:48:25.412857 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"713673d3111aaad9cb1f45532af699081b23b80e699f2e325873e8d82eaa41f6\": container with ID starting with 713673d3111aaad9cb1f45532af699081b23b80e699f2e325873e8d82eaa41f6 not found: ID does not exist" containerID="713673d3111aaad9cb1f45532af699081b23b80e699f2e325873e8d82eaa41f6" Oct 30 09:48:25.412931 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:25.412885 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"713673d3111aaad9cb1f45532af699081b23b80e699f2e325873e8d82eaa41f6"} err="failed to get container status \"713673d3111aaad9cb1f45532af699081b23b80e699f2e325873e8d82eaa41f6\": rpc error: code = NotFound desc = could not find container \"713673d3111aaad9cb1f45532af699081b23b80e699f2e325873e8d82eaa41f6\": container with ID starting with 713673d3111aaad9cb1f45532af699081b23b80e699f2e325873e8d82eaa41f6 not found: ID does not exist" Oct 30 09:48:25.424795 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:25.424774 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c130167-0158-4c0e-a55a-4aaa9a1a70e4-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7c130167-0158-4c0e-a55a-4aaa9a1a70e4-provider-credentials\") pod \"7c130167-0158-4c0e-a55a-4aaa9a1a70e4\" (UID: \"7c130167-0158-4c0e-a55a-4aaa9a1a70e4\") " Oct 30 09:48:25.424947 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:25.424932 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c130167-0158-4c0e-a55a-4aaa9a1a70e4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7c130167-0158-4c0e-a55a-4aaa9a1a70e4-routing-overlay-volume\") pod \"7c130167-0158-4c0e-a55a-4aaa9a1a70e4\" (UID: \"7c130167-0158-4c0e-a55a-4aaa9a1a70e4\") " Oct 30 09:48:25.425026 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:25.425012 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c130167-0158-4c0e-a55a-4aaa9a1a70e4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c130167-0158-4c0e-a55a-4aaa9a1a70e4-plugins-config-volume\") pod \"7c130167-0158-4c0e-a55a-4aaa9a1a70e4\" (UID: \"7c130167-0158-4c0e-a55a-4aaa9a1a70e4\") " Oct 30 09:48:25.425090 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:25.425079 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c130167-0158-4c0e-a55a-4aaa9a1a70e4-kube-api-access-7hszx\" (UniqueName: \"kubernetes.io/projected/7c130167-0158-4c0e-a55a-4aaa9a1a70e4-kube-api-access-7hszx\") pod \"7c130167-0158-4c0e-a55a-4aaa9a1a70e4\" (UID: \"7c130167-0158-4c0e-a55a-4aaa9a1a70e4\") " Oct 30 09:48:25.425477 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:25.425422 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c130167-0158-4c0e-a55a-4aaa9a1a70e4-routing-overlay-volume" pod "7c130167-0158-4c0e-a55a-4aaa9a1a70e4" (UID: "7c130167-0158-4c0e-a55a-4aaa9a1a70e4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:48:25.425477 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:25.425454 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c130167-0158-4c0e-a55a-4aaa9a1a70e4-plugins-config-volume" pod "7c130167-0158-4c0e-a55a-4aaa9a1a70e4" (UID: "7c130167-0158-4c0e-a55a-4aaa9a1a70e4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:48:25.425983 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:25.425954 4562 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7c130167-0158-4c0e-a55a-4aaa9a1a70e4-routing-overlay-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:48:25.426042 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:25.425982 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c130167-0158-4c0e-a55a-4aaa9a1a70e4-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:48:25.427153 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:25.427127 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c130167-0158-4c0e-a55a-4aaa9a1a70e4-provider-credentials" pod "7c130167-0158-4c0e-a55a-4aaa9a1a70e4" (UID: "7c130167-0158-4c0e-a55a-4aaa9a1a70e4"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:48:25.427643 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:25.427620 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c130167-0158-4c0e-a55a-4aaa9a1a70e4-kube-api-access-7hszx" pod "7c130167-0158-4c0e-a55a-4aaa9a1a70e4" (UID: "7c130167-0158-4c0e-a55a-4aaa9a1a70e4"). InnerVolumeSpecName "kube-api-access-7hszx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:48:25.527944 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:25.527873 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hszx\" (UniqueName: \"kubernetes.io/projected/7c130167-0158-4c0e-a55a-4aaa9a1a70e4-kube-api-access-7hszx\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:48:25.527944 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:25.527903 4562 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7c130167-0158-4c0e-a55a-4aaa9a1a70e4-provider-credentials\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:48:25.751367 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:25.751337 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-59686ffd74-722wt"] Oct 30 09:48:25.764113 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:25.764084 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-59686ffd74-722wt"] Oct 30 09:48:25.876363 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:25.876284 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c130167-0158-4c0e-a55a-4aaa9a1a70e4" path="/var/lib/kubelet/pods/7c130167-0158-4c0e-a55a-4aaa9a1a70e4/volumes" Oct 30 09:48:28.356274 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:28.356235 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5cb55trxn"] Oct 30 09:48:28.357064 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:28.357046 4562 cpu_manager.go:414] "containerMap: removing container" podUID="7c130167-0158-4c0e-a55a-4aaa9a1a70e4" containerName="payload-processing" Oct 30 09:48:28.357120 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:28.357065 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c130167-0158-4c0e-a55a-4aaa9a1a70e4" containerName="payload-processing" Oct 30 09:48:28.357219 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:28.357210 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c130167-0158-4c0e-a55a-4aaa9a1a70e4" containerName="payload-processing" Oct 30 09:48:28.362761 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:28.362740 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5cb55trxn" Oct 30 09:48:28.379253 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:28.379226 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5cb55trxn"] Oct 30 09:48:28.387888 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:28.387696 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3d9b36\"/\"payload-processing-external-model-plugins-e2e-extmod-b-3d9b36\"" Oct 30 09:48:28.387888 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:28.387738 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3d9b36\"/\"agc-cred-cab5f6d18ff4\"" Oct 30 09:48:28.387888 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:28.387830 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3d9b36\"/\"routing-overlay\"" Oct 30 09:48:28.387888 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:28.387848 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3d9b36\"/\"kube-root-ca.crt\"" Oct 30 09:48:28.387888 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:28.387867 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3d9b36\"/\"openshift-service-ca.crt\"" Oct 30 09:48:28.391367 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:28.391343 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3d9b36\"/\"payload-processing-external-model-e2e--dd1a53db-dockercfg-p4mth\"" Oct 30 09:48:28.410751 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:28.410720 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f7614a9-e7c1-4340-bf3e-4db072e415e7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-5cb55trxn\" (UID: \"7f7614a9-e7c1-4340-bf3e-4db072e415e7\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5cb55trxn" Oct 30 09:48:28.410923 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:28.410846 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7f7614a9-e7c1-4340-bf3e-4db072e415e7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-5cb55trxn\" (UID: \"7f7614a9-e7c1-4340-bf3e-4db072e415e7\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5cb55trxn" Oct 30 09:48:28.411335 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:28.411306 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7f7614a9-e7c1-4340-bf3e-4db072e415e7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-5cb55trxn\" (UID: \"7f7614a9-e7c1-4340-bf3e-4db072e415e7\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5cb55trxn" Oct 30 09:48:28.411480 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:28.411431 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzhz8\" (UniqueName: \"kubernetes.io/projected/7f7614a9-e7c1-4340-bf3e-4db072e415e7-kube-api-access-dzhz8\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-5cb55trxn\" (UID: \"7f7614a9-e7c1-4340-bf3e-4db072e415e7\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5cb55trxn" Oct 30 09:48:28.543404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:28.538967 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7f7614a9-e7c1-4340-bf3e-4db072e415e7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-5cb55trxn\" (UID: \"7f7614a9-e7c1-4340-bf3e-4db072e415e7\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5cb55trxn" Oct 30 09:48:28.543404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:28.539087 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzhz8\" (UniqueName: \"kubernetes.io/projected/7f7614a9-e7c1-4340-bf3e-4db072e415e7-kube-api-access-dzhz8\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-5cb55trxn\" (UID: \"7f7614a9-e7c1-4340-bf3e-4db072e415e7\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5cb55trxn" Oct 30 09:48:28.543404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:28.539169 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f7614a9-e7c1-4340-bf3e-4db072e415e7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-5cb55trxn\" (UID: \"7f7614a9-e7c1-4340-bf3e-4db072e415e7\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5cb55trxn" Oct 30 09:48:28.543404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:28.539299 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7f7614a9-e7c1-4340-bf3e-4db072e415e7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-5cb55trxn\" (UID: \"7f7614a9-e7c1-4340-bf3e-4db072e415e7\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5cb55trxn" Oct 30 09:48:28.564764 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:28.554288 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7f7614a9-e7c1-4340-bf3e-4db072e415e7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-5cb55trxn\" (UID: \"7f7614a9-e7c1-4340-bf3e-4db072e415e7\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5cb55trxn" Oct 30 09:48:28.564764 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:28.555111 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f7614a9-e7c1-4340-bf3e-4db072e415e7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-5cb55trxn\" (UID: \"7f7614a9-e7c1-4340-bf3e-4db072e415e7\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5cb55trxn" Oct 30 09:48:28.564764 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:28.556890 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7f7614a9-e7c1-4340-bf3e-4db072e415e7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-5cb55trxn\" (UID: \"7f7614a9-e7c1-4340-bf3e-4db072e415e7\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5cb55trxn" Oct 30 09:48:28.637479 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:28.630095 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzhz8\" (UniqueName: \"kubernetes.io/projected/7f7614a9-e7c1-4340-bf3e-4db072e415e7-kube-api-access-dzhz8\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-5cb55trxn\" (UID: \"7f7614a9-e7c1-4340-bf3e-4db072e415e7\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5cb55trxn" Oct 30 09:48:28.679550 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:28.679498 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5cb55trxn" Oct 30 09:48:28.708283 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:48:28.708245 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:48:29.016658 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:48:29.016617 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f7614a9_e7c1_4340_bf3e_4db072e415e7.slice/crio-81265971e495d9c0613929f6a495e8d37c8dacd5d00a4ce62eed4f9e4f3c42b6 WatchSource:0}: Error finding container 81265971e495d9c0613929f6a495e8d37c8dacd5d00a4ce62eed4f9e4f3c42b6: Status 404 returned error can't find the container with id 81265971e495d9c0613929f6a495e8d37c8dacd5d00a4ce62eed4f9e4f3c42b6 Oct 30 09:48:29.018834 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:29.018812 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5cb55trxn"] Oct 30 09:48:29.463995 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:29.463956 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5cb55trxn" event={"ID":"7f7614a9-e7c1-4340-bf3e-4db072e415e7","Type":"ContainerStarted","Data":"25678d9729f7cc64eca9f2b230a6ca77a58e41cae8a8c4753fad262b706fe445"} Oct 30 09:48:29.464394 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:29.464016 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5cb55trxn" event={"ID":"7f7614a9-e7c1-4340-bf3e-4db072e415e7","Type":"ContainerStarted","Data":"81265971e495d9c0613929f6a495e8d37c8dacd5d00a4ce62eed4f9e4f3c42b6"} Oct 30 09:48:29.464394 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:29.464175 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5cb55trxn" Oct 30 09:48:29.509254 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:29.509206 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5cb55trxn" podStartSLOduration=1.50919406 podStartE2EDuration="1.50919406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:48:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:48:29.507930874 +0000 UTC m=+3269.769491863" watchObservedRunningTime="2026-10-30 09:48:29.50919406 +0000 UTC m=+3269.770755095" Oct 30 09:48:36.571581 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:36.571544 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-f76d9mwcj"] Oct 30 09:48:36.575990 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:36.575974 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-f76d9mwcj" Oct 30 09:48:36.595302 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:36.595273 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-f76d9mwcj"] Oct 30 09:48:36.603348 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:36.602961 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-88cbc7\"/\"payload-processing-external-model-e2e--185456a2-dockercfg-dfrl7\"" Oct 30 09:48:36.603348 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:36.603003 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-88cbc7\"/\"openshift-service-ca.crt\"" Oct 30 09:48:36.603348 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:36.603012 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-88cbc7\"/\"routing-overlay\"" Oct 30 09:48:36.603348 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:36.603066 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-88cbc7\"/\"agc-cred-568a5da76981\"" Oct 30 09:48:36.604316 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:36.604293 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-88cbc7\"/\"kube-root-ca.crt\"" Oct 30 09:48:36.604316 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:36.604291 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-88cbc7\"/\"payload-processing-external-model-plugins-e2e-extmod-a-88cbc7\"" Oct 30 09:48:36.699413 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:36.697951 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/315c3134-24c5-4106-877b-8834a97862dc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-f76d9mwcj\" (UID: \"315c3134-24c5-4106-877b-8834a97862dc\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-f76d9mwcj" Oct 30 09:48:36.699413 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:36.698179 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/315c3134-24c5-4106-877b-8834a97862dc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-f76d9mwcj\" (UID: \"315c3134-24c5-4106-877b-8834a97862dc\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-f76d9mwcj" Oct 30 09:48:36.699413 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:36.698250 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/315c3134-24c5-4106-877b-8834a97862dc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-f76d9mwcj\" (UID: \"315c3134-24c5-4106-877b-8834a97862dc\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-f76d9mwcj" Oct 30 09:48:36.699413 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:36.698419 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdcbc\" (UniqueName: \"kubernetes.io/projected/315c3134-24c5-4106-877b-8834a97862dc-kube-api-access-bdcbc\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-f76d9mwcj\" (UID: \"315c3134-24c5-4106-877b-8834a97862dc\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-f76d9mwcj" Oct 30 09:48:36.802605 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:36.802564 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/315c3134-24c5-4106-877b-8834a97862dc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-f76d9mwcj\" (UID: \"315c3134-24c5-4106-877b-8834a97862dc\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-f76d9mwcj" Oct 30 09:48:36.803990 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:36.803961 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/315c3134-24c5-4106-877b-8834a97862dc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-f76d9mwcj\" (UID: \"315c3134-24c5-4106-877b-8834a97862dc\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-f76d9mwcj" Oct 30 09:48:36.807670 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:36.806501 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdcbc\" (UniqueName: \"kubernetes.io/projected/315c3134-24c5-4106-877b-8834a97862dc-kube-api-access-bdcbc\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-f76d9mwcj\" (UID: \"315c3134-24c5-4106-877b-8834a97862dc\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-f76d9mwcj" Oct 30 09:48:36.807670 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:36.806639 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/315c3134-24c5-4106-877b-8834a97862dc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-f76d9mwcj\" (UID: \"315c3134-24c5-4106-877b-8834a97862dc\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-f76d9mwcj" Oct 30 09:48:36.807670 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:36.803845 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/315c3134-24c5-4106-877b-8834a97862dc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-f76d9mwcj\" (UID: \"315c3134-24c5-4106-877b-8834a97862dc\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-f76d9mwcj" Oct 30 09:48:36.807670 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:36.804706 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/315c3134-24c5-4106-877b-8834a97862dc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-f76d9mwcj\" (UID: \"315c3134-24c5-4106-877b-8834a97862dc\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-f76d9mwcj" Oct 30 09:48:36.839504 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:36.821285 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/315c3134-24c5-4106-877b-8834a97862dc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-f76d9mwcj\" (UID: \"315c3134-24c5-4106-877b-8834a97862dc\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-f76d9mwcj" Oct 30 09:48:36.852416 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:36.852254 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdcbc\" (UniqueName: \"kubernetes.io/projected/315c3134-24c5-4106-877b-8834a97862dc-kube-api-access-bdcbc\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-f76d9mwcj\" (UID: \"315c3134-24c5-4106-877b-8834a97862dc\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-f76d9mwcj" Oct 30 09:48:36.892493 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:36.892458 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-f76d9mwcj" Oct 30 09:48:37.209119 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:48:37.209090 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod315c3134_24c5_4106_877b_8834a97862dc.slice/crio-e18e2e612a9a21b2988648b10c169e7e8c434f858057df527b88347bb513c338 WatchSource:0}: Error finding container e18e2e612a9a21b2988648b10c169e7e8c434f858057df527b88347bb513c338: Status 404 returned error can't find the container with id e18e2e612a9a21b2988648b10c169e7e8c434f858057df527b88347bb513c338 Oct 30 09:48:37.210746 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:37.210718 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-f76d9mwcj"] Oct 30 09:48:37.571746 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:37.571648 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-f76d9mwcj" event={"ID":"315c3134-24c5-4106-877b-8834a97862dc","Type":"ContainerStarted","Data":"a782dd9de13c844f4a6ed7d1d6f5e93c39e15d115f6c998a97ee92086ae99260"} Oct 30 09:48:37.571746 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:37.571718 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-f76d9mwcj" Oct 30 09:48:37.571746 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:37.571740 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-f76d9mwcj" event={"ID":"315c3134-24c5-4106-877b-8834a97862dc","Type":"ContainerStarted","Data":"e18e2e612a9a21b2988648b10c169e7e8c434f858057df527b88347bb513c338"} Oct 30 09:48:37.616553 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:37.616493 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-f76d9mwcj" podStartSLOduration=1.616480194 podStartE2EDuration="1.616480194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:48:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:48:37.612577197 +0000 UTC m=+3277.874138184" watchObservedRunningTime="2026-10-30 09:48:37.616480194 +0000 UTC m=+3277.878041201" Oct 30 09:48:40.476417 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:40.476359 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5cb55trxn" Oct 30 09:48:48.583563 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:48:48.583476 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-f76d9mwcj" Oct 30 09:48:58.598224 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:48:58.598189 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:49:00.470459 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:49:00.470425 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6q7n_1f51db5a-5720-4dea-ad02-f336b35d43b1/ovn-acl-logging/0.log" Oct 30 09:49:00.485505 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:49:00.485473 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6q7n_1f51db5a-5720-4dea-ad02-f336b35d43b1/ovn-acl-logging/0.log" Oct 30 09:49:01.169425 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:49:01.166566 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5fbb64fb6b-mgvv7"] Oct 30 09:49:01.200371 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:49:01.200338 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5fbb64fb6b-mgvv7" Oct 30 09:49:01.222404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:49:01.221637 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-908e572dbf63\"" Oct 30 09:49:01.222404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:49:01.221736 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-ff81f0cb94c6\"" Oct 30 09:49:01.236072 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:49:01.236044 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 30 09:49:01.236528 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:49:01.236506 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-pwntl\"" Oct 30 09:49:01.237243 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:49:01.237222 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 30 09:49:01.237736 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:49:01.237717 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 30 09:49:01.239323 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:49:01.239297 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 30 09:49:01.313401 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:49:01.313350 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5fbb64fb6b-mgvv7"] Oct 30 09:49:01.327615 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:49:01.326567 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1218471-2eca-49fc-8efc-fc6c801b9ffd-plugins-config-volume\") pod \"payload-processing-external-model-5fbb64fb6b-mgvv7\" (UID: \"f1218471-2eca-49fc-8efc-fc6c801b9ffd\") " pod="models-as-a-service/payload-processing-external-model-5fbb64fb6b-mgvv7" Oct 30 09:49:01.327615 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:49:01.326650 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f1218471-2eca-49fc-8efc-fc6c801b9ffd-provider-credentials\") pod \"payload-processing-external-model-5fbb64fb6b-mgvv7\" (UID: \"f1218471-2eca-49fc-8efc-fc6c801b9ffd\") " pod="models-as-a-service/payload-processing-external-model-5fbb64fb6b-mgvv7" Oct 30 09:49:01.327615 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:49:01.326749 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f1218471-2eca-49fc-8efc-fc6c801b9ffd-routing-overlay-volume\") pod \"payload-processing-external-model-5fbb64fb6b-mgvv7\" (UID: \"f1218471-2eca-49fc-8efc-fc6c801b9ffd\") " pod="models-as-a-service/payload-processing-external-model-5fbb64fb6b-mgvv7" Oct 30 09:49:01.327615 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:49:01.326978 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mshn\" (UniqueName: \"kubernetes.io/projected/f1218471-2eca-49fc-8efc-fc6c801b9ffd-kube-api-access-2mshn\") pod \"payload-processing-external-model-5fbb64fb6b-mgvv7\" (UID: \"f1218471-2eca-49fc-8efc-fc6c801b9ffd\") " pod="models-as-a-service/payload-processing-external-model-5fbb64fb6b-mgvv7" Oct 30 09:49:01.429205 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:49:01.429116 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f1218471-2eca-49fc-8efc-fc6c801b9ffd-routing-overlay-volume\") pod \"payload-processing-external-model-5fbb64fb6b-mgvv7\" (UID: \"f1218471-2eca-49fc-8efc-fc6c801b9ffd\") " pod="models-as-a-service/payload-processing-external-model-5fbb64fb6b-mgvv7" Oct 30 09:49:01.429360 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:49:01.429343 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mshn\" (UniqueName: \"kubernetes.io/projected/f1218471-2eca-49fc-8efc-fc6c801b9ffd-kube-api-access-2mshn\") pod \"payload-processing-external-model-5fbb64fb6b-mgvv7\" (UID: \"f1218471-2eca-49fc-8efc-fc6c801b9ffd\") " pod="models-as-a-service/payload-processing-external-model-5fbb64fb6b-mgvv7" Oct 30 09:49:01.429516 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:49:01.429497 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1218471-2eca-49fc-8efc-fc6c801b9ffd-plugins-config-volume\") pod \"payload-processing-external-model-5fbb64fb6b-mgvv7\" (UID: \"f1218471-2eca-49fc-8efc-fc6c801b9ffd\") " pod="models-as-a-service/payload-processing-external-model-5fbb64fb6b-mgvv7" Oct 30 09:49:01.429566 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:49:01.429555 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f1218471-2eca-49fc-8efc-fc6c801b9ffd-provider-credentials\") pod \"payload-processing-external-model-5fbb64fb6b-mgvv7\" (UID: \"f1218471-2eca-49fc-8efc-fc6c801b9ffd\") " pod="models-as-a-service/payload-processing-external-model-5fbb64fb6b-mgvv7" Oct 30 09:49:01.429955 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:49:01.429929 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f1218471-2eca-49fc-8efc-fc6c801b9ffd-routing-overlay-volume\") pod \"payload-processing-external-model-5fbb64fb6b-mgvv7\" (UID: \"f1218471-2eca-49fc-8efc-fc6c801b9ffd\") " pod="models-as-a-service/payload-processing-external-model-5fbb64fb6b-mgvv7" Oct 30 09:49:01.430237 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:49:01.430216 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1218471-2eca-49fc-8efc-fc6c801b9ffd-plugins-config-volume\") pod \"payload-processing-external-model-5fbb64fb6b-mgvv7\" (UID: \"f1218471-2eca-49fc-8efc-fc6c801b9ffd\") " pod="models-as-a-service/payload-processing-external-model-5fbb64fb6b-mgvv7" Oct 30 09:49:01.432105 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:49:01.432079 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f1218471-2eca-49fc-8efc-fc6c801b9ffd-provider-credentials\") pod \"payload-processing-external-model-5fbb64fb6b-mgvv7\" (UID: \"f1218471-2eca-49fc-8efc-fc6c801b9ffd\") " pod="models-as-a-service/payload-processing-external-model-5fbb64fb6b-mgvv7" Oct 30 09:49:01.536746 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:49:01.536712 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mshn\" (UniqueName: \"kubernetes.io/projected/f1218471-2eca-49fc-8efc-fc6c801b9ffd-kube-api-access-2mshn\") pod \"payload-processing-external-model-5fbb64fb6b-mgvv7\" (UID: \"f1218471-2eca-49fc-8efc-fc6c801b9ffd\") " pod="models-as-a-service/payload-processing-external-model-5fbb64fb6b-mgvv7" Oct 30 09:49:01.639291 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:49:01.639253 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5fbb64fb6b-mgvv7" Oct 30 09:49:01.951052 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:49:01.951024 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1218471_2eca_49fc_8efc_fc6c801b9ffd.slice/crio-c7c9bc95158ce9471b711ea3194776f85ebac6a093ed7693c02b33a03b42a2ea WatchSource:0}: Error finding container c7c9bc95158ce9471b711ea3194776f85ebac6a093ed7693c02b33a03b42a2ea: Status 404 returned error can't find the container with id c7c9bc95158ce9471b711ea3194776f85ebac6a093ed7693c02b33a03b42a2ea Oct 30 09:49:01.953532 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:49:01.953507 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5fbb64fb6b-mgvv7"] Oct 30 09:49:02.919512 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:49:02.919471 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5fbb64fb6b-mgvv7" event={"ID":"f1218471-2eca-49fc-8efc-fc6c801b9ffd","Type":"ContainerStarted","Data":"e7a459bbe3efb4b0d6783254ce43b8d0dd535d090d75fdace38abd6dbeb4e0f9"} Oct 30 09:49:02.920006 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:49:02.919538 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5fbb64fb6b-mgvv7" event={"ID":"f1218471-2eca-49fc-8efc-fc6c801b9ffd","Type":"ContainerStarted","Data":"c7c9bc95158ce9471b711ea3194776f85ebac6a093ed7693c02b33a03b42a2ea"} Oct 30 09:49:02.920006 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:49:02.919569 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5fbb64fb6b-mgvv7" Oct 30 09:49:02.969334 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:49:02.969280 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5fbb64fb6b-mgvv7" podStartSLOduration=1.969256595 podStartE2EDuration="1.969256595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:49:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:49:02.965728524 +0000 UTC m=+3303.227289513" watchObservedRunningTime="2026-10-30 09:49:02.969256595 +0000 UTC m=+3303.230817583" Oct 30 09:49:08.530915 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:49:08.530877 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2f6ee2-684d557c9-gf6cq"] Oct 30 09:49:08.536012 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:49:08.535991 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f6ee2-684d557c9-gf6cq" Oct 30 09:49:08.552936 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:49:08.552907 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2f6ee2-684d557c9-gf6cq"] Oct 30 09:49:08.562424 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:49:08.560959 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-2f6ee2-dockercfg-5vmkc\"" Oct 30 09:49:08.562424 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:49:08.560983 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-2f6ee2\"" Oct 30 09:49:08.803455 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:49:08.778413 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrqbn\" (UniqueName: \"kubernetes.io/projected/78e55c08-322e-464e-a798-2d3b2d894972-kube-api-access-qrqbn\") pod \"payload-pre-processing-e2e-inf-a-2f6ee2-684d557c9-gf6cq\" (UID: \"78e55c08-322e-464e-a798-2d3b2d894972\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f6ee2-684d557c9-gf6cq" Oct 30 09:49:08.803455 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:49:08.778594 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78e55c08-322e-464e-a798-2d3b2d894972-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2f6ee2-684d557c9-gf6cq\" (UID: \"78e55c08-322e-464e-a798-2d3b2d894972\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f6ee2-684d557c9-gf6cq" Oct 30 09:49:08.859531 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:49:08.859497 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2f6ee2-844666dcbd-56ppf"] Oct 30 09:49:08.870362 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:49:08.870334 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2f6ee2-844666dcbd-56ppf"] Oct 30 09:49:08.870529 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:49:08.870503 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2f6ee2-844666dcbd-56ppf" Oct 30 09:49:08.879914 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:49:08.879881 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36c4eb3b-f628-48a2-82b6-aa1826bbf0c6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2f6ee2-844666dcbd-56ppf\" (UID: \"36c4eb3b-f628-48a2-82b6-aa1826bbf0c6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2f6ee2-844666dcbd-56ppf" Oct 30 09:49:08.880026 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:49:08.879960 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrqbn\" (UniqueName: \"kubernetes.io/projected/78e55c08-322e-464e-a798-2d3b2d894972-kube-api-access-qrqbn\") pod \"payload-pre-processing-e2e-inf-a-2f6ee2-684d557c9-gf6cq\" (UID: \"78e55c08-322e-464e-a798-2d3b2d894972\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f6ee2-684d557c9-gf6cq" Oct 30 09:49:08.880077 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:49:08.880069 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78e55c08-322e-464e-a798-2d3b2d894972-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2f6ee2-684d557c9-gf6cq\" (UID: \"78e55c08-322e-464e-a798-2d3b2d894972\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f6ee2-684d557c9-gf6cq" Oct 30 09:49:08.880141 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:49:08.880123 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr8p9\" (UniqueName: \"kubernetes.io/projected/36c4eb3b-f628-48a2-82b6-aa1826bbf0c6-kube-api-access-xr8p9\") pod \"payload-processing-e2e-inf-a-2f6ee2-844666dcbd-56ppf\" (UID: \"36c4eb3b-f628-48a2-82b6-aa1826bbf0c6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2f6ee2-844666dcbd-56ppf" Oct 30 09:49:08.880845 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:49:08.880824 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78e55c08-322e-464e-a798-2d3b2d894972-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2f6ee2-684d557c9-gf6cq\" (UID: \"78e55c08-322e-464e-a798-2d3b2d894972\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f6ee2-684d557c9-gf6cq" Oct 30 09:49:08.900191 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:49:08.900168 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrqbn\" (UniqueName: \"kubernetes.io/projected/78e55c08-322e-464e-a798-2d3b2d894972-kube-api-access-qrqbn\") pod \"payload-pre-processing-e2e-inf-a-2f6ee2-684d557c9-gf6cq\" (UID: \"78e55c08-322e-464e-a798-2d3b2d894972\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f6ee2-684d557c9-gf6cq" Oct 30 09:49:08.981762 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:49:08.981729 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr8p9\" (UniqueName: \"kubernetes.io/projected/36c4eb3b-f628-48a2-82b6-aa1826bbf0c6-kube-api-access-xr8p9\") pod \"payload-processing-e2e-inf-a-2f6ee2-844666dcbd-56ppf\" (UID: \"36c4eb3b-f628-48a2-82b6-aa1826bbf0c6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2f6ee2-844666dcbd-56ppf" Oct 30 09:49:08.981932 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:49:08.981871 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36c4eb3b-f628-48a2-82b6-aa1826bbf0c6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2f6ee2-844666dcbd-56ppf\" (UID: \"36c4eb3b-f628-48a2-82b6-aa1826bbf0c6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2f6ee2-844666dcbd-56ppf" Oct 30 09:49:08.982582 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:49:08.982563 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36c4eb3b-f628-48a2-82b6-aa1826bbf0c6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2f6ee2-844666dcbd-56ppf\" (UID: \"36c4eb3b-f628-48a2-82b6-aa1826bbf0c6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2f6ee2-844666dcbd-56ppf" Oct 30 09:49:09.001532 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:49:09.001508 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr8p9\" (UniqueName: \"kubernetes.io/projected/36c4eb3b-f628-48a2-82b6-aa1826bbf0c6-kube-api-access-xr8p9\") pod \"payload-processing-e2e-inf-a-2f6ee2-844666dcbd-56ppf\" (UID: \"36c4eb3b-f628-48a2-82b6-aa1826bbf0c6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2f6ee2-844666dcbd-56ppf" Oct 30 09:49:09.152806 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:49:09.152740 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f6ee2-684d557c9-gf6cq" Oct 30 09:49:09.186672 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:49:09.186642 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2f6ee2-844666dcbd-56ppf" Oct 30 09:49:09.458681 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:49:09.458643 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78e55c08_322e_464e_a798_2d3b2d894972.slice/crio-cea4333656c8e203ea2ba3de09577bc4575aceff77de2b803867bd798018488e WatchSource:0}: Error finding container cea4333656c8e203ea2ba3de09577bc4575aceff77de2b803867bd798018488e: Status 404 returned error can't find the container with id cea4333656c8e203ea2ba3de09577bc4575aceff77de2b803867bd798018488e Oct 30 09:49:09.460542 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:49:09.460515 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2f6ee2-684d557c9-gf6cq"] Oct 30 09:49:09.500920 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:49:09.500889 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36c4eb3b_f628_48a2_82b6_aa1826bbf0c6.slice/crio-80736d71329f2a29dd13f1804aeb633be5fdd354e6f25006f9f565a557dafb44 WatchSource:0}: Error finding container 80736d71329f2a29dd13f1804aeb633be5fdd354e6f25006f9f565a557dafb44: Status 404 returned error can't find the container with id 80736d71329f2a29dd13f1804aeb633be5fdd354e6f25006f9f565a557dafb44 Oct 30 09:49:09.503055 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:49:09.503030 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2f6ee2-844666dcbd-56ppf"] Oct 30 09:49:09.988037 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:49:09.987971 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f6ee2-684d557c9-gf6cq" event={"ID":"78e55c08-322e-464e-a798-2d3b2d894972","Type":"ContainerStarted","Data":"17b2b9376656089f75041188893d68877fd15b9e9719194dfde5615eb2037c5b"} Oct 30 09:49:09.988582 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:49:09.988116 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f6ee2-684d557c9-gf6cq" event={"ID":"78e55c08-322e-464e-a798-2d3b2d894972","Type":"ContainerStarted","Data":"cea4333656c8e203ea2ba3de09577bc4575aceff77de2b803867bd798018488e"} Oct 30 09:49:09.988821 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:49:09.988797 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f6ee2-684d557c9-gf6cq" Oct 30 09:49:09.990794 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:49:09.990768 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2f6ee2-844666dcbd-56ppf" event={"ID":"36c4eb3b-f628-48a2-82b6-aa1826bbf0c6","Type":"ContainerStarted","Data":"8ca6619bf30eedb4671b3a0f00b5f91ba7721b029a3e1678ed45b93513584e02"} Oct 30 09:49:09.990918 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:49:09.990815 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2f6ee2-844666dcbd-56ppf" event={"ID":"36c4eb3b-f628-48a2-82b6-aa1826bbf0c6","Type":"ContainerStarted","Data":"80736d71329f2a29dd13f1804aeb633be5fdd354e6f25006f9f565a557dafb44"} Oct 30 09:49:09.990918 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:49:09.990837 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2f6ee2-844666dcbd-56ppf" Oct 30 09:49:10.028536 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:49:10.028489 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f6ee2-684d557c9-gf6cq" podStartSLOduration=2.028476401 podStartE2EDuration="2.028476401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:49:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:49:10.027609676 +0000 UTC m=+3310.289170663" watchObservedRunningTime="2026-10-30 09:49:10.028476401 +0000 UTC m=+3310.290037389" Oct 30 09:49:13.954619 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:49:13.954586 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5fbb64fb6b-mgvv7" Oct 30 09:49:13.995569 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:49:13.995518 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-2f6ee2-844666dcbd-56ppf" podStartSLOduration=5.9955036029999995 podStartE2EDuration="5.995503603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:49:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:49:10.069546049 +0000 UTC m=+3310.331107037" watchObservedRunningTime="2026-10-30 09:49:13.995503603 +0000 UTC m=+3314.257064626" Oct 30 09:49:21.002787 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:49:21.002750 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2f6ee2-844666dcbd-56ppf" Oct 30 09:49:21.003329 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:49:21.002851 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f6ee2-684d557c9-gf6cq" Oct 30 09:49:28.597328 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:49:28.597292 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:49:58.596929 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:49:58.596889 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:50:28.599396 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:50:28.599342 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:50:40.817144 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:50:40.817101 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-bdd208-784594d77d-z6qzr"] Oct 30 09:50:40.822308 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:50:40.822284 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-bdd208-784594d77d-z6qzr" Oct 30 09:50:40.841948 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:50:40.841920 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-bdd208-784594d77d-z6qzr"] Oct 30 09:50:40.846517 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:50:40.846474 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-bdd208\"" Oct 30 09:50:40.856269 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:50:40.856232 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcea2042-49d4-4703-a59c-fdb378c3299c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-bdd208-784594d77d-z6qzr\" (UID: \"dcea2042-49d4-4703-a59c-fdb378c3299c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-bdd208-784594d77d-z6qzr" Oct 30 09:50:40.856461 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:50:40.856345 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c26vn\" (UniqueName: \"kubernetes.io/projected/dcea2042-49d4-4703-a59c-fdb378c3299c-kube-api-access-c26vn\") pod \"payload-processing-e2e-inf-b-bdd208-784594d77d-z6qzr\" (UID: \"dcea2042-49d4-4703-a59c-fdb378c3299c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-bdd208-784594d77d-z6qzr" Oct 30 09:50:40.881193 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:50:40.881159 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bdd208-6dff5fbd47-69qxt"] Oct 30 09:50:40.886493 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:50:40.886461 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bdd208-6dff5fbd47-69qxt" Oct 30 09:50:40.901956 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:50:40.901928 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-bdd208-dockercfg-2lqs6\"" Oct 30 09:50:40.904890 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:50:40.904794 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bdd208-6dff5fbd47-69qxt"] Oct 30 09:50:40.990740 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:50:40.989497 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcea2042-49d4-4703-a59c-fdb378c3299c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-bdd208-784594d77d-z6qzr\" (UID: \"dcea2042-49d4-4703-a59c-fdb378c3299c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-bdd208-784594d77d-z6qzr" Oct 30 09:50:40.992527 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:50:40.992497 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4f0d99e-ad40-4806-8cda-148b2ba50945-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-bdd208-6dff5fbd47-69qxt\" (UID: \"e4f0d99e-ad40-4806-8cda-148b2ba50945\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bdd208-6dff5fbd47-69qxt" Oct 30 09:50:41.036067 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:50:41.036034 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c26vn\" (UniqueName: \"kubernetes.io/projected/dcea2042-49d4-4703-a59c-fdb378c3299c-kube-api-access-c26vn\") pod \"payload-processing-e2e-inf-b-bdd208-784594d77d-z6qzr\" (UID: \"dcea2042-49d4-4703-a59c-fdb378c3299c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-bdd208-784594d77d-z6qzr" Oct 30 09:50:41.037638 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:50:41.037607 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnh6p\" (UniqueName: \"kubernetes.io/projected/e4f0d99e-ad40-4806-8cda-148b2ba50945-kube-api-access-rnh6p\") pod \"payload-pre-processing-e2e-inf-b-bdd208-6dff5fbd47-69qxt\" (UID: \"e4f0d99e-ad40-4806-8cda-148b2ba50945\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bdd208-6dff5fbd47-69qxt" Oct 30 09:50:41.037967 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:50:40.992396 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcea2042-49d4-4703-a59c-fdb378c3299c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-bdd208-784594d77d-z6qzr\" (UID: \"dcea2042-49d4-4703-a59c-fdb378c3299c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-bdd208-784594d77d-z6qzr" Oct 30 09:50:41.106327 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:50:41.106245 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c26vn\" (UniqueName: \"kubernetes.io/projected/dcea2042-49d4-4703-a59c-fdb378c3299c-kube-api-access-c26vn\") pod \"payload-processing-e2e-inf-b-bdd208-784594d77d-z6qzr\" (UID: \"dcea2042-49d4-4703-a59c-fdb378c3299c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-bdd208-784594d77d-z6qzr" Oct 30 09:50:41.144652 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:50:41.140490 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnh6p\" (UniqueName: \"kubernetes.io/projected/e4f0d99e-ad40-4806-8cda-148b2ba50945-kube-api-access-rnh6p\") pod \"payload-pre-processing-e2e-inf-b-bdd208-6dff5fbd47-69qxt\" (UID: \"e4f0d99e-ad40-4806-8cda-148b2ba50945\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bdd208-6dff5fbd47-69qxt" Oct 30 09:50:41.144652 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:50:41.140755 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4f0d99e-ad40-4806-8cda-148b2ba50945-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-bdd208-6dff5fbd47-69qxt\" (UID: \"e4f0d99e-ad40-4806-8cda-148b2ba50945\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bdd208-6dff5fbd47-69qxt" Oct 30 09:50:41.144652 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:50:41.141964 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4f0d99e-ad40-4806-8cda-148b2ba50945-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-bdd208-6dff5fbd47-69qxt\" (UID: \"e4f0d99e-ad40-4806-8cda-148b2ba50945\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bdd208-6dff5fbd47-69qxt" Oct 30 09:50:41.156784 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:50:41.156752 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-bdd208-784594d77d-z6qzr" Oct 30 09:50:41.187618 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:50:41.186436 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnh6p\" (UniqueName: \"kubernetes.io/projected/e4f0d99e-ad40-4806-8cda-148b2ba50945-kube-api-access-rnh6p\") pod \"payload-pre-processing-e2e-inf-b-bdd208-6dff5fbd47-69qxt\" (UID: \"e4f0d99e-ad40-4806-8cda-148b2ba50945\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bdd208-6dff5fbd47-69qxt" Oct 30 09:50:41.214702 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:50:41.214412 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bdd208-6dff5fbd47-69qxt" Oct 30 09:50:41.487171 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:50:41.487143 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcea2042_49d4_4703_a59c_fdb378c3299c.slice/crio-44572447af7542973d8d65589213accb19ecdbbdebf5642636e4bd74a650dd70 WatchSource:0}: Error finding container 44572447af7542973d8d65589213accb19ecdbbdebf5642636e4bd74a650dd70: Status 404 returned error can't find the container with id 44572447af7542973d8d65589213accb19ecdbbdebf5642636e4bd74a650dd70 Oct 30 09:50:41.489724 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:50:41.489699 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-bdd208-784594d77d-z6qzr"] Oct 30 09:50:41.540030 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:50:41.539992 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4f0d99e_ad40_4806_8cda_148b2ba50945.slice/crio-441c683a6fa8efa2c7fb637a5477a41f125743c850f671796fbdefabc5a4e350 WatchSource:0}: Error finding container 441c683a6fa8efa2c7fb637a5477a41f125743c850f671796fbdefabc5a4e350: Status 404 returned error can't find the container with id 441c683a6fa8efa2c7fb637a5477a41f125743c850f671796fbdefabc5a4e350 Oct 30 09:50:41.541693 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:50:41.541672 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bdd208-6dff5fbd47-69qxt"] Oct 30 09:50:41.912272 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:50:41.912183 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-bdd208-784594d77d-z6qzr" event={"ID":"dcea2042-49d4-4703-a59c-fdb378c3299c","Type":"ContainerStarted","Data":"b4175ed8790dea968d2520bb5f7b0c432e370d743980fe5b2de34c5d0bf9d198"} Oct 30 09:50:41.912272 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:50:41.912243 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-bdd208-784594d77d-z6qzr" event={"ID":"dcea2042-49d4-4703-a59c-fdb378c3299c","Type":"ContainerStarted","Data":"44572447af7542973d8d65589213accb19ecdbbdebf5642636e4bd74a650dd70"} Oct 30 09:50:41.912795 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:50:41.912408 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-bdd208-784594d77d-z6qzr" Oct 30 09:50:41.914009 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:50:41.913982 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bdd208-6dff5fbd47-69qxt" event={"ID":"e4f0d99e-ad40-4806-8cda-148b2ba50945","Type":"ContainerStarted","Data":"746442c8724d9cf10a1839423c6052d8110106277aee6f024a59df1e2508ca3f"} Oct 30 09:50:41.914165 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:50:41.914024 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bdd208-6dff5fbd47-69qxt" event={"ID":"e4f0d99e-ad40-4806-8cda-148b2ba50945","Type":"ContainerStarted","Data":"441c683a6fa8efa2c7fb637a5477a41f125743c850f671796fbdefabc5a4e350"} Oct 30 09:50:41.914165 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:50:41.914065 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bdd208-6dff5fbd47-69qxt" Oct 30 09:50:41.952248 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:50:41.952199 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-bdd208-784594d77d-z6qzr" podStartSLOduration=1.952187657 podStartE2EDuration="1.952187657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:50:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:50:41.951549807 +0000 UTC m=+3402.213110795" watchObservedRunningTime="2026-10-30 09:50:41.952187657 +0000 UTC m=+3402.213748645" Oct 30 09:50:41.995192 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:50:41.995147 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bdd208-6dff5fbd47-69qxt" podStartSLOduration=1.995135571 podStartE2EDuration="1.995135571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:50:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:50:41.990752819 +0000 UTC m=+3402.252313808" watchObservedRunningTime="2026-10-30 09:50:41.995135571 +0000 UTC m=+3402.256696558" Oct 30 09:50:52.924664 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:50:52.924630 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bdd208-6dff5fbd47-69qxt" Oct 30 09:50:52.925053 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:50:52.924774 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-bdd208-784594d77d-z6qzr" Oct 30 09:50:58.598840 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:50:58.598801 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:51:00.831969 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:00.831935 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-56dc545b7c-d92fs"] Oct 30 09:51:00.837250 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:00.837226 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56dc545b7c-d92fs" Oct 30 09:51:00.853822 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:00.853793 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-56dc545b7c-d92fs"] Oct 30 09:51:00.976132 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:00.976044 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e6b77f09-6ef1-437c-8fc6-2720b10122e6-provider-credentials\") pod \"payload-processing-external-model-56dc545b7c-d92fs\" (UID: \"e6b77f09-6ef1-437c-8fc6-2720b10122e6\") " pod="models-as-a-service/payload-processing-external-model-56dc545b7c-d92fs" Oct 30 09:51:00.976433 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:00.976409 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6b77f09-6ef1-437c-8fc6-2720b10122e6-plugins-config-volume\") pod \"payload-processing-external-model-56dc545b7c-d92fs\" (UID: \"e6b77f09-6ef1-437c-8fc6-2720b10122e6\") " pod="models-as-a-service/payload-processing-external-model-56dc545b7c-d92fs" Oct 30 09:51:00.976605 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:00.976576 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rnlb\" (UniqueName: \"kubernetes.io/projected/e6b77f09-6ef1-437c-8fc6-2720b10122e6-kube-api-access-8rnlb\") pod \"payload-processing-external-model-56dc545b7c-d92fs\" (UID: \"e6b77f09-6ef1-437c-8fc6-2720b10122e6\") " pod="models-as-a-service/payload-processing-external-model-56dc545b7c-d92fs" Oct 30 09:51:00.976771 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:00.976748 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e6b77f09-6ef1-437c-8fc6-2720b10122e6-routing-overlay-volume\") pod \"payload-processing-external-model-56dc545b7c-d92fs\" (UID: \"e6b77f09-6ef1-437c-8fc6-2720b10122e6\") " pod="models-as-a-service/payload-processing-external-model-56dc545b7c-d92fs" Oct 30 09:51:01.078261 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:01.078227 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e6b77f09-6ef1-437c-8fc6-2720b10122e6-provider-credentials\") pod \"payload-processing-external-model-56dc545b7c-d92fs\" (UID: \"e6b77f09-6ef1-437c-8fc6-2720b10122e6\") " pod="models-as-a-service/payload-processing-external-model-56dc545b7c-d92fs" Oct 30 09:51:01.078471 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:01.078275 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6b77f09-6ef1-437c-8fc6-2720b10122e6-plugins-config-volume\") pod \"payload-processing-external-model-56dc545b7c-d92fs\" (UID: \"e6b77f09-6ef1-437c-8fc6-2720b10122e6\") " pod="models-as-a-service/payload-processing-external-model-56dc545b7c-d92fs" Oct 30 09:51:01.078471 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:01.078316 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rnlb\" (UniqueName: \"kubernetes.io/projected/e6b77f09-6ef1-437c-8fc6-2720b10122e6-kube-api-access-8rnlb\") pod \"payload-processing-external-model-56dc545b7c-d92fs\" (UID: \"e6b77f09-6ef1-437c-8fc6-2720b10122e6\") " pod="models-as-a-service/payload-processing-external-model-56dc545b7c-d92fs" Oct 30 09:51:01.078471 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:01.078360 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e6b77f09-6ef1-437c-8fc6-2720b10122e6-routing-overlay-volume\") pod \"payload-processing-external-model-56dc545b7c-d92fs\" (UID: \"e6b77f09-6ef1-437c-8fc6-2720b10122e6\") " pod="models-as-a-service/payload-processing-external-model-56dc545b7c-d92fs" Oct 30 09:51:01.079293 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:01.079264 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e6b77f09-6ef1-437c-8fc6-2720b10122e6-routing-overlay-volume\") pod \"payload-processing-external-model-56dc545b7c-d92fs\" (UID: \"e6b77f09-6ef1-437c-8fc6-2720b10122e6\") " pod="models-as-a-service/payload-processing-external-model-56dc545b7c-d92fs" Oct 30 09:51:01.079598 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:01.079538 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6b77f09-6ef1-437c-8fc6-2720b10122e6-plugins-config-volume\") pod \"payload-processing-external-model-56dc545b7c-d92fs\" (UID: \"e6b77f09-6ef1-437c-8fc6-2720b10122e6\") " pod="models-as-a-service/payload-processing-external-model-56dc545b7c-d92fs" Oct 30 09:51:01.081115 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:01.081094 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e6b77f09-6ef1-437c-8fc6-2720b10122e6-provider-credentials\") pod \"payload-processing-external-model-56dc545b7c-d92fs\" (UID: \"e6b77f09-6ef1-437c-8fc6-2720b10122e6\") " pod="models-as-a-service/payload-processing-external-model-56dc545b7c-d92fs" Oct 30 09:51:01.098919 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:01.098854 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rnlb\" (UniqueName: \"kubernetes.io/projected/e6b77f09-6ef1-437c-8fc6-2720b10122e6-kube-api-access-8rnlb\") pod \"payload-processing-external-model-56dc545b7c-d92fs\" (UID: \"e6b77f09-6ef1-437c-8fc6-2720b10122e6\") " pod="models-as-a-service/payload-processing-external-model-56dc545b7c-d92fs" Oct 30 09:51:01.176551 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:01.164685 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56dc545b7c-d92fs" Oct 30 09:51:01.592830 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:51:01.592791 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6b77f09_6ef1_437c_8fc6_2720b10122e6.slice/crio-6c66bdf058cbbb3a779e8c0815a99a8a335cd638fc1febd953ab0a4f58fe0a2d WatchSource:0}: Error finding container 6c66bdf058cbbb3a779e8c0815a99a8a335cd638fc1febd953ab0a4f58fe0a2d: Status 404 returned error can't find the container with id 6c66bdf058cbbb3a779e8c0815a99a8a335cd638fc1febd953ab0a4f58fe0a2d Oct 30 09:51:01.594262 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:01.594236 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-56dc545b7c-d92fs"] Oct 30 09:51:02.077327 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:02.077276 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56dc545b7c-d92fs" event={"ID":"e6b77f09-6ef1-437c-8fc6-2720b10122e6","Type":"ContainerStarted","Data":"5aeabf936c06ba2168c9238a726c3f3bf0d5b9cbbf8b0ec7b5908df163187955"} Oct 30 09:51:02.077857 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:02.077337 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56dc545b7c-d92fs" event={"ID":"e6b77f09-6ef1-437c-8fc6-2720b10122e6","Type":"ContainerStarted","Data":"6c66bdf058cbbb3a779e8c0815a99a8a335cd638fc1febd953ab0a4f58fe0a2d"} Oct 30 09:51:02.077857 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:02.077406 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-56dc545b7c-d92fs" Oct 30 09:51:02.117946 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:02.117889 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-56dc545b7c-d92fs" podStartSLOduration=2.117874599 podStartE2EDuration="2.117874599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:51:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:51:02.117044635 +0000 UTC m=+3422.378605703" watchObservedRunningTime="2026-10-30 09:51:02.117874599 +0000 UTC m=+3422.379435776" Oct 30 09:51:08.274743 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:08.274709 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-82ccfd-6b4c8757ff-rc85k"] Oct 30 09:51:08.279900 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:08.279876 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82ccfd-6b4c8757ff-rc85k" Oct 30 09:51:08.298347 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:08.298317 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-82ccfd-6b4c8757ff-rc85k"] Oct 30 09:51:08.304578 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:08.304551 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-82ccfd\"" Oct 30 09:51:08.304578 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:08.304561 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-82ccfd-dockercfg-bwrq5\"" Oct 30 09:51:08.321374 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:08.321342 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e7713d8-4bdf-4134-9361-e4c0bca6b525-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-82ccfd-6b4c8757ff-rc85k\" (UID: \"2e7713d8-4bdf-4134-9361-e4c0bca6b525\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82ccfd-6b4c8757ff-rc85k" Oct 30 09:51:08.321702 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:08.321674 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q22z\" (UniqueName: \"kubernetes.io/projected/2e7713d8-4bdf-4134-9361-e4c0bca6b525-kube-api-access-8q22z\") pod \"payload-pre-processing-e2e-inf-a-82ccfd-6b4c8757ff-rc85k\" (UID: \"2e7713d8-4bdf-4134-9361-e4c0bca6b525\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82ccfd-6b4c8757ff-rc85k" Oct 30 09:51:08.469447 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:08.463756 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q22z\" (UniqueName: \"kubernetes.io/projected/2e7713d8-4bdf-4134-9361-e4c0bca6b525-kube-api-access-8q22z\") pod \"payload-pre-processing-e2e-inf-a-82ccfd-6b4c8757ff-rc85k\" (UID: \"2e7713d8-4bdf-4134-9361-e4c0bca6b525\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82ccfd-6b4c8757ff-rc85k" Oct 30 09:51:08.469447 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:08.463958 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e7713d8-4bdf-4134-9361-e4c0bca6b525-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-82ccfd-6b4c8757ff-rc85k\" (UID: \"2e7713d8-4bdf-4134-9361-e4c0bca6b525\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82ccfd-6b4c8757ff-rc85k" Oct 30 09:51:08.469447 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:08.465186 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e7713d8-4bdf-4134-9361-e4c0bca6b525-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-82ccfd-6b4c8757ff-rc85k\" (UID: \"2e7713d8-4bdf-4134-9361-e4c0bca6b525\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82ccfd-6b4c8757ff-rc85k" Oct 30 09:51:08.556417 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:08.556316 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q22z\" (UniqueName: \"kubernetes.io/projected/2e7713d8-4bdf-4134-9361-e4c0bca6b525-kube-api-access-8q22z\") pod \"payload-pre-processing-e2e-inf-a-82ccfd-6b4c8757ff-rc85k\" (UID: \"2e7713d8-4bdf-4134-9361-e4c0bca6b525\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82ccfd-6b4c8757ff-rc85k" Oct 30 09:51:08.602634 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:08.598676 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82ccfd-6b4c8757ff-rc85k" Oct 30 09:51:08.953585 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:51:08.953552 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e7713d8_4bdf_4134_9361_e4c0bca6b525.slice/crio-70f203685be4c151ba806356655974f6a237ba927f5b46f70b5f8345e7de18b9 WatchSource:0}: Error finding container 70f203685be4c151ba806356655974f6a237ba927f5b46f70b5f8345e7de18b9: Status 404 returned error can't find the container with id 70f203685be4c151ba806356655974f6a237ba927f5b46f70b5f8345e7de18b9 Oct 30 09:51:08.957355 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:08.957327 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-82ccfd-6b4c8757ff-rc85k"] Oct 30 09:51:09.175677 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:09.175642 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82ccfd-6b4c8757ff-rc85k" event={"ID":"2e7713d8-4bdf-4134-9361-e4c0bca6b525","Type":"ContainerStarted","Data":"a8aa32d2203aa520c798dd93955b1022bf6d685380f68ad131ac93d6a131cc31"} Oct 30 09:51:09.175853 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:09.175694 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82ccfd-6b4c8757ff-rc85k" event={"ID":"2e7713d8-4bdf-4134-9361-e4c0bca6b525","Type":"ContainerStarted","Data":"70f203685be4c151ba806356655974f6a237ba927f5b46f70b5f8345e7de18b9"} Oct 30 09:51:09.175853 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:09.175837 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82ccfd-6b4c8757ff-rc85k" Oct 30 09:51:09.219307 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:09.219203 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82ccfd-6b4c8757ff-rc85k" podStartSLOduration=1.219189483 podStartE2EDuration="1.219189483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:51:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:51:09.216448641 +0000 UTC m=+3429.478009649" watchObservedRunningTime="2026-10-30 09:51:09.219189483 +0000 UTC m=+3429.480750470" Oct 30 09:51:13.134880 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:13.134849 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-56dc545b7c-d92fs" Oct 30 09:51:13.208038 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:13.208002 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5fbb64fb6b-mgvv7"] Oct 30 09:51:18.209115 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:18.209069 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5fbb64fb6b-mgvv7" podUID="f1218471-2eca-49fc-8efc-fc6c801b9ffd" containerName="payload-processing" containerID="cri-o://e7a459bbe3efb4b0d6783254ce43b8d0dd535d090d75fdace38abd6dbeb4e0f9" gracePeriod=25 Oct 30 09:51:18.838411 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:18.827664 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-extmod-shr-user-0ab35b/e2e-usage-sim-e02533-5d4ff8cf6d-455dp"] Oct 30 09:51:18.857748 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:18.854890 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-0ab35b/e2e-usage-sim-e02533-5d4ff8cf6d-455dp" Oct 30 09:51:18.860239 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:18.860164 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-0ab35b/e2e-usage-sim-e02533-5d4ff8cf6d-455dp"] Oct 30 09:51:18.882817 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:18.882440 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-0ab35b\"/\"kube-root-ca.crt\"" Oct 30 09:51:18.917422 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:18.900696 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-0ab35b\"/\"default-dockercfg-xw8qc\"" Oct 30 09:51:18.917422 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:18.900793 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-0ab35b\"/\"e2e-usage-sim-e02533-cert\"" Oct 30 09:51:18.917422 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:18.900873 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-0ab35b\"/\"openshift-service-ca.crt\"" Oct 30 09:51:19.061405 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:19.049135 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp9rg\" (UniqueName: \"kubernetes.io/projected/e32372d7-a2c3-44d4-9872-0ac6c3091bb9-kube-api-access-vp9rg\") pod \"e2e-usage-sim-e02533-5d4ff8cf6d-455dp\" (UID: \"e32372d7-a2c3-44d4-9872-0ac6c3091bb9\") " pod="e2e-extmod-shr-user-0ab35b/e2e-usage-sim-e02533-5d4ff8cf6d-455dp" Oct 30 09:51:19.061405 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:19.049229 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/e32372d7-a2c3-44d4-9872-0ac6c3091bb9-tls\") pod \"e2e-usage-sim-e02533-5d4ff8cf6d-455dp\" (UID: \"e32372d7-a2c3-44d4-9872-0ac6c3091bb9\") " pod="e2e-extmod-shr-user-0ab35b/e2e-usage-sim-e02533-5d4ff8cf6d-455dp" Oct 30 09:51:19.153028 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:19.152929 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp9rg\" (UniqueName: \"kubernetes.io/projected/e32372d7-a2c3-44d4-9872-0ac6c3091bb9-kube-api-access-vp9rg\") pod \"e2e-usage-sim-e02533-5d4ff8cf6d-455dp\" (UID: \"e32372d7-a2c3-44d4-9872-0ac6c3091bb9\") " pod="e2e-extmod-shr-user-0ab35b/e2e-usage-sim-e02533-5d4ff8cf6d-455dp" Oct 30 09:51:19.153216 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:19.153029 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/e32372d7-a2c3-44d4-9872-0ac6c3091bb9-tls\") pod \"e2e-usage-sim-e02533-5d4ff8cf6d-455dp\" (UID: \"e32372d7-a2c3-44d4-9872-0ac6c3091bb9\") " pod="e2e-extmod-shr-user-0ab35b/e2e-usage-sim-e02533-5d4ff8cf6d-455dp" Oct 30 09:51:19.157073 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:19.157049 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/e32372d7-a2c3-44d4-9872-0ac6c3091bb9-tls\") pod \"e2e-usage-sim-e02533-5d4ff8cf6d-455dp\" (UID: \"e32372d7-a2c3-44d4-9872-0ac6c3091bb9\") " pod="e2e-extmod-shr-user-0ab35b/e2e-usage-sim-e02533-5d4ff8cf6d-455dp" Oct 30 09:51:19.174212 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:19.174180 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp9rg\" (UniqueName: \"kubernetes.io/projected/e32372d7-a2c3-44d4-9872-0ac6c3091bb9-kube-api-access-vp9rg\") pod \"e2e-usage-sim-e02533-5d4ff8cf6d-455dp\" (UID: \"e32372d7-a2c3-44d4-9872-0ac6c3091bb9\") " pod="e2e-extmod-shr-user-0ab35b/e2e-usage-sim-e02533-5d4ff8cf6d-455dp" Oct 30 09:51:19.246410 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:19.244502 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-0ab35b/e2e-usage-sim-e02533-5d4ff8cf6d-455dp" Oct 30 09:51:19.654121 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:19.654080 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-0ab35b/e2e-usage-sim-e02533-5d4ff8cf6d-455dp"] Oct 30 09:51:19.665351 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:51:19.664583 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode32372d7_a2c3_44d4_9872_0ac6c3091bb9.slice/crio-9fb0297826715d78ff90ccb4dc6fbcaa80352bbfdd6208684f1ff807b8f7e31a WatchSource:0}: Error finding container 9fb0297826715d78ff90ccb4dc6fbcaa80352bbfdd6208684f1ff807b8f7e31a: Status 404 returned error can't find the container with id 9fb0297826715d78ff90ccb4dc6fbcaa80352bbfdd6208684f1ff807b8f7e31a Oct 30 09:51:20.094995 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:20.094962 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-85fdzxq9s"] Oct 30 09:51:20.100025 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:20.100003 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-85fdzxq9s" Oct 30 09:51:20.116296 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:20.116268 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3d9b36\"/\"agc-cred-244fab4a7add\"" Oct 30 09:51:20.118637 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:20.118612 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-85fdzxq9s"] Oct 30 09:51:20.187448 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:20.187418 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82ccfd-6b4c8757ff-rc85k" Oct 30 09:51:20.230465 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:20.228665 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06051f1f-c3c6-4c87-8cd4-1cb7b6c4f170-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-85fdzxq9s\" (UID: \"06051f1f-c3c6-4c87-8cd4-1cb7b6c4f170\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-85fdzxq9s" Oct 30 09:51:20.230465 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:20.229514 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/06051f1f-c3c6-4c87-8cd4-1cb7b6c4f170-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-85fdzxq9s\" (UID: \"06051f1f-c3c6-4c87-8cd4-1cb7b6c4f170\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-85fdzxq9s" Oct 30 09:51:20.230465 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:20.229642 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/06051f1f-c3c6-4c87-8cd4-1cb7b6c4f170-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-85fdzxq9s\" (UID: \"06051f1f-c3c6-4c87-8cd4-1cb7b6c4f170\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-85fdzxq9s" Oct 30 09:51:20.230465 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:20.229794 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9rv7\" (UniqueName: \"kubernetes.io/projected/06051f1f-c3c6-4c87-8cd4-1cb7b6c4f170-kube-api-access-l9rv7\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-85fdzxq9s\" (UID: \"06051f1f-c3c6-4c87-8cd4-1cb7b6c4f170\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-85fdzxq9s" Oct 30 09:51:20.333994 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:20.333953 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/06051f1f-c3c6-4c87-8cd4-1cb7b6c4f170-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-85fdzxq9s\" (UID: \"06051f1f-c3c6-4c87-8cd4-1cb7b6c4f170\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-85fdzxq9s" Oct 30 09:51:20.335450 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:20.335295 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/06051f1f-c3c6-4c87-8cd4-1cb7b6c4f170-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-85fdzxq9s\" (UID: \"06051f1f-c3c6-4c87-8cd4-1cb7b6c4f170\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-85fdzxq9s" Oct 30 09:51:20.335690 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:20.335659 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/06051f1f-c3c6-4c87-8cd4-1cb7b6c4f170-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-85fdzxq9s\" (UID: \"06051f1f-c3c6-4c87-8cd4-1cb7b6c4f170\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-85fdzxq9s" Oct 30 09:51:20.335863 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:20.335842 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9rv7\" (UniqueName: \"kubernetes.io/projected/06051f1f-c3c6-4c87-8cd4-1cb7b6c4f170-kube-api-access-l9rv7\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-85fdzxq9s\" (UID: \"06051f1f-c3c6-4c87-8cd4-1cb7b6c4f170\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-85fdzxq9s" Oct 30 09:51:20.336266 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:20.336040 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06051f1f-c3c6-4c87-8cd4-1cb7b6c4f170-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-85fdzxq9s\" (UID: \"06051f1f-c3c6-4c87-8cd4-1cb7b6c4f170\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-85fdzxq9s" Oct 30 09:51:20.336917 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:20.336894 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06051f1f-c3c6-4c87-8cd4-1cb7b6c4f170-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-85fdzxq9s\" (UID: \"06051f1f-c3c6-4c87-8cd4-1cb7b6c4f170\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-85fdzxq9s" Oct 30 09:51:20.347914 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:20.347252 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/06051f1f-c3c6-4c87-8cd4-1cb7b6c4f170-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-85fdzxq9s\" (UID: \"06051f1f-c3c6-4c87-8cd4-1cb7b6c4f170\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-85fdzxq9s" Oct 30 09:51:20.360051 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:20.360018 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9rv7\" (UniqueName: \"kubernetes.io/projected/06051f1f-c3c6-4c87-8cd4-1cb7b6c4f170-kube-api-access-l9rv7\") pod \"payload-processing-external-model-e2e-extmod-b-3d9b36-85fdzxq9s\" (UID: \"06051f1f-c3c6-4c87-8cd4-1cb7b6c4f170\") " pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-85fdzxq9s" Oct 30 09:51:20.363421 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:20.363254 4562 generic.go:356] "Generic (PLEG): container finished" podID="f1218471-2eca-49fc-8efc-fc6c801b9ffd" containerID="e7a459bbe3efb4b0d6783254ce43b8d0dd535d090d75fdace38abd6dbeb4e0f9" exitCode=0 Oct 30 09:51:20.363648 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:20.363374 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5fbb64fb6b-mgvv7" event={"ID":"f1218471-2eca-49fc-8efc-fc6c801b9ffd","Type":"ContainerDied","Data":"e7a459bbe3efb4b0d6783254ce43b8d0dd535d090d75fdace38abd6dbeb4e0f9"} Oct 30 09:51:20.366443 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:20.366417 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-0ab35b/e2e-usage-sim-e02533-5d4ff8cf6d-455dp" event={"ID":"e32372d7-a2c3-44d4-9872-0ac6c3091bb9","Type":"ContainerStarted","Data":"9de20d9f01f9beef813e3a740e1a1ee8d4fd313f28f0d838c83a2aa488934283"} Oct 30 09:51:20.370449 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:20.370428 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-0ab35b/e2e-usage-sim-e02533-5d4ff8cf6d-455dp" event={"ID":"e32372d7-a2c3-44d4-9872-0ac6c3091bb9","Type":"ContainerStarted","Data":"9fb0297826715d78ff90ccb4dc6fbcaa80352bbfdd6208684f1ff807b8f7e31a"} Oct 30 09:51:20.370624 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:20.370608 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-extmod-shr-user-0ab35b/e2e-usage-sim-e02533-5d4ff8cf6d-455dp" Oct 30 09:51:20.373511 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:20.373491 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-extmod-shr-user-0ab35b/e2e-usage-sim-e02533-5d4ff8cf6d-455dp" Oct 30 09:51:20.412758 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:20.412694 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-extmod-shr-user-0ab35b/e2e-usage-sim-e02533-5d4ff8cf6d-455dp" podStartSLOduration=2.412675265 podStartE2EDuration="2.412675265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:51:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:51:20.407638132 +0000 UTC m=+3440.669199121" watchObservedRunningTime="2026-10-30 09:51:20.412675265 +0000 UTC m=+3440.674236253" Oct 30 09:51:20.417149 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:20.417124 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-85fdzxq9s" Oct 30 09:51:20.475574 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:20.475535 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5fbb64fb6b-mgvv7" Oct 30 09:51:20.539892 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:20.539853 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1218471-2eca-49fc-8efc-fc6c801b9ffd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1218471-2eca-49fc-8efc-fc6c801b9ffd-plugins-config-volume\") pod \"f1218471-2eca-49fc-8efc-fc6c801b9ffd\" (UID: \"f1218471-2eca-49fc-8efc-fc6c801b9ffd\") " Oct 30 09:51:20.540097 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:20.539919 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1218471-2eca-49fc-8efc-fc6c801b9ffd-kube-api-access-2mshn\" (UniqueName: \"kubernetes.io/projected/f1218471-2eca-49fc-8efc-fc6c801b9ffd-kube-api-access-2mshn\") pod \"f1218471-2eca-49fc-8efc-fc6c801b9ffd\" (UID: \"f1218471-2eca-49fc-8efc-fc6c801b9ffd\") " Oct 30 09:51:20.540097 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:20.540014 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1218471-2eca-49fc-8efc-fc6c801b9ffd-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f1218471-2eca-49fc-8efc-fc6c801b9ffd-routing-overlay-volume\") pod \"f1218471-2eca-49fc-8efc-fc6c801b9ffd\" (UID: \"f1218471-2eca-49fc-8efc-fc6c801b9ffd\") " Oct 30 09:51:20.540097 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:20.540078 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1218471-2eca-49fc-8efc-fc6c801b9ffd-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f1218471-2eca-49fc-8efc-fc6c801b9ffd-provider-credentials\") pod \"f1218471-2eca-49fc-8efc-fc6c801b9ffd\" (UID: \"f1218471-2eca-49fc-8efc-fc6c801b9ffd\") " Oct 30 09:51:20.540841 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:20.540812 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1218471-2eca-49fc-8efc-fc6c801b9ffd-routing-overlay-volume" pod "f1218471-2eca-49fc-8efc-fc6c801b9ffd" (UID: "f1218471-2eca-49fc-8efc-fc6c801b9ffd"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:51:20.541160 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:20.541075 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1218471-2eca-49fc-8efc-fc6c801b9ffd-plugins-config-volume" pod "f1218471-2eca-49fc-8efc-fc6c801b9ffd" (UID: "f1218471-2eca-49fc-8efc-fc6c801b9ffd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:51:20.543218 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:20.543189 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1218471-2eca-49fc-8efc-fc6c801b9ffd-kube-api-access-2mshn" pod "f1218471-2eca-49fc-8efc-fc6c801b9ffd" (UID: "f1218471-2eca-49fc-8efc-fc6c801b9ffd"). InnerVolumeSpecName "kube-api-access-2mshn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:51:20.544129 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:20.544109 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1218471-2eca-49fc-8efc-fc6c801b9ffd-provider-credentials" pod "f1218471-2eca-49fc-8efc-fc6c801b9ffd" (UID: "f1218471-2eca-49fc-8efc-fc6c801b9ffd"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:51:20.687214 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:20.687063 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1218471-2eca-49fc-8efc-fc6c801b9ffd-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:51:20.687214 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:20.687098 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mshn\" (UniqueName: \"kubernetes.io/projected/f1218471-2eca-49fc-8efc-fc6c801b9ffd-kube-api-access-2mshn\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:51:20.687214 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:20.687109 4562 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f1218471-2eca-49fc-8efc-fc6c801b9ffd-routing-overlay-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:51:20.687214 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:20.687119 4562 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f1218471-2eca-49fc-8efc-fc6c801b9ffd-provider-credentials\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:51:20.743959 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:51:20.743929 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06051f1f_c3c6_4c87_8cd4_1cb7b6c4f170.slice/crio-22eb32d2a142179e86fc7b2de5f0493dbb93c8edea9b6405155935be6886f6c1 WatchSource:0}: Error finding container 22eb32d2a142179e86fc7b2de5f0493dbb93c8edea9b6405155935be6886f6c1: Status 404 returned error can't find the container with id 22eb32d2a142179e86fc7b2de5f0493dbb93c8edea9b6405155935be6886f6c1 Oct 30 09:51:20.745879 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:20.745855 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-85fdzxq9s"] Oct 30 09:51:20.973622 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:20.973591 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-7dc7rbmf2"] Oct 30 09:51:20.974486 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:20.974469 4562 cpu_manager.go:414] "containerMap: removing container" podUID="f1218471-2eca-49fc-8efc-fc6c801b9ffd" containerName="payload-processing" Oct 30 09:51:20.974550 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:20.974487 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1218471-2eca-49fc-8efc-fc6c801b9ffd" containerName="payload-processing" Oct 30 09:51:20.974643 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:20.974634 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1218471-2eca-49fc-8efc-fc6c801b9ffd" containerName="payload-processing" Oct 30 09:51:20.978369 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:20.978354 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-7dc7rbmf2" Oct 30 09:51:20.993656 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:20.993613 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d60c68\"/\"agc-cred-7655d9e07e16\"" Oct 30 09:51:20.993792 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:20.993745 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d60c68\"/\"kube-root-ca.crt\"" Oct 30 09:51:20.993792 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:20.993757 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d60c68\"/\"payload-processing-external-model-e2e--2110ea34-dockercfg-68rjw\"" Oct 30 09:51:20.993792 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:20.993766 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d60c68\"/\"openshift-service-ca.crt\"" Oct 30 09:51:20.993792 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:20.993760 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d60c68\"/\"routing-overlay\"" Oct 30 09:51:20.994008 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:20.993808 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d60c68\"/\"agc-cred-448d95ada9ba\"" Oct 30 09:51:20.994008 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:20.993811 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d60c68\"/\"payload-processing-external-model-plugins-e2e-extmod-a-d60c68\"" Oct 30 09:51:20.996956 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:20.996932 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-7dc7rbmf2"] Oct 30 09:51:21.092969 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:21.092936 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w5w5\" (UniqueName: \"kubernetes.io/projected/b1a4232c-bb1f-4fb4-921a-8ada7bc47d6c-kube-api-access-7w5w5\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-7dc7rbmf2\" (UID: \"b1a4232c-bb1f-4fb4-921a-8ada7bc47d6c\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-7dc7rbmf2" Oct 30 09:51:21.093139 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:21.093110 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1a4232c-bb1f-4fb4-921a-8ada7bc47d6c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-7dc7rbmf2\" (UID: \"b1a4232c-bb1f-4fb4-921a-8ada7bc47d6c\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-7dc7rbmf2" Oct 30 09:51:21.093190 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:21.093161 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b1a4232c-bb1f-4fb4-921a-8ada7bc47d6c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-7dc7rbmf2\" (UID: \"b1a4232c-bb1f-4fb4-921a-8ada7bc47d6c\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-7dc7rbmf2" Oct 30 09:51:21.093300 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:21.093283 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b1a4232c-bb1f-4fb4-921a-8ada7bc47d6c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-7dc7rbmf2\" (UID: \"b1a4232c-bb1f-4fb4-921a-8ada7bc47d6c\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-7dc7rbmf2" Oct 30 09:51:21.194431 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:21.194375 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w5w5\" (UniqueName: \"kubernetes.io/projected/b1a4232c-bb1f-4fb4-921a-8ada7bc47d6c-kube-api-access-7w5w5\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-7dc7rbmf2\" (UID: \"b1a4232c-bb1f-4fb4-921a-8ada7bc47d6c\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-7dc7rbmf2" Oct 30 09:51:21.194604 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:21.194586 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1a4232c-bb1f-4fb4-921a-8ada7bc47d6c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-7dc7rbmf2\" (UID: \"b1a4232c-bb1f-4fb4-921a-8ada7bc47d6c\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-7dc7rbmf2" Oct 30 09:51:21.194670 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:21.194657 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b1a4232c-bb1f-4fb4-921a-8ada7bc47d6c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-7dc7rbmf2\" (UID: \"b1a4232c-bb1f-4fb4-921a-8ada7bc47d6c\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-7dc7rbmf2" Oct 30 09:51:21.194810 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:21.194791 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b1a4232c-bb1f-4fb4-921a-8ada7bc47d6c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-7dc7rbmf2\" (UID: \"b1a4232c-bb1f-4fb4-921a-8ada7bc47d6c\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-7dc7rbmf2" Oct 30 09:51:21.195306 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:21.195282 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b1a4232c-bb1f-4fb4-921a-8ada7bc47d6c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-7dc7rbmf2\" (UID: \"b1a4232c-bb1f-4fb4-921a-8ada7bc47d6c\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-7dc7rbmf2" Oct 30 09:51:21.195416 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:21.195398 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1a4232c-bb1f-4fb4-921a-8ada7bc47d6c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-7dc7rbmf2\" (UID: \"b1a4232c-bb1f-4fb4-921a-8ada7bc47d6c\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-7dc7rbmf2" Oct 30 09:51:21.197300 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:21.197281 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b1a4232c-bb1f-4fb4-921a-8ada7bc47d6c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-7dc7rbmf2\" (UID: \"b1a4232c-bb1f-4fb4-921a-8ada7bc47d6c\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-7dc7rbmf2" Oct 30 09:51:21.214171 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:21.214141 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w5w5\" (UniqueName: \"kubernetes.io/projected/b1a4232c-bb1f-4fb4-921a-8ada7bc47d6c-kube-api-access-7w5w5\") pod \"payload-processing-external-model-e2e-extmod-a-d60c68-7dc7rbmf2\" (UID: \"b1a4232c-bb1f-4fb4-921a-8ada7bc47d6c\") " pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-7dc7rbmf2" Oct 30 09:51:21.293476 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:21.293409 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-7dc7rbmf2" Oct 30 09:51:21.385161 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:21.384499 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5fbb64fb6b-mgvv7" event={"ID":"f1218471-2eca-49fc-8efc-fc6c801b9ffd","Type":"ContainerDied","Data":"c7c9bc95158ce9471b711ea3194776f85ebac6a093ed7693c02b33a03b42a2ea"} Oct 30 09:51:21.385161 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:21.384572 4562 scope.go:122] "RemoveContainer" containerID="e7a459bbe3efb4b0d6783254ce43b8d0dd535d090d75fdace38abd6dbeb4e0f9" Oct 30 09:51:21.387108 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:21.387088 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5fbb64fb6b-mgvv7" Oct 30 09:51:21.389018 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:21.388998 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-85fdzxq9s" event={"ID":"06051f1f-c3c6-4c87-8cd4-1cb7b6c4f170","Type":"ContainerStarted","Data":"d9b90853d7460963a64fd012b39b8edeb59a925e121feaea142da39265370fbc"} Oct 30 09:51:21.389235 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:21.389189 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-85fdzxq9s" event={"ID":"06051f1f-c3c6-4c87-8cd4-1cb7b6c4f170","Type":"ContainerStarted","Data":"22eb32d2a142179e86fc7b2de5f0493dbb93c8edea9b6405155935be6886f6c1"} Oct 30 09:51:21.389449 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:21.389432 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-85fdzxq9s" Oct 30 09:51:21.443272 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:21.443239 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5fbb64fb6b-mgvv7"] Oct 30 09:51:21.447760 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:21.447732 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5fbb64fb6b-mgvv7"] Oct 30 09:51:21.485478 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:21.485420 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-85fdzxq9s" podStartSLOduration=1.485406148 podStartE2EDuration="1.485406148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:51:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:51:21.481580176 +0000 UTC m=+3441.743141183" watchObservedRunningTime="2026-10-30 09:51:21.485406148 +0000 UTC m=+3441.746967166" Oct 30 09:51:21.602423 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:51:21.602377 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1a4232c_bb1f_4fb4_921a_8ada7bc47d6c.slice/crio-3aa3eda39dc98611400c9c54c879e99edb6c116f689cddf8f00fc68ad00e9840 WatchSource:0}: Error finding container 3aa3eda39dc98611400c9c54c879e99edb6c116f689cddf8f00fc68ad00e9840: Status 404 returned error can't find the container with id 3aa3eda39dc98611400c9c54c879e99edb6c116f689cddf8f00fc68ad00e9840 Oct 30 09:51:21.604358 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:21.604335 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-7dc7rbmf2"] Oct 30 09:51:21.887294 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:21.887205 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1218471-2eca-49fc-8efc-fc6c801b9ffd" path="/var/lib/kubelet/pods/f1218471-2eca-49fc-8efc-fc6c801b9ffd/volumes" Oct 30 09:51:22.400562 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:22.400523 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-7dc7rbmf2" event={"ID":"b1a4232c-bb1f-4fb4-921a-8ada7bc47d6c","Type":"ContainerStarted","Data":"1fb3e96233b04078b673692c17b3584359b8e8cac26f254330a2de9ddf9b3953"} Oct 30 09:51:22.401034 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:22.400583 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-7dc7rbmf2" event={"ID":"b1a4232c-bb1f-4fb4-921a-8ada7bc47d6c","Type":"ContainerStarted","Data":"3aa3eda39dc98611400c9c54c879e99edb6c116f689cddf8f00fc68ad00e9840"} Oct 30 09:51:22.401034 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:22.400618 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-7dc7rbmf2" Oct 30 09:51:22.445032 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:22.444981 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-7dc7rbmf2" podStartSLOduration=2.444968251 podStartE2EDuration="2.444968251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:51:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:51:22.440925814 +0000 UTC m=+3442.702486802" watchObservedRunningTime="2026-10-30 09:51:22.444968251 +0000 UTC m=+3442.706529238" Oct 30 09:51:28.598189 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:51:28.598153 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:51:32.405106 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:32.405072 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-85fdzxq9s" Oct 30 09:51:32.479506 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:32.479474 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5cb55trxn"] Oct 30 09:51:33.412985 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:33.412958 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-7dc7rbmf2" Oct 30 09:51:37.480223 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:37.480177 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5cb55trxn" podUID="7f7614a9-e7c1-4340-bf3e-4db072e415e7" containerName="payload-processing" containerID="cri-o://25678d9729f7cc64eca9f2b230a6ca77a58e41cae8a8c4753fad262b706fe445" gracePeriod=25 Oct 30 09:51:37.770657 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:37.770624 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5cb55trxn" Oct 30 09:51:37.935131 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:37.935100 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f7614a9-e7c1-4340-bf3e-4db072e415e7-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7f7614a9-e7c1-4340-bf3e-4db072e415e7-routing-overlay-volume\") pod \"7f7614a9-e7c1-4340-bf3e-4db072e415e7\" (UID: \"7f7614a9-e7c1-4340-bf3e-4db072e415e7\") " Oct 30 09:51:37.935332 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:37.935143 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f7614a9-e7c1-4340-bf3e-4db072e415e7-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7f7614a9-e7c1-4340-bf3e-4db072e415e7-provider-credentials\") pod \"7f7614a9-e7c1-4340-bf3e-4db072e415e7\" (UID: \"7f7614a9-e7c1-4340-bf3e-4db072e415e7\") " Oct 30 09:51:37.935332 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:37.935197 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f7614a9-e7c1-4340-bf3e-4db072e415e7-kube-api-access-dzhz8\" (UniqueName: \"kubernetes.io/projected/7f7614a9-e7c1-4340-bf3e-4db072e415e7-kube-api-access-dzhz8\") pod \"7f7614a9-e7c1-4340-bf3e-4db072e415e7\" (UID: \"7f7614a9-e7c1-4340-bf3e-4db072e415e7\") " Oct 30 09:51:37.935332 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:37.935327 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f7614a9-e7c1-4340-bf3e-4db072e415e7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f7614a9-e7c1-4340-bf3e-4db072e415e7-plugins-config-volume\") pod \"7f7614a9-e7c1-4340-bf3e-4db072e415e7\" (UID: \"7f7614a9-e7c1-4340-bf3e-4db072e415e7\") " Oct 30 09:51:37.935685 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:37.935550 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f7614a9-e7c1-4340-bf3e-4db072e415e7-routing-overlay-volume" pod "7f7614a9-e7c1-4340-bf3e-4db072e415e7" (UID: "7f7614a9-e7c1-4340-bf3e-4db072e415e7"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:51:37.935803 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:37.935752 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f7614a9-e7c1-4340-bf3e-4db072e415e7-plugins-config-volume" pod "7f7614a9-e7c1-4340-bf3e-4db072e415e7" (UID: "7f7614a9-e7c1-4340-bf3e-4db072e415e7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:51:37.936115 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:37.936095 4562 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7f7614a9-e7c1-4340-bf3e-4db072e415e7-routing-overlay-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:51:37.936173 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:37.936118 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f7614a9-e7c1-4340-bf3e-4db072e415e7-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:51:37.937544 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:37.937517 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f7614a9-e7c1-4340-bf3e-4db072e415e7-kube-api-access-dzhz8" pod "7f7614a9-e7c1-4340-bf3e-4db072e415e7" (UID: "7f7614a9-e7c1-4340-bf3e-4db072e415e7"). InnerVolumeSpecName "kube-api-access-dzhz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:51:37.937635 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:37.937592 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f7614a9-e7c1-4340-bf3e-4db072e415e7-provider-credentials" pod "7f7614a9-e7c1-4340-bf3e-4db072e415e7" (UID: "7f7614a9-e7c1-4340-bf3e-4db072e415e7"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:51:37.971220 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:37.971186 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-f76d9mwcj"] Oct 30 09:51:38.038369 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:38.038275 4562 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7f7614a9-e7c1-4340-bf3e-4db072e415e7-provider-credentials\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:51:38.038369 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:38.038302 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzhz8\" (UniqueName: \"kubernetes.io/projected/7f7614a9-e7c1-4340-bf3e-4db072e415e7-kube-api-access-dzhz8\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:51:38.548482 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:38.548442 4562 generic.go:356] "Generic (PLEG): container finished" podID="7f7614a9-e7c1-4340-bf3e-4db072e415e7" containerID="25678d9729f7cc64eca9f2b230a6ca77a58e41cae8a8c4753fad262b706fe445" exitCode=0 Oct 30 09:51:38.548967 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:38.548491 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5cb55trxn" event={"ID":"7f7614a9-e7c1-4340-bf3e-4db072e415e7","Type":"ContainerDied","Data":"25678d9729f7cc64eca9f2b230a6ca77a58e41cae8a8c4753fad262b706fe445"} Oct 30 09:51:38.548967 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:38.548495 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5cb55trxn" Oct 30 09:51:38.548967 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:38.548539 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5cb55trxn" event={"ID":"7f7614a9-e7c1-4340-bf3e-4db072e415e7","Type":"ContainerDied","Data":"81265971e495d9c0613929f6a495e8d37c8dacd5d00a4ce62eed4f9e4f3c42b6"} Oct 30 09:51:38.548967 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:38.548565 4562 scope.go:122] "RemoveContainer" containerID="25678d9729f7cc64eca9f2b230a6ca77a58e41cae8a8c4753fad262b706fe445" Oct 30 09:51:38.564405 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:38.562184 4562 scope.go:122] "RemoveContainer" containerID="25678d9729f7cc64eca9f2b230a6ca77a58e41cae8a8c4753fad262b706fe445" Oct 30 09:51:38.564753 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:51:38.564728 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25678d9729f7cc64eca9f2b230a6ca77a58e41cae8a8c4753fad262b706fe445\": container with ID starting with 25678d9729f7cc64eca9f2b230a6ca77a58e41cae8a8c4753fad262b706fe445 not found: ID does not exist" containerID="25678d9729f7cc64eca9f2b230a6ca77a58e41cae8a8c4753fad262b706fe445" Oct 30 09:51:38.564827 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:38.564762 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25678d9729f7cc64eca9f2b230a6ca77a58e41cae8a8c4753fad262b706fe445"} err="failed to get container status \"25678d9729f7cc64eca9f2b230a6ca77a58e41cae8a8c4753fad262b706fe445\": rpc error: code = NotFound desc = could not find container \"25678d9729f7cc64eca9f2b230a6ca77a58e41cae8a8c4753fad262b706fe445\": container with ID starting with 25678d9729f7cc64eca9f2b230a6ca77a58e41cae8a8c4753fad262b706fe445 not found: ID does not exist" Oct 30 09:51:38.604031 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:38.604003 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5cb55trxn"] Oct 30 09:51:38.616537 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:38.616488 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-5cb55trxn"] Oct 30 09:51:39.878567 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:39.878532 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f7614a9-e7c1-4340-bf3e-4db072e415e7" path="/var/lib/kubelet/pods/7f7614a9-e7c1-4340-bf3e-4db072e415e7/volumes" Oct 30 09:51:42.972005 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:42.971963 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-f76d9mwcj" podUID="315c3134-24c5-4106-877b-8834a97862dc" containerName="payload-processing" containerID="cri-o://a782dd9de13c844f4a6ed7d1d6f5e93c39e15d115f6c998a97ee92086ae99260" gracePeriod=25 Oct 30 09:51:43.276933 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:43.276903 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-f76d9mwcj" Oct 30 09:51:43.322744 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:43.322700 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/315c3134-24c5-4106-877b-8834a97862dc-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/315c3134-24c5-4106-877b-8834a97862dc-routing-overlay-volume\") pod \"315c3134-24c5-4106-877b-8834a97862dc\" (UID: \"315c3134-24c5-4106-877b-8834a97862dc\") " Oct 30 09:51:43.322937 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:43.322792 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/315c3134-24c5-4106-877b-8834a97862dc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/315c3134-24c5-4106-877b-8834a97862dc-plugins-config-volume\") pod \"315c3134-24c5-4106-877b-8834a97862dc\" (UID: \"315c3134-24c5-4106-877b-8834a97862dc\") " Oct 30 09:51:43.323069 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:43.323048 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/315c3134-24c5-4106-877b-8834a97862dc-kube-api-access-bdcbc\" (UniqueName: \"kubernetes.io/projected/315c3134-24c5-4106-877b-8834a97862dc-kube-api-access-bdcbc\") pod \"315c3134-24c5-4106-877b-8834a97862dc\" (UID: \"315c3134-24c5-4106-877b-8834a97862dc\") " Oct 30 09:51:43.323151 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:43.323122 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/315c3134-24c5-4106-877b-8834a97862dc-routing-overlay-volume" pod "315c3134-24c5-4106-877b-8834a97862dc" (UID: "315c3134-24c5-4106-877b-8834a97862dc"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:51:43.323236 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:43.323131 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/315c3134-24c5-4106-877b-8834a97862dc-provider-credentials\" (UniqueName: \"kubernetes.io/projected/315c3134-24c5-4106-877b-8834a97862dc-provider-credentials\") pod \"315c3134-24c5-4106-877b-8834a97862dc\" (UID: \"315c3134-24c5-4106-877b-8834a97862dc\") " Oct 30 09:51:43.323300 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:43.323226 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/315c3134-24c5-4106-877b-8834a97862dc-plugins-config-volume" pod "315c3134-24c5-4106-877b-8834a97862dc" (UID: "315c3134-24c5-4106-877b-8834a97862dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:51:43.325873 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:43.325844 4562 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/315c3134-24c5-4106-877b-8834a97862dc-routing-overlay-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:51:43.326003 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:43.325877 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/315c3134-24c5-4106-877b-8834a97862dc-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:51:43.335547 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:43.334940 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/315c3134-24c5-4106-877b-8834a97862dc-kube-api-access-bdcbc" pod "315c3134-24c5-4106-877b-8834a97862dc" (UID: "315c3134-24c5-4106-877b-8834a97862dc"). InnerVolumeSpecName "kube-api-access-bdcbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:51:43.341122 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:43.341088 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/315c3134-24c5-4106-877b-8834a97862dc-provider-credentials" pod "315c3134-24c5-4106-877b-8834a97862dc" (UID: "315c3134-24c5-4106-877b-8834a97862dc"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:51:43.430429 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:43.430361 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdcbc\" (UniqueName: \"kubernetes.io/projected/315c3134-24c5-4106-877b-8834a97862dc-kube-api-access-bdcbc\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:51:43.430429 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:43.430432 4562 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/315c3134-24c5-4106-877b-8834a97862dc-provider-credentials\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:51:43.592286 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:43.592190 4562 generic.go:356] "Generic (PLEG): container finished" podID="315c3134-24c5-4106-877b-8834a97862dc" containerID="a782dd9de13c844f4a6ed7d1d6f5e93c39e15d115f6c998a97ee92086ae99260" exitCode=0 Oct 30 09:51:43.592286 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:43.592263 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-f76d9mwcj" Oct 30 09:51:43.592598 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:43.592279 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-f76d9mwcj" event={"ID":"315c3134-24c5-4106-877b-8834a97862dc","Type":"ContainerDied","Data":"a782dd9de13c844f4a6ed7d1d6f5e93c39e15d115f6c998a97ee92086ae99260"} Oct 30 09:51:43.592598 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:43.592362 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-f76d9mwcj" event={"ID":"315c3134-24c5-4106-877b-8834a97862dc","Type":"ContainerDied","Data":"e18e2e612a9a21b2988648b10c169e7e8c434f858057df527b88347bb513c338"} Oct 30 09:51:43.592598 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:43.592418 4562 scope.go:122] "RemoveContainer" containerID="a782dd9de13c844f4a6ed7d1d6f5e93c39e15d115f6c998a97ee92086ae99260" Oct 30 09:51:43.605946 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:43.605923 4562 scope.go:122] "RemoveContainer" containerID="a782dd9de13c844f4a6ed7d1d6f5e93c39e15d115f6c998a97ee92086ae99260" Oct 30 09:51:43.606306 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:51:43.606282 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a782dd9de13c844f4a6ed7d1d6f5e93c39e15d115f6c998a97ee92086ae99260\": container with ID starting with a782dd9de13c844f4a6ed7d1d6f5e93c39e15d115f6c998a97ee92086ae99260 not found: ID does not exist" containerID="a782dd9de13c844f4a6ed7d1d6f5e93c39e15d115f6c998a97ee92086ae99260" Oct 30 09:51:43.606419 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:43.606332 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a782dd9de13c844f4a6ed7d1d6f5e93c39e15d115f6c998a97ee92086ae99260"} err="failed to get container status \"a782dd9de13c844f4a6ed7d1d6f5e93c39e15d115f6c998a97ee92086ae99260\": rpc error: code = NotFound desc = could not find container \"a782dd9de13c844f4a6ed7d1d6f5e93c39e15d115f6c998a97ee92086ae99260\": container with ID starting with a782dd9de13c844f4a6ed7d1d6f5e93c39e15d115f6c998a97ee92086ae99260 not found: ID does not exist" Oct 30 09:51:43.651090 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:43.651053 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-f76d9mwcj"] Oct 30 09:51:43.655868 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:43.655839 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-f76d9mwcj"] Oct 30 09:51:43.877677 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:43.877590 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="315c3134-24c5-4106-877b-8834a97862dc" path="/var/lib/kubelet/pods/315c3134-24c5-4106-877b-8834a97862dc/volumes" Oct 30 09:51:50.982569 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:50.982537 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-82ccfd-6df95cf9bc-9nhdz"] Oct 30 09:51:50.983410 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:50.983391 4562 cpu_manager.go:414] "containerMap: removing container" podUID="315c3134-24c5-4106-877b-8834a97862dc" containerName="payload-processing" Oct 30 09:51:50.983469 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:50.983410 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="315c3134-24c5-4106-877b-8834a97862dc" containerName="payload-processing" Oct 30 09:51:50.983469 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:50.983437 4562 cpu_manager.go:414] "containerMap: removing container" podUID="7f7614a9-e7c1-4340-bf3e-4db072e415e7" containerName="payload-processing" Oct 30 09:51:50.983469 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:50.983442 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f7614a9-e7c1-4340-bf3e-4db072e415e7" containerName="payload-processing" Oct 30 09:51:50.983589 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:50.983580 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f7614a9-e7c1-4340-bf3e-4db072e415e7" containerName="payload-processing" Oct 30 09:51:50.983627 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:50.983601 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="315c3134-24c5-4106-877b-8834a97862dc" containerName="payload-processing" Oct 30 09:51:50.988894 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:50.988874 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82ccfd-6df95cf9bc-9nhdz" Oct 30 09:51:51.007605 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:51.007552 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-82ccfd-6df95cf9bc-9nhdz"] Oct 30 09:51:51.014228 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:51.014201 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-82ccfd-serving-cert\"" Oct 30 09:51:51.148088 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:51.148059 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/723c7f11-8c4d-4095-a4c4-fd76af32865c-maas-api-tls\") pod \"maas-api-e2e-inf-a-82ccfd-6df95cf9bc-9nhdz\" (UID: \"723c7f11-8c4d-4095-a4c4-fd76af32865c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82ccfd-6df95cf9bc-9nhdz" Oct 30 09:51:51.148254 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:51.148110 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/723c7f11-8c4d-4095-a4c4-fd76af32865c-metrics-certs\") pod \"maas-api-e2e-inf-a-82ccfd-6df95cf9bc-9nhdz\" (UID: \"723c7f11-8c4d-4095-a4c4-fd76af32865c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82ccfd-6df95cf9bc-9nhdz" Oct 30 09:51:51.148330 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:51.148305 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvdgw\" (UniqueName: \"kubernetes.io/projected/723c7f11-8c4d-4095-a4c4-fd76af32865c-kube-api-access-lvdgw\") pod \"maas-api-e2e-inf-a-82ccfd-6df95cf9bc-9nhdz\" (UID: \"723c7f11-8c4d-4095-a4c4-fd76af32865c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82ccfd-6df95cf9bc-9nhdz" Oct 30 09:51:51.148374 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:51.148366 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/723c7f11-8c4d-4095-a4c4-fd76af32865c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-82ccfd-6df95cf9bc-9nhdz\" (UID: \"723c7f11-8c4d-4095-a4c4-fd76af32865c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82ccfd-6df95cf9bc-9nhdz" Oct 30 09:51:51.258328 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:51.253483 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvdgw\" (UniqueName: \"kubernetes.io/projected/723c7f11-8c4d-4095-a4c4-fd76af32865c-kube-api-access-lvdgw\") pod \"maas-api-e2e-inf-a-82ccfd-6df95cf9bc-9nhdz\" (UID: \"723c7f11-8c4d-4095-a4c4-fd76af32865c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82ccfd-6df95cf9bc-9nhdz" Oct 30 09:51:51.258328 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:51.253587 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/723c7f11-8c4d-4095-a4c4-fd76af32865c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-82ccfd-6df95cf9bc-9nhdz\" (UID: \"723c7f11-8c4d-4095-a4c4-fd76af32865c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82ccfd-6df95cf9bc-9nhdz" Oct 30 09:51:51.258328 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:51.253853 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/723c7f11-8c4d-4095-a4c4-fd76af32865c-maas-api-tls\") pod \"maas-api-e2e-inf-a-82ccfd-6df95cf9bc-9nhdz\" (UID: \"723c7f11-8c4d-4095-a4c4-fd76af32865c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82ccfd-6df95cf9bc-9nhdz" Oct 30 09:51:51.258328 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:51.253904 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/723c7f11-8c4d-4095-a4c4-fd76af32865c-metrics-certs\") pod \"maas-api-e2e-inf-a-82ccfd-6df95cf9bc-9nhdz\" (UID: \"723c7f11-8c4d-4095-a4c4-fd76af32865c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82ccfd-6df95cf9bc-9nhdz" Oct 30 09:51:51.258328 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:51.255326 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/723c7f11-8c4d-4095-a4c4-fd76af32865c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-82ccfd-6df95cf9bc-9nhdz\" (UID: \"723c7f11-8c4d-4095-a4c4-fd76af32865c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82ccfd-6df95cf9bc-9nhdz" Oct 30 09:51:51.262815 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:51.262742 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/723c7f11-8c4d-4095-a4c4-fd76af32865c-metrics-certs\") pod \"maas-api-e2e-inf-a-82ccfd-6df95cf9bc-9nhdz\" (UID: \"723c7f11-8c4d-4095-a4c4-fd76af32865c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82ccfd-6df95cf9bc-9nhdz" Oct 30 09:51:51.264797 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:51.264767 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/723c7f11-8c4d-4095-a4c4-fd76af32865c-maas-api-tls\") pod \"maas-api-e2e-inf-a-82ccfd-6df95cf9bc-9nhdz\" (UID: \"723c7f11-8c4d-4095-a4c4-fd76af32865c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82ccfd-6df95cf9bc-9nhdz" Oct 30 09:51:51.275471 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:51.275444 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvdgw\" (UniqueName: \"kubernetes.io/projected/723c7f11-8c4d-4095-a4c4-fd76af32865c-kube-api-access-lvdgw\") pod \"maas-api-e2e-inf-a-82ccfd-6df95cf9bc-9nhdz\" (UID: \"723c7f11-8c4d-4095-a4c4-fd76af32865c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82ccfd-6df95cf9bc-9nhdz" Oct 30 09:51:51.308528 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:51.308495 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82ccfd-6df95cf9bc-9nhdz" Oct 30 09:51:51.615873 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:51:51.615832 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod723c7f11_8c4d_4095_a4c4_fd76af32865c.slice/crio-01e28ca412f2671c79668df406293db4c6113e9d2c88b6e1055313228c58cbc5 WatchSource:0}: Error finding container 01e28ca412f2671c79668df406293db4c6113e9d2c88b6e1055313228c58cbc5: Status 404 returned error can't find the container with id 01e28ca412f2671c79668df406293db4c6113e9d2c88b6e1055313228c58cbc5 Oct 30 09:51:51.617099 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:51.617077 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-82ccfd-6df95cf9bc-9nhdz"] Oct 30 09:51:51.617742 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:51.617725 4562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 09:51:51.655662 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:51.655623 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82ccfd-6df95cf9bc-9nhdz" event={"ID":"723c7f11-8c4d-4095-a4c4-fd76af32865c","Type":"ContainerStarted","Data":"01e28ca412f2671c79668df406293db4c6113e9d2c88b6e1055313228c58cbc5"} Oct 30 09:51:52.666118 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:52.666078 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82ccfd-6df95cf9bc-9nhdz" event={"ID":"723c7f11-8c4d-4095-a4c4-fd76af32865c","Type":"ContainerStarted","Data":"aa6556277b4ae257e0548d8d44cec0154ca11ad9a1a84519889047e19ea12471"} Oct 30 09:51:52.666592 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:52.666149 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82ccfd-6df95cf9bc-9nhdz" Oct 30 09:51:52.706942 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:52.706891 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82ccfd-6df95cf9bc-9nhdz" podStartSLOduration=2.265711164 podStartE2EDuration="2.706877258s" totalImagesPullingTime="441.166094ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:51:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:51:52.70542937 +0000 UTC m=+3472.966990383" watchObservedRunningTime="2026-10-30 09:51:52.706877258 +0000 UTC m=+3472.968438246" Oct 30 09:51:58.598103 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:51:58.598060 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:51:58.681539 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:51:58.681509 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82ccfd-6df95cf9bc-9nhdz" Oct 30 09:52:28.598117 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:52:28.598077 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:52:48.210819 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.210772 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg"] Oct 30 09:52:48.215558 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.215532 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg" Oct 30 09:52:48.224753 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.224724 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq"] Oct 30 09:52:48.239842 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.239713 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq" Oct 30 09:52:48.249824 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.249745 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg"] Oct 30 09:52:48.253840 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.253812 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq"] Oct 30 09:52:48.322588 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.322553 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-82ccfd-openshift-default-dockercfg-rlbdn\"" Oct 30 09:52:48.352554 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.352509 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37d8a054-775f-4a08-baff-90631bbec3be-istio-envoy\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq\" (UID: \"37d8a054-775f-4a08-baff-90631bbec3be\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq" Oct 30 09:52:48.352782 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.352649 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72746dcd-3b5e-46fe-8ec7-eb177b70be60-credential-socket\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg\" (UID: \"72746dcd-3b5e-46fe-8ec7-eb177b70be60\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg" Oct 30 09:52:48.352782 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.352688 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72746dcd-3b5e-46fe-8ec7-eb177b70be60-istiod-ca-cert\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg\" (UID: \"72746dcd-3b5e-46fe-8ec7-eb177b70be60\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg" Oct 30 09:52:48.352782 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.352722 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37d8a054-775f-4a08-baff-90631bbec3be-istio-token\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq\" (UID: \"37d8a054-775f-4a08-baff-90631bbec3be\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq" Oct 30 09:52:48.352782 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.352738 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37d8a054-775f-4a08-baff-90631bbec3be-workload-certs\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq\" (UID: \"37d8a054-775f-4a08-baff-90631bbec3be\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq" Oct 30 09:52:48.353050 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.352823 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72746dcd-3b5e-46fe-8ec7-eb177b70be60-istio-data\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg\" (UID: \"72746dcd-3b5e-46fe-8ec7-eb177b70be60\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg" Oct 30 09:52:48.353050 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.352866 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72746dcd-3b5e-46fe-8ec7-eb177b70be60-istio-podinfo\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg\" (UID: \"72746dcd-3b5e-46fe-8ec7-eb177b70be60\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg" Oct 30 09:52:48.353050 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.352891 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm9cp\" (UniqueName: \"kubernetes.io/projected/37d8a054-775f-4a08-baff-90631bbec3be-kube-api-access-bm9cp\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq\" (UID: \"37d8a054-775f-4a08-baff-90631bbec3be\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq" Oct 30 09:52:48.353050 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.353001 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72746dcd-3b5e-46fe-8ec7-eb177b70be60-workload-certs\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg\" (UID: \"72746dcd-3b5e-46fe-8ec7-eb177b70be60\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg" Oct 30 09:52:48.353050 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.353049 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37d8a054-775f-4a08-baff-90631bbec3be-workload-socket\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq\" (UID: \"37d8a054-775f-4a08-baff-90631bbec3be\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq" Oct 30 09:52:48.353350 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.353158 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37d8a054-775f-4a08-baff-90631bbec3be-credential-socket\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq\" (UID: \"37d8a054-775f-4a08-baff-90631bbec3be\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq" Oct 30 09:52:48.353350 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.353210 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37d8a054-775f-4a08-baff-90631bbec3be-istio-data\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq\" (UID: \"37d8a054-775f-4a08-baff-90631bbec3be\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq" Oct 30 09:52:48.353350 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.353257 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72746dcd-3b5e-46fe-8ec7-eb177b70be60-istio-token\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg\" (UID: \"72746dcd-3b5e-46fe-8ec7-eb177b70be60\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg" Oct 30 09:52:48.353550 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.353363 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37d8a054-775f-4a08-baff-90631bbec3be-istiod-ca-cert\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq\" (UID: \"37d8a054-775f-4a08-baff-90631bbec3be\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq" Oct 30 09:52:48.353550 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.353416 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l725g\" (UniqueName: \"kubernetes.io/projected/72746dcd-3b5e-46fe-8ec7-eb177b70be60-kube-api-access-l725g\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg\" (UID: \"72746dcd-3b5e-46fe-8ec7-eb177b70be60\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg" Oct 30 09:52:48.353550 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.353444 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72746dcd-3b5e-46fe-8ec7-eb177b70be60-workload-socket\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg\" (UID: \"72746dcd-3b5e-46fe-8ec7-eb177b70be60\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg" Oct 30 09:52:48.353550 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.353479 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37d8a054-775f-4a08-baff-90631bbec3be-istio-podinfo\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq\" (UID: \"37d8a054-775f-4a08-baff-90631bbec3be\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq" Oct 30 09:52:48.353550 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.353506 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72746dcd-3b5e-46fe-8ec7-eb177b70be60-istio-envoy\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg\" (UID: \"72746dcd-3b5e-46fe-8ec7-eb177b70be60\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg" Oct 30 09:52:48.455679 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.455633 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37d8a054-775f-4a08-baff-90631bbec3be-istiod-ca-cert\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq\" (UID: \"37d8a054-775f-4a08-baff-90631bbec3be\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq" Oct 30 09:52:48.455893 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.455699 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l725g\" (UniqueName: \"kubernetes.io/projected/72746dcd-3b5e-46fe-8ec7-eb177b70be60-kube-api-access-l725g\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg\" (UID: \"72746dcd-3b5e-46fe-8ec7-eb177b70be60\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg" Oct 30 09:52:48.455893 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.455731 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72746dcd-3b5e-46fe-8ec7-eb177b70be60-workload-socket\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg\" (UID: \"72746dcd-3b5e-46fe-8ec7-eb177b70be60\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg" Oct 30 09:52:48.455893 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.455763 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37d8a054-775f-4a08-baff-90631bbec3be-istio-podinfo\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq\" (UID: \"37d8a054-775f-4a08-baff-90631bbec3be\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq" Oct 30 09:52:48.455893 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.455791 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72746dcd-3b5e-46fe-8ec7-eb177b70be60-istio-envoy\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg\" (UID: \"72746dcd-3b5e-46fe-8ec7-eb177b70be60\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg" Oct 30 09:52:48.456549 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.455924 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37d8a054-775f-4a08-baff-90631bbec3be-istio-envoy\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq\" (UID: \"37d8a054-775f-4a08-baff-90631bbec3be\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq" Oct 30 09:52:48.456549 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.455980 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72746dcd-3b5e-46fe-8ec7-eb177b70be60-credential-socket\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg\" (UID: \"72746dcd-3b5e-46fe-8ec7-eb177b70be60\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg" Oct 30 09:52:48.456549 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.456015 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72746dcd-3b5e-46fe-8ec7-eb177b70be60-istiod-ca-cert\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg\" (UID: \"72746dcd-3b5e-46fe-8ec7-eb177b70be60\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg" Oct 30 09:52:48.456549 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.456064 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37d8a054-775f-4a08-baff-90631bbec3be-istio-token\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq\" (UID: \"37d8a054-775f-4a08-baff-90631bbec3be\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq" Oct 30 09:52:48.456549 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.456087 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37d8a054-775f-4a08-baff-90631bbec3be-workload-certs\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq\" (UID: \"37d8a054-775f-4a08-baff-90631bbec3be\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq" Oct 30 09:52:48.456549 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.456132 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72746dcd-3b5e-46fe-8ec7-eb177b70be60-istio-data\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg\" (UID: \"72746dcd-3b5e-46fe-8ec7-eb177b70be60\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg" Oct 30 09:52:48.456549 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.456198 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72746dcd-3b5e-46fe-8ec7-eb177b70be60-istio-podinfo\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg\" (UID: \"72746dcd-3b5e-46fe-8ec7-eb177b70be60\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg" Oct 30 09:52:48.456549 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.456239 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm9cp\" (UniqueName: \"kubernetes.io/projected/37d8a054-775f-4a08-baff-90631bbec3be-kube-api-access-bm9cp\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq\" (UID: \"37d8a054-775f-4a08-baff-90631bbec3be\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq" Oct 30 09:52:48.456549 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.456268 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72746dcd-3b5e-46fe-8ec7-eb177b70be60-workload-certs\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg\" (UID: \"72746dcd-3b5e-46fe-8ec7-eb177b70be60\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg" Oct 30 09:52:48.456549 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.456304 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37d8a054-775f-4a08-baff-90631bbec3be-workload-socket\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq\" (UID: \"37d8a054-775f-4a08-baff-90631bbec3be\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq" Oct 30 09:52:48.456549 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.456351 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37d8a054-775f-4a08-baff-90631bbec3be-credential-socket\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq\" (UID: \"37d8a054-775f-4a08-baff-90631bbec3be\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq" Oct 30 09:52:48.456549 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.456408 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37d8a054-775f-4a08-baff-90631bbec3be-istio-data\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq\" (UID: \"37d8a054-775f-4a08-baff-90631bbec3be\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq" Oct 30 09:52:48.456549 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.456453 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72746dcd-3b5e-46fe-8ec7-eb177b70be60-istio-token\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg\" (UID: \"72746dcd-3b5e-46fe-8ec7-eb177b70be60\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg" Oct 30 09:52:48.457438 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.457373 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37d8a054-775f-4a08-baff-90631bbec3be-istiod-ca-cert\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq\" (UID: \"37d8a054-775f-4a08-baff-90631bbec3be\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq" Oct 30 09:52:48.457795 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.457772 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37d8a054-775f-4a08-baff-90631bbec3be-workload-certs\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq\" (UID: \"37d8a054-775f-4a08-baff-90631bbec3be\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq" Oct 30 09:52:48.457931 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.457844 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72746dcd-3b5e-46fe-8ec7-eb177b70be60-workload-certs\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg\" (UID: \"72746dcd-3b5e-46fe-8ec7-eb177b70be60\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg" Oct 30 09:52:48.458117 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.458097 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72746dcd-3b5e-46fe-8ec7-eb177b70be60-istio-data\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg\" (UID: \"72746dcd-3b5e-46fe-8ec7-eb177b70be60\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg" Oct 30 09:52:48.458225 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.458202 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72746dcd-3b5e-46fe-8ec7-eb177b70be60-credential-socket\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg\" (UID: \"72746dcd-3b5e-46fe-8ec7-eb177b70be60\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg" Oct 30 09:52:48.458361 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.458218 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72746dcd-3b5e-46fe-8ec7-eb177b70be60-istiod-ca-cert\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg\" (UID: \"72746dcd-3b5e-46fe-8ec7-eb177b70be60\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg" Oct 30 09:52:48.458546 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.458501 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72746dcd-3b5e-46fe-8ec7-eb177b70be60-workload-socket\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg\" (UID: \"72746dcd-3b5e-46fe-8ec7-eb177b70be60\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg" Oct 30 09:52:48.458917 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.458889 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37d8a054-775f-4a08-baff-90631bbec3be-istio-data\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq\" (UID: \"37d8a054-775f-4a08-baff-90631bbec3be\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq" Oct 30 09:52:48.459100 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.459077 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37d8a054-775f-4a08-baff-90631bbec3be-workload-socket\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq\" (UID: \"37d8a054-775f-4a08-baff-90631bbec3be\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq" Oct 30 09:52:48.459285 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.459263 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37d8a054-775f-4a08-baff-90631bbec3be-credential-socket\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq\" (UID: \"37d8a054-775f-4a08-baff-90631bbec3be\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq" Oct 30 09:52:48.460694 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.460671 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37d8a054-775f-4a08-baff-90631bbec3be-istio-podinfo\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq\" (UID: \"37d8a054-775f-4a08-baff-90631bbec3be\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq" Oct 30 09:52:48.461074 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.461025 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72746dcd-3b5e-46fe-8ec7-eb177b70be60-istio-envoy\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg\" (UID: \"72746dcd-3b5e-46fe-8ec7-eb177b70be60\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg" Oct 30 09:52:48.461074 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.461027 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37d8a054-775f-4a08-baff-90631bbec3be-istio-envoy\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq\" (UID: \"37d8a054-775f-4a08-baff-90631bbec3be\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq" Oct 30 09:52:48.461201 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.461129 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72746dcd-3b5e-46fe-8ec7-eb177b70be60-istio-podinfo\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg\" (UID: \"72746dcd-3b5e-46fe-8ec7-eb177b70be60\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg" Oct 30 09:52:48.478449 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.478426 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72746dcd-3b5e-46fe-8ec7-eb177b70be60-istio-token\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg\" (UID: \"72746dcd-3b5e-46fe-8ec7-eb177b70be60\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg" Oct 30 09:52:48.482789 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.482751 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37d8a054-775f-4a08-baff-90631bbec3be-istio-token\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq\" (UID: \"37d8a054-775f-4a08-baff-90631bbec3be\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq" Oct 30 09:52:48.483101 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.483078 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm9cp\" (UniqueName: \"kubernetes.io/projected/37d8a054-775f-4a08-baff-90631bbec3be-kube-api-access-bm9cp\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq\" (UID: \"37d8a054-775f-4a08-baff-90631bbec3be\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq" Oct 30 09:52:48.483416 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.483357 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l725g\" (UniqueName: \"kubernetes.io/projected/72746dcd-3b5e-46fe-8ec7-eb177b70be60-kube-api-access-l725g\") pod \"e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg\" (UID: \"72746dcd-3b5e-46fe-8ec7-eb177b70be60\") " pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg" Oct 30 09:52:48.545919 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.545881 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg" Oct 30 09:52:48.566283 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.566248 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq" Oct 30 09:52:48.867753 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:52:48.867717 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72746dcd_3b5e_46fe_8ec7_eb177b70be60.slice/crio-4fda04e728ba066cc8ad6bc0748cf203df3f51d1f8cff17373a10f1ddf0c32b7 WatchSource:0}: Error finding container 4fda04e728ba066cc8ad6bc0748cf203df3f51d1f8cff17373a10f1ddf0c32b7: Status 404 returned error can't find the container with id 4fda04e728ba066cc8ad6bc0748cf203df3f51d1f8cff17373a10f1ddf0c32b7 Oct 30 09:52:48.869803 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.869783 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg"] Oct 30 09:52:48.870606 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.870579 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:52:48.870658 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.870642 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:52:48.870700 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.870673 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:52:48.902719 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.902688 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq"] Oct 30 09:52:48.914034 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:52:48.914009 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37d8a054_775f_4a08_baff_90631bbec3be.slice/crio-4c6e560bb3b7fa42a82c43957773baf42f74297c4710dddd7b6a3850ed45637e WatchSource:0}: Error finding container 4c6e560bb3b7fa42a82c43957773baf42f74297c4710dddd7b6a3850ed45637e: Status 404 returned error can't find the container with id 4c6e560bb3b7fa42a82c43957773baf42f74297c4710dddd7b6a3850ed45637e Oct 30 09:52:48.917228 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.917192 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:52:48.917328 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.917266 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:52:48.917328 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:48.917296 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:52:49.175803 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:49.175701 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq" event={"ID":"37d8a054-775f-4a08-baff-90631bbec3be","Type":"ContainerStarted","Data":"299d3835c6120870e50dde60384201ac3fe0a834ee5a57d76e0d2dd58e810a0b"} Oct 30 09:52:49.175803 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:49.175777 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq" event={"ID":"37d8a054-775f-4a08-baff-90631bbec3be","Type":"ContainerStarted","Data":"4c6e560bb3b7fa42a82c43957773baf42f74297c4710dddd7b6a3850ed45637e"} Oct 30 09:52:49.177736 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:49.177702 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg" event={"ID":"72746dcd-3b5e-46fe-8ec7-eb177b70be60","Type":"ContainerStarted","Data":"a27353c0140cd23dd26d1ca41ca64d69838bda49128753571f1898e9e671319c"} Oct 30 09:52:49.177872 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:49.177763 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg" event={"ID":"72746dcd-3b5e-46fe-8ec7-eb177b70be60","Type":"ContainerStarted","Data":"4fda04e728ba066cc8ad6bc0748cf203df3f51d1f8cff17373a10f1ddf0c32b7"} Oct 30 09:52:49.224046 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:49.223984 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq" podStartSLOduration=1.223967532 podStartE2EDuration="1.223967532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:52:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:52:49.218021977 +0000 UTC m=+3529.479582964" watchObservedRunningTime="2026-10-30 09:52:49.223967532 +0000 UTC m=+3529.485528518" Oct 30 09:52:49.265796 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:49.265738 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg" podStartSLOduration=1.265722421 podStartE2EDuration="1.265722421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:52:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:52:49.264087909 +0000 UTC m=+3529.525648935" watchObservedRunningTime="2026-10-30 09:52:49.265722421 +0000 UTC m=+3529.527283453" Oct 30 09:52:49.546499 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:49.546453 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg" Oct 30 09:52:49.548147 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:49.548118 4562 patch_prober.go:29] interesting pod/e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body= Oct 30 09:52:49.548307 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:49.548176 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg" podUID="72746dcd-3b5e-46fe-8ec7-eb177b70be60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" Oct 30 09:52:49.566809 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:49.566772 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq" Oct 30 09:52:49.568299 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:49.568273 4562 patch_prober.go:29] interesting pod/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body= Oct 30 09:52:49.568435 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:49.568319 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq" podUID="37d8a054-775f-4a08-baff-90631bbec3be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" Oct 30 09:52:50.617404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:50.592634 4562 patch_prober.go:29] interesting pod/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body= Oct 30 09:52:50.617404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:50.592697 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq" podUID="37d8a054-775f-4a08-baff-90631bbec3be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" Oct 30 09:52:50.617404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:50.593781 4562 patch_prober.go:29] interesting pod/e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body= Oct 30 09:52:50.617404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:50.593823 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg" podUID="72746dcd-3b5e-46fe-8ec7-eb177b70be60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" Oct 30 09:52:51.563407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:51.561449 4562 patch_prober.go:29] interesting pod/e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body= Oct 30 09:52:51.563407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:51.561532 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg" podUID="72746dcd-3b5e-46fe-8ec7-eb177b70be60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" Oct 30 09:52:51.580282 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:51.572607 4562 patch_prober.go:29] interesting pod/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body= Oct 30 09:52:51.580282 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:51.572656 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq" podUID="37d8a054-775f-4a08-baff-90631bbec3be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" Oct 30 09:52:52.550486 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:52.546849 4562 patch_prober.go:29] interesting pod/e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body= Oct 30 09:52:52.550486 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:52.546956 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg" podUID="72746dcd-3b5e-46fe-8ec7-eb177b70be60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" Oct 30 09:52:52.575406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:52.574512 4562 patch_prober.go:29] interesting pod/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body= Oct 30 09:52:52.575406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:52.574593 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq" podUID="37d8a054-775f-4a08-baff-90631bbec3be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" Oct 30 09:52:53.575036 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:53.574989 4562 patch_prober.go:29] interesting pod/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body= Oct 30 09:52:53.575685 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:53.575638 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq" podUID="37d8a054-775f-4a08-baff-90631bbec3be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" Oct 30 09:52:53.576551 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:53.576513 4562 patch_prober.go:29] interesting pod/e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body= Oct 30 09:52:53.576727 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:53.576706 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg" podUID="72746dcd-3b5e-46fe-8ec7-eb177b70be60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" Oct 30 09:52:54.546843 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:54.546801 4562 patch_prober.go:29] interesting pod/e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body= Oct 30 09:52:54.547162 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:54.547131 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg" podUID="72746dcd-3b5e-46fe-8ec7-eb177b70be60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" Oct 30 09:52:54.576796 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:54.576759 4562 patch_prober.go:29] interesting pod/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body= Oct 30 09:52:54.577344 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:54.577312 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq" podUID="37d8a054-775f-4a08-baff-90631bbec3be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" Oct 30 09:52:55.570530 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:55.570487 4562 patch_prober.go:29] interesting pod/e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body= Oct 30 09:52:55.570880 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:55.570842 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg" podUID="72746dcd-3b5e-46fe-8ec7-eb177b70be60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" Oct 30 09:52:55.571819 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:55.571795 4562 patch_prober.go:29] interesting pod/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body= Oct 30 09:52:55.572001 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:55.571979 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq" podUID="37d8a054-775f-4a08-baff-90631bbec3be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" Oct 30 09:52:56.551961 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:56.551928 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg" Oct 30 09:52:56.552429 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:56.552412 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg" Oct 30 09:52:56.553099 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:56.553079 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg" Oct 30 09:52:56.572172 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:56.572148 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq" Oct 30 09:52:56.572447 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:56.572427 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq" Oct 30 09:52:56.573177 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:52:56.573160 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq" Oct 30 09:52:58.599577 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:52:58.599543 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:53:00.147816 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:00.147782 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-56dc545b7c-d92fs"] Oct 30 09:53:05.153521 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:05.153462 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-56dc545b7c-d92fs" podUID="e6b77f09-6ef1-437c-8fc6-2720b10122e6" containerName="payload-processing" containerID="cri-o://5aeabf936c06ba2168c9238a726c3f3bf0d5b9cbbf8b0ec7b5908df163187955" gracePeriod=25 Oct 30 09:53:05.459341 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:05.459299 4562 generic.go:356] "Generic (PLEG): container finished" podID="e6b77f09-6ef1-437c-8fc6-2720b10122e6" containerID="5aeabf936c06ba2168c9238a726c3f3bf0d5b9cbbf8b0ec7b5908df163187955" exitCode=0 Oct 30 09:53:05.459707 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:05.459623 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56dc545b7c-d92fs" event={"ID":"e6b77f09-6ef1-437c-8fc6-2720b10122e6","Type":"ContainerDied","Data":"5aeabf936c06ba2168c9238a726c3f3bf0d5b9cbbf8b0ec7b5908df163187955"} Oct 30 09:53:05.554234 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:05.554201 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56dc545b7c-d92fs" Oct 30 09:53:05.576990 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:05.576962 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6b77f09-6ef1-437c-8fc6-2720b10122e6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e6b77f09-6ef1-437c-8fc6-2720b10122e6-provider-credentials\") pod \"e6b77f09-6ef1-437c-8fc6-2720b10122e6\" (UID: \"e6b77f09-6ef1-437c-8fc6-2720b10122e6\") " Oct 30 09:53:05.577136 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:05.577019 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6b77f09-6ef1-437c-8fc6-2720b10122e6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6b77f09-6ef1-437c-8fc6-2720b10122e6-plugins-config-volume\") pod \"e6b77f09-6ef1-437c-8fc6-2720b10122e6\" (UID: \"e6b77f09-6ef1-437c-8fc6-2720b10122e6\") " Oct 30 09:53:05.577136 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:05.577037 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6b77f09-6ef1-437c-8fc6-2720b10122e6-kube-api-access-8rnlb\" (UniqueName: \"kubernetes.io/projected/e6b77f09-6ef1-437c-8fc6-2720b10122e6-kube-api-access-8rnlb\") pod \"e6b77f09-6ef1-437c-8fc6-2720b10122e6\" (UID: \"e6b77f09-6ef1-437c-8fc6-2720b10122e6\") " Oct 30 09:53:05.577136 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:05.577121 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6b77f09-6ef1-437c-8fc6-2720b10122e6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e6b77f09-6ef1-437c-8fc6-2720b10122e6-routing-overlay-volume\") pod \"e6b77f09-6ef1-437c-8fc6-2720b10122e6\" (UID: \"e6b77f09-6ef1-437c-8fc6-2720b10122e6\") " Oct 30 09:53:05.577506 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:05.577480 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6b77f09-6ef1-437c-8fc6-2720b10122e6-plugins-config-volume" pod "e6b77f09-6ef1-437c-8fc6-2720b10122e6" (UID: "e6b77f09-6ef1-437c-8fc6-2720b10122e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:53:05.577610 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:05.577527 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6b77f09-6ef1-437c-8fc6-2720b10122e6-routing-overlay-volume" pod "e6b77f09-6ef1-437c-8fc6-2720b10122e6" (UID: "e6b77f09-6ef1-437c-8fc6-2720b10122e6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:53:05.577610 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:05.577578 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6b77f09-6ef1-437c-8fc6-2720b10122e6-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:53:05.579329 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:05.579305 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6b77f09-6ef1-437c-8fc6-2720b10122e6-kube-api-access-8rnlb" pod "e6b77f09-6ef1-437c-8fc6-2720b10122e6" (UID: "e6b77f09-6ef1-437c-8fc6-2720b10122e6"). InnerVolumeSpecName "kube-api-access-8rnlb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:53:05.579549 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:05.579328 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6b77f09-6ef1-437c-8fc6-2720b10122e6-provider-credentials" pod "e6b77f09-6ef1-437c-8fc6-2720b10122e6" (UID: "e6b77f09-6ef1-437c-8fc6-2720b10122e6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:53:05.678609 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:05.678525 4562 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e6b77f09-6ef1-437c-8fc6-2720b10122e6-provider-credentials\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:53:05.678609 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:05.678555 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rnlb\" (UniqueName: \"kubernetes.io/projected/e6b77f09-6ef1-437c-8fc6-2720b10122e6-kube-api-access-8rnlb\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:53:05.678609 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:05.678569 4562 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e6b77f09-6ef1-437c-8fc6-2720b10122e6-routing-overlay-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:53:06.007363 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:06.007325 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-035d7d/payload-processing-external-model-e2e-extmod-b-035d7d-669bxqm7s"] Oct 30 09:53:06.008339 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:06.008312 4562 cpu_manager.go:414] "containerMap: removing container" podUID="e6b77f09-6ef1-437c-8fc6-2720b10122e6" containerName="payload-processing" Oct 30 09:53:06.008339 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:06.008338 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6b77f09-6ef1-437c-8fc6-2720b10122e6" containerName="payload-processing" Oct 30 09:53:06.008576 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:06.008561 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6b77f09-6ef1-437c-8fc6-2720b10122e6" containerName="payload-processing" Oct 30 09:53:06.013622 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:06.013598 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-035d7d/payload-processing-external-model-e2e-extmod-b-035d7d-669bxqm7s" Oct 30 09:53:06.033189 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:06.033155 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-035d7d/payload-processing-external-model-e2e-extmod-b-035d7d-669bxqm7s"] Oct 30 09:53:06.034707 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:06.034682 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-035d7d\"/\"kube-root-ca.crt\"" Oct 30 09:53:06.037947 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:06.037607 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-035d7d\"/\"openshift-service-ca.crt\"" Oct 30 09:53:06.037947 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:06.037672 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-035d7d\"/\"agc-cred-bddb26f4a5b9\"" Oct 30 09:53:06.037947 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:06.037756 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-035d7d\"/\"routing-overlay\"" Oct 30 09:53:06.037947 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:06.037822 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-035d7d\"/\"payload-processing-external-model-plugins-e2e-extmod-b-035d7d\"" Oct 30 09:53:06.037947 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:06.037919 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-035d7d\"/\"payload-processing-external-model-e2e--0e40e469-dockercfg-qxtqs\"" Oct 30 09:53:06.096224 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:06.096189 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9988e29-df3c-4449-883a-41f7f750da62-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-035d7d-669bxqm7s\" (UID: \"c9988e29-df3c-4449-883a-41f7f750da62\") " pod="ai-tenant-e2e-extmod-b-035d7d/payload-processing-external-model-e2e-extmod-b-035d7d-669bxqm7s" Oct 30 09:53:06.096421 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:06.096338 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krmgl\" (UniqueName: \"kubernetes.io/projected/c9988e29-df3c-4449-883a-41f7f750da62-kube-api-access-krmgl\") pod \"payload-processing-external-model-e2e-extmod-b-035d7d-669bxqm7s\" (UID: \"c9988e29-df3c-4449-883a-41f7f750da62\") " pod="ai-tenant-e2e-extmod-b-035d7d/payload-processing-external-model-e2e-extmod-b-035d7d-669bxqm7s" Oct 30 09:53:06.096517 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:06.096500 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9988e29-df3c-4449-883a-41f7f750da62-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-035d7d-669bxqm7s\" (UID: \"c9988e29-df3c-4449-883a-41f7f750da62\") " pod="ai-tenant-e2e-extmod-b-035d7d/payload-processing-external-model-e2e-extmod-b-035d7d-669bxqm7s" Oct 30 09:53:06.096580 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:06.096550 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9988e29-df3c-4449-883a-41f7f750da62-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-035d7d-669bxqm7s\" (UID: \"c9988e29-df3c-4449-883a-41f7f750da62\") " pod="ai-tenant-e2e-extmod-b-035d7d/payload-processing-external-model-e2e-extmod-b-035d7d-669bxqm7s" Oct 30 09:53:06.212502 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:06.202174 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9988e29-df3c-4449-883a-41f7f750da62-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-035d7d-669bxqm7s\" (UID: \"c9988e29-df3c-4449-883a-41f7f750da62\") " pod="ai-tenant-e2e-extmod-b-035d7d/payload-processing-external-model-e2e-extmod-b-035d7d-669bxqm7s" Oct 30 09:53:06.212502 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:06.202267 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9988e29-df3c-4449-883a-41f7f750da62-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-035d7d-669bxqm7s\" (UID: \"c9988e29-df3c-4449-883a-41f7f750da62\") " pod="ai-tenant-e2e-extmod-b-035d7d/payload-processing-external-model-e2e-extmod-b-035d7d-669bxqm7s" Oct 30 09:53:06.212502 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:06.202339 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9988e29-df3c-4449-883a-41f7f750da62-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-035d7d-669bxqm7s\" (UID: \"c9988e29-df3c-4449-883a-41f7f750da62\") " pod="ai-tenant-e2e-extmod-b-035d7d/payload-processing-external-model-e2e-extmod-b-035d7d-669bxqm7s" Oct 30 09:53:06.212502 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:06.202481 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krmgl\" (UniqueName: \"kubernetes.io/projected/c9988e29-df3c-4449-883a-41f7f750da62-kube-api-access-krmgl\") pod \"payload-processing-external-model-e2e-extmod-b-035d7d-669bxqm7s\" (UID: \"c9988e29-df3c-4449-883a-41f7f750da62\") " pod="ai-tenant-e2e-extmod-b-035d7d/payload-processing-external-model-e2e-extmod-b-035d7d-669bxqm7s" Oct 30 09:53:06.214590 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:06.214521 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9988e29-df3c-4449-883a-41f7f750da62-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-035d7d-669bxqm7s\" (UID: \"c9988e29-df3c-4449-883a-41f7f750da62\") " pod="ai-tenant-e2e-extmod-b-035d7d/payload-processing-external-model-e2e-extmod-b-035d7d-669bxqm7s" Oct 30 09:53:06.215414 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:06.215366 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9988e29-df3c-4449-883a-41f7f750da62-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-035d7d-669bxqm7s\" (UID: \"c9988e29-df3c-4449-883a-41f7f750da62\") " pod="ai-tenant-e2e-extmod-b-035d7d/payload-processing-external-model-e2e-extmod-b-035d7d-669bxqm7s" Oct 30 09:53:06.218476 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:06.218449 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9988e29-df3c-4449-883a-41f7f750da62-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-035d7d-669bxqm7s\" (UID: \"c9988e29-df3c-4449-883a-41f7f750da62\") " pod="ai-tenant-e2e-extmod-b-035d7d/payload-processing-external-model-e2e-extmod-b-035d7d-669bxqm7s" Oct 30 09:53:06.237184 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:06.237146 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krmgl\" (UniqueName: \"kubernetes.io/projected/c9988e29-df3c-4449-883a-41f7f750da62-kube-api-access-krmgl\") pod \"payload-processing-external-model-e2e-extmod-b-035d7d-669bxqm7s\" (UID: \"c9988e29-df3c-4449-883a-41f7f750da62\") " pod="ai-tenant-e2e-extmod-b-035d7d/payload-processing-external-model-e2e-extmod-b-035d7d-669bxqm7s" Oct 30 09:53:06.340007 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:06.339974 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-035d7d/payload-processing-external-model-e2e-extmod-b-035d7d-669bxqm7s" Oct 30 09:53:06.471334 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:06.470278 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56dc545b7c-d92fs" Oct 30 09:53:06.471334 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:06.470348 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56dc545b7c-d92fs" event={"ID":"e6b77f09-6ef1-437c-8fc6-2720b10122e6","Type":"ContainerDied","Data":"6c66bdf058cbbb3a779e8c0815a99a8a335cd638fc1febd953ab0a4f58fe0a2d"} Oct 30 09:53:06.471334 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:06.470445 4562 scope.go:122] "RemoveContainer" containerID="5aeabf936c06ba2168c9238a726c3f3bf0d5b9cbbf8b0ec7b5908df163187955" Oct 30 09:53:06.531988 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:06.531958 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-56dc545b7c-d92fs"] Oct 30 09:53:06.536223 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:06.536198 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-56dc545b7c-d92fs"] Oct 30 09:53:06.686841 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:53:06.686808 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9988e29_df3c_4449_883a_41f7f750da62.slice/crio-838ab236ee8a52f0b3aa9fc89a220ed32f533f1e3f8ab1d1f8e7a4de73b2be57 WatchSource:0}: Error finding container 838ab236ee8a52f0b3aa9fc89a220ed32f533f1e3f8ab1d1f8e7a4de73b2be57: Status 404 returned error can't find the container with id 838ab236ee8a52f0b3aa9fc89a220ed32f533f1e3f8ab1d1f8e7a4de73b2be57 Oct 30 09:53:06.688731 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:06.688698 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-035d7d/payload-processing-external-model-e2e-extmod-b-035d7d-669bxqm7s"] Oct 30 09:53:07.480854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:07.480822 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-035d7d/payload-processing-external-model-e2e-extmod-b-035d7d-669bxqm7s" event={"ID":"c9988e29-df3c-4449-883a-41f7f750da62","Type":"ContainerStarted","Data":"4ed410297ba50347fe6f2eba467fffbd7b3f7ec9366d68c6143da7d64fd413ac"} Oct 30 09:53:07.481256 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:07.480878 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-035d7d/payload-processing-external-model-e2e-extmod-b-035d7d-669bxqm7s" event={"ID":"c9988e29-df3c-4449-883a-41f7f750da62","Type":"ContainerStarted","Data":"838ab236ee8a52f0b3aa9fc89a220ed32f533f1e3f8ab1d1f8e7a4de73b2be57"} Oct 30 09:53:07.481256 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:07.480973 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-035d7d/payload-processing-external-model-e2e-extmod-b-035d7d-669bxqm7s" Oct 30 09:53:07.523675 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:07.523629 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-035d7d/payload-processing-external-model-e2e-extmod-b-035d7d-669bxqm7s" podStartSLOduration=2.523615675 podStartE2EDuration="2.523615675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:53:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:53:07.520250097 +0000 UTC m=+3547.781811085" watchObservedRunningTime="2026-10-30 09:53:07.523615675 +0000 UTC m=+3547.785176664" Oct 30 09:53:07.877244 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:07.877149 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6b77f09-6ef1-437c-8fc6-2720b10122e6" path="/var/lib/kubelet/pods/e6b77f09-6ef1-437c-8fc6-2720b10122e6/volumes" Oct 30 09:53:10.598739 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:10.598697 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-035d7d/payload-processing-external-model-e2e-extmod-b-035d7d-669bxqm7s"] Oct 30 09:53:15.599135 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:15.599050 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-035d7d/payload-processing-external-model-e2e-extmod-b-035d7d-669bxqm7s" podUID="c9988e29-df3c-4449-883a-41f7f750da62" containerName="payload-processing" containerID="cri-o://4ed410297ba50347fe6f2eba467fffbd7b3f7ec9366d68c6143da7d64fd413ac" gracePeriod=25 Oct 30 09:53:15.922943 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:15.922905 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-035d7d/payload-processing-external-model-e2e-extmod-b-035d7d-669bxqm7s" Oct 30 09:53:16.030532 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:16.030494 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9988e29-df3c-4449-883a-41f7f750da62-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9988e29-df3c-4449-883a-41f7f750da62-provider-credentials\") pod \"c9988e29-df3c-4449-883a-41f7f750da62\" (UID: \"c9988e29-df3c-4449-883a-41f7f750da62\") " Oct 30 09:53:16.030784 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:16.030737 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9988e29-df3c-4449-883a-41f7f750da62-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9988e29-df3c-4449-883a-41f7f750da62-routing-overlay-volume\") pod \"c9988e29-df3c-4449-883a-41f7f750da62\" (UID: \"c9988e29-df3c-4449-883a-41f7f750da62\") " Oct 30 09:53:16.030907 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:16.030799 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9988e29-df3c-4449-883a-41f7f750da62-kube-api-access-krmgl\" (UniqueName: \"kubernetes.io/projected/c9988e29-df3c-4449-883a-41f7f750da62-kube-api-access-krmgl\") pod \"c9988e29-df3c-4449-883a-41f7f750da62\" (UID: \"c9988e29-df3c-4449-883a-41f7f750da62\") " Oct 30 09:53:16.030989 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:16.030974 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9988e29-df3c-4449-883a-41f7f750da62-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9988e29-df3c-4449-883a-41f7f750da62-plugins-config-volume\") pod \"c9988e29-df3c-4449-883a-41f7f750da62\" (UID: \"c9988e29-df3c-4449-883a-41f7f750da62\") " Oct 30 09:53:16.060498 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:16.059567 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9988e29-df3c-4449-883a-41f7f750da62-plugins-config-volume" pod "c9988e29-df3c-4449-883a-41f7f750da62" (UID: "c9988e29-df3c-4449-883a-41f7f750da62"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:53:16.075528 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:16.061935 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9988e29-df3c-4449-883a-41f7f750da62-routing-overlay-volume" pod "c9988e29-df3c-4449-883a-41f7f750da62" (UID: "c9988e29-df3c-4449-883a-41f7f750da62"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:53:16.076734 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:16.076024 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9988e29-df3c-4449-883a-41f7f750da62-kube-api-access-krmgl" pod "c9988e29-df3c-4449-883a-41f7f750da62" (UID: "c9988e29-df3c-4449-883a-41f7f750da62"). InnerVolumeSpecName "kube-api-access-krmgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:53:16.129624 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:16.129566 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9988e29-df3c-4449-883a-41f7f750da62-provider-credentials" pod "c9988e29-df3c-4449-883a-41f7f750da62" (UID: "c9988e29-df3c-4449-883a-41f7f750da62"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:53:16.158540 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:16.138196 4562 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9988e29-df3c-4449-883a-41f7f750da62-provider-credentials\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:53:16.158540 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:16.138236 4562 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9988e29-df3c-4449-883a-41f7f750da62-routing-overlay-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:53:16.158540 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:16.138250 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krmgl\" (UniqueName: \"kubernetes.io/projected/c9988e29-df3c-4449-883a-41f7f750da62-kube-api-access-krmgl\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:53:16.158540 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:16.138262 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9988e29-df3c-4449-883a-41f7f750da62-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:53:16.557558 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:16.557515 4562 generic.go:356] "Generic (PLEG): container finished" podID="c9988e29-df3c-4449-883a-41f7f750da62" containerID="4ed410297ba50347fe6f2eba467fffbd7b3f7ec9366d68c6143da7d64fd413ac" exitCode=0 Oct 30 09:53:16.557775 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:16.557563 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-035d7d/payload-processing-external-model-e2e-extmod-b-035d7d-669bxqm7s" Oct 30 09:53:16.557775 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:16.557602 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-035d7d/payload-processing-external-model-e2e-extmod-b-035d7d-669bxqm7s" event={"ID":"c9988e29-df3c-4449-883a-41f7f750da62","Type":"ContainerDied","Data":"4ed410297ba50347fe6f2eba467fffbd7b3f7ec9366d68c6143da7d64fd413ac"} Oct 30 09:53:16.557775 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:16.557663 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-035d7d/payload-processing-external-model-e2e-extmod-b-035d7d-669bxqm7s" event={"ID":"c9988e29-df3c-4449-883a-41f7f750da62","Type":"ContainerDied","Data":"838ab236ee8a52f0b3aa9fc89a220ed32f533f1e3f8ab1d1f8e7a4de73b2be57"} Oct 30 09:53:16.557775 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:16.557687 4562 scope.go:122] "RemoveContainer" containerID="4ed410297ba50347fe6f2eba467fffbd7b3f7ec9366d68c6143da7d64fd413ac" Oct 30 09:53:16.569102 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:16.569081 4562 scope.go:122] "RemoveContainer" containerID="4ed410297ba50347fe6f2eba467fffbd7b3f7ec9366d68c6143da7d64fd413ac" Oct 30 09:53:16.569439 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:53:16.569415 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ed410297ba50347fe6f2eba467fffbd7b3f7ec9366d68c6143da7d64fd413ac\": container with ID starting with 4ed410297ba50347fe6f2eba467fffbd7b3f7ec9366d68c6143da7d64fd413ac not found: ID does not exist" containerID="4ed410297ba50347fe6f2eba467fffbd7b3f7ec9366d68c6143da7d64fd413ac" Oct 30 09:53:16.569534 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:16.569448 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ed410297ba50347fe6f2eba467fffbd7b3f7ec9366d68c6143da7d64fd413ac"} err="failed to get container status \"4ed410297ba50347fe6f2eba467fffbd7b3f7ec9366d68c6143da7d64fd413ac\": rpc error: code = NotFound desc = could not find container \"4ed410297ba50347fe6f2eba467fffbd7b3f7ec9366d68c6143da7d64fd413ac\": container with ID starting with 4ed410297ba50347fe6f2eba467fffbd7b3f7ec9366d68c6143da7d64fd413ac not found: ID does not exist" Oct 30 09:53:16.629145 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:16.629107 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-035d7d/payload-processing-external-model-e2e-extmod-b-035d7d-669bxqm7s"] Oct 30 09:53:16.639089 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:16.639059 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-035d7d/payload-processing-external-model-e2e-extmod-b-035d7d-669bxqm7s"] Oct 30 09:53:17.909753 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:17.909706 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9988e29-df3c-4449-883a-41f7f750da62" path="/var/lib/kubelet/pods/c9988e29-df3c-4449-883a-41f7f750da62/volumes" Oct 30 09:53:20.021738 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:20.021706 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz"] Oct 30 09:53:20.022648 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:20.022630 4562 cpu_manager.go:414] "containerMap: removing container" podUID="c9988e29-df3c-4449-883a-41f7f750da62" containerName="payload-processing" Oct 30 09:53:20.022648 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:20.022648 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9988e29-df3c-4449-883a-41f7f750da62" containerName="payload-processing" Oct 30 09:53:20.022806 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:20.022795 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9988e29-df3c-4449-883a-41f7f750da62" containerName="payload-processing" Oct 30 09:53:20.026888 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:20.026868 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz" Oct 30 09:53:20.057122 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:20.057094 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-bdd208-openshift-default-dockercfg-hzsdt\"" Oct 30 09:53:20.064820 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:20.064792 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz"] Oct 30 09:53:20.201256 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:20.201213 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjmc8\" (UniqueName: \"kubernetes.io/projected/04c56890-74da-4365-b273-13751658dd1a-kube-api-access-vjmc8\") pod \"e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz\" (UID: \"04c56890-74da-4365-b273-13751658dd1a\") " pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz" Oct 30 09:53:20.201459 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:20.201260 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04c56890-74da-4365-b273-13751658dd1a-istio-envoy\") pod \"e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz\" (UID: \"04c56890-74da-4365-b273-13751658dd1a\") " pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz" Oct 30 09:53:20.201459 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:20.201365 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04c56890-74da-4365-b273-13751658dd1a-istio-token\") pod \"e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz\" (UID: \"04c56890-74da-4365-b273-13751658dd1a\") " pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz" Oct 30 09:53:20.201571 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:20.201485 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04c56890-74da-4365-b273-13751658dd1a-istio-podinfo\") pod \"e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz\" (UID: \"04c56890-74da-4365-b273-13751658dd1a\") " pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz" Oct 30 09:53:20.201571 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:20.201523 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04c56890-74da-4365-b273-13751658dd1a-workload-certs\") pod \"e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz\" (UID: \"04c56890-74da-4365-b273-13751658dd1a\") " pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz" Oct 30 09:53:20.201692 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:20.201628 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04c56890-74da-4365-b273-13751658dd1a-workload-socket\") pod \"e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz\" (UID: \"04c56890-74da-4365-b273-13751658dd1a\") " pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz" Oct 30 09:53:20.201692 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:20.201679 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04c56890-74da-4365-b273-13751658dd1a-istio-data\") pod \"e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz\" (UID: \"04c56890-74da-4365-b273-13751658dd1a\") " pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz" Oct 30 09:53:20.201780 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:20.201728 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04c56890-74da-4365-b273-13751658dd1a-credential-socket\") pod \"e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz\" (UID: \"04c56890-74da-4365-b273-13751658dd1a\") " pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz" Oct 30 09:53:20.201780 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:20.201747 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04c56890-74da-4365-b273-13751658dd1a-istiod-ca-cert\") pod \"e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz\" (UID: \"04c56890-74da-4365-b273-13751658dd1a\") " pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz" Oct 30 09:53:20.302937 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:20.302856 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04c56890-74da-4365-b273-13751658dd1a-istio-podinfo\") pod \"e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz\" (UID: \"04c56890-74da-4365-b273-13751658dd1a\") " pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz" Oct 30 09:53:20.302937 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:20.302917 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04c56890-74da-4365-b273-13751658dd1a-workload-certs\") pod \"e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz\" (UID: \"04c56890-74da-4365-b273-13751658dd1a\") " pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz" Oct 30 09:53:20.303179 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:20.302971 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04c56890-74da-4365-b273-13751658dd1a-workload-socket\") pod \"e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz\" (UID: \"04c56890-74da-4365-b273-13751658dd1a\") " pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz" Oct 30 09:53:20.303179 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:20.303013 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04c56890-74da-4365-b273-13751658dd1a-istio-data\") pod \"e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz\" (UID: \"04c56890-74da-4365-b273-13751658dd1a\") " pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz" Oct 30 09:53:20.303179 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:20.303052 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04c56890-74da-4365-b273-13751658dd1a-credential-socket\") pod \"e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz\" (UID: \"04c56890-74da-4365-b273-13751658dd1a\") " pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz" Oct 30 09:53:20.303179 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:20.303071 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04c56890-74da-4365-b273-13751658dd1a-istiod-ca-cert\") pod \"e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz\" (UID: \"04c56890-74da-4365-b273-13751658dd1a\") " pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz" Oct 30 09:53:20.303430 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:20.303301 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjmc8\" (UniqueName: \"kubernetes.io/projected/04c56890-74da-4365-b273-13751658dd1a-kube-api-access-vjmc8\") pod \"e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz\" (UID: \"04c56890-74da-4365-b273-13751658dd1a\") " pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz" Oct 30 09:53:20.303430 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:20.303334 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04c56890-74da-4365-b273-13751658dd1a-istio-envoy\") pod \"e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz\" (UID: \"04c56890-74da-4365-b273-13751658dd1a\") " pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz" Oct 30 09:53:20.303430 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:20.303398 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04c56890-74da-4365-b273-13751658dd1a-istio-token\") pod \"e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz\" (UID: \"04c56890-74da-4365-b273-13751658dd1a\") " pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz" Oct 30 09:53:20.303648 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:20.303421 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04c56890-74da-4365-b273-13751658dd1a-workload-certs\") pod \"e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz\" (UID: \"04c56890-74da-4365-b273-13751658dd1a\") " pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz" Oct 30 09:53:20.303648 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:20.303503 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04c56890-74da-4365-b273-13751658dd1a-credential-socket\") pod \"e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz\" (UID: \"04c56890-74da-4365-b273-13751658dd1a\") " pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz" Oct 30 09:53:20.303788 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:20.303652 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04c56890-74da-4365-b273-13751658dd1a-workload-socket\") pod \"e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz\" (UID: \"04c56890-74da-4365-b273-13751658dd1a\") " pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz" Oct 30 09:53:20.304105 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:20.304081 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04c56890-74da-4365-b273-13751658dd1a-istiod-ca-cert\") pod \"e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz\" (UID: \"04c56890-74da-4365-b273-13751658dd1a\") " pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz" Oct 30 09:53:20.304275 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:20.304257 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04c56890-74da-4365-b273-13751658dd1a-istio-data\") pod \"e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz\" (UID: \"04c56890-74da-4365-b273-13751658dd1a\") " pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz" Oct 30 09:53:20.305750 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:20.305725 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04c56890-74da-4365-b273-13751658dd1a-istio-podinfo\") pod \"e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz\" (UID: \"04c56890-74da-4365-b273-13751658dd1a\") " pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz" Oct 30 09:53:20.306257 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:20.306231 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04c56890-74da-4365-b273-13751658dd1a-istio-envoy\") pod \"e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz\" (UID: \"04c56890-74da-4365-b273-13751658dd1a\") " pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz" Oct 30 09:53:20.324165 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:20.324138 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04c56890-74da-4365-b273-13751658dd1a-istio-token\") pod \"e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz\" (UID: \"04c56890-74da-4365-b273-13751658dd1a\") " pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz" Oct 30 09:53:20.324369 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:20.324343 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjmc8\" (UniqueName: \"kubernetes.io/projected/04c56890-74da-4365-b273-13751658dd1a-kube-api-access-vjmc8\") pod \"e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz\" (UID: \"04c56890-74da-4365-b273-13751658dd1a\") " pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz" Oct 30 09:53:20.342948 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:20.342918 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz" Oct 30 09:53:20.660441 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:53:20.660410 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04c56890_74da_4365_b273_13751658dd1a.slice/crio-8c8b98717325156e89011259198090b25ca115f3092b3ca4d63d23659ea0ec43 WatchSource:0}: Error finding container 8c8b98717325156e89011259198090b25ca115f3092b3ca4d63d23659ea0ec43: Status 404 returned error can't find the container with id 8c8b98717325156e89011259198090b25ca115f3092b3ca4d63d23659ea0ec43 Oct 30 09:53:20.662041 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:20.661987 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz"] Oct 30 09:53:20.663356 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:20.663326 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:53:20.663444 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:20.663409 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:53:20.663486 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:20.663454 4562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 09:53:21.601747 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:21.601707 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz" event={"ID":"04c56890-74da-4365-b273-13751658dd1a","Type":"ContainerStarted","Data":"b68d33df98d10d578a9b113d5168530e948f5655067a56e110ba6421856728a4"} Oct 30 09:53:21.602160 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:21.601766 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz" event={"ID":"04c56890-74da-4365-b273-13751658dd1a","Type":"ContainerStarted","Data":"8c8b98717325156e89011259198090b25ca115f3092b3ca4d63d23659ea0ec43"} Oct 30 09:53:22.365428 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:22.363487 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz" Oct 30 09:53:22.367288 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:22.367251 4562 patch_prober.go:29] interesting pod/e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body= Oct 30 09:53:22.367502 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:22.367318 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz" podUID="04c56890-74da-4365-b273-13751658dd1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" Oct 30 09:53:23.356408 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:23.354812 4562 patch_prober.go:29] interesting pod/e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body= Oct 30 09:53:23.356408 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:23.354875 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz" podUID="04c56890-74da-4365-b273-13751658dd1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" Oct 30 09:53:24.360408 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:24.355769 4562 patch_prober.go:29] interesting pod/e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body= Oct 30 09:53:24.360408 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:24.355834 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz" podUID="04c56890-74da-4365-b273-13751658dd1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" Oct 30 09:53:25.354406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:25.345752 4562 patch_prober.go:29] interesting pod/e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body= Oct 30 09:53:25.354406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:25.345817 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz" podUID="04c56890-74da-4365-b273-13751658dd1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" Oct 30 09:53:26.355407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:26.345802 4562 patch_prober.go:29] interesting pod/e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body= Oct 30 09:53:26.355407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:26.345867 4562 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz" podUID="04c56890-74da-4365-b273-13751658dd1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" Oct 30 09:53:27.758509 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:27.758468 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz" Oct 30 09:53:27.759263 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:27.759236 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz" Oct 30 09:53:27.762057 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:27.762031 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz" Oct 30 09:53:27.829349 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:27.829300 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz" podStartSLOduration=8.829287766 podStartE2EDuration="8.829287766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:53:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:53:21.672775012 +0000 UTC m=+3561.934336000" watchObservedRunningTime="2026-10-30 09:53:27.829287766 +0000 UTC m=+3568.090848731" Oct 30 09:53:28.601413 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:53:28.601348 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:53:28.608267 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:28.608230 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-0ab35b/e2e-usage-sim-e02533-5d4ff8cf6d-455dp"] Oct 30 09:53:28.608474 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:28.608447 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-extmod-shr-user-0ab35b/e2e-usage-sim-e02533-5d4ff8cf6d-455dp" podUID="e32372d7-a2c3-44d4-9872-0ac6c3091bb9" containerName="sim" containerID="cri-o://9de20d9f01f9beef813e3a740e1a1ee8d4fd313f28f0d838c83a2aa488934283" gracePeriod=30 Oct 30 09:53:29.387127 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:29.387098 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-0ab35b/e2e-usage-sim-e02533-5d4ff8cf6d-455dp" Oct 30 09:53:29.544840 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:29.544754 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e32372d7-a2c3-44d4-9872-0ac6c3091bb9-tls\" (UniqueName: \"kubernetes.io/secret/e32372d7-a2c3-44d4-9872-0ac6c3091bb9-tls\") pod \"e32372d7-a2c3-44d4-9872-0ac6c3091bb9\" (UID: \"e32372d7-a2c3-44d4-9872-0ac6c3091bb9\") " Oct 30 09:53:29.545085 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:29.545061 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e32372d7-a2c3-44d4-9872-0ac6c3091bb9-kube-api-access-vp9rg\" (UniqueName: \"kubernetes.io/projected/e32372d7-a2c3-44d4-9872-0ac6c3091bb9-kube-api-access-vp9rg\") pod \"e32372d7-a2c3-44d4-9872-0ac6c3091bb9\" (UID: \"e32372d7-a2c3-44d4-9872-0ac6c3091bb9\") " Oct 30 09:53:29.547185 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:29.547154 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e32372d7-a2c3-44d4-9872-0ac6c3091bb9-tls" pod "e32372d7-a2c3-44d4-9872-0ac6c3091bb9" (UID: "e32372d7-a2c3-44d4-9872-0ac6c3091bb9"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:53:29.547334 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:29.547308 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e32372d7-a2c3-44d4-9872-0ac6c3091bb9-kube-api-access-vp9rg" pod "e32372d7-a2c3-44d4-9872-0ac6c3091bb9" (UID: "e32372d7-a2c3-44d4-9872-0ac6c3091bb9"). InnerVolumeSpecName "kube-api-access-vp9rg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:53:29.647257 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:29.647225 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vp9rg\" (UniqueName: \"kubernetes.io/projected/e32372d7-a2c3-44d4-9872-0ac6c3091bb9-kube-api-access-vp9rg\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:53:29.647257 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:29.647254 4562 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/e32372d7-a2c3-44d4-9872-0ac6c3091bb9-tls\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:53:29.687238 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:29.687206 4562 generic.go:356] "Generic (PLEG): container finished" podID="e32372d7-a2c3-44d4-9872-0ac6c3091bb9" containerID="9de20d9f01f9beef813e3a740e1a1ee8d4fd313f28f0d838c83a2aa488934283" exitCode=0 Oct 30 09:53:29.687420 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:29.687256 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-0ab35b/e2e-usage-sim-e02533-5d4ff8cf6d-455dp" Oct 30 09:53:29.687420 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:29.687282 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-0ab35b/e2e-usage-sim-e02533-5d4ff8cf6d-455dp" event={"ID":"e32372d7-a2c3-44d4-9872-0ac6c3091bb9","Type":"ContainerDied","Data":"9de20d9f01f9beef813e3a740e1a1ee8d4fd313f28f0d838c83a2aa488934283"} Oct 30 09:53:29.687420 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:29.687323 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-0ab35b/e2e-usage-sim-e02533-5d4ff8cf6d-455dp" event={"ID":"e32372d7-a2c3-44d4-9872-0ac6c3091bb9","Type":"ContainerDied","Data":"9fb0297826715d78ff90ccb4dc6fbcaa80352bbfdd6208684f1ff807b8f7e31a"} Oct 30 09:53:29.687420 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:29.687340 4562 scope.go:122] "RemoveContainer" containerID="9de20d9f01f9beef813e3a740e1a1ee8d4fd313f28f0d838c83a2aa488934283" Oct 30 09:53:29.712870 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:29.712851 4562 scope.go:122] "RemoveContainer" containerID="9de20d9f01f9beef813e3a740e1a1ee8d4fd313f28f0d838c83a2aa488934283" Oct 30 09:53:29.713150 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:53:29.713130 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9de20d9f01f9beef813e3a740e1a1ee8d4fd313f28f0d838c83a2aa488934283\": container with ID starting with 9de20d9f01f9beef813e3a740e1a1ee8d4fd313f28f0d838c83a2aa488934283 not found: ID does not exist" containerID="9de20d9f01f9beef813e3a740e1a1ee8d4fd313f28f0d838c83a2aa488934283" Oct 30 09:53:29.713202 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:29.713157 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9de20d9f01f9beef813e3a740e1a1ee8d4fd313f28f0d838c83a2aa488934283"} err="failed to get container status \"9de20d9f01f9beef813e3a740e1a1ee8d4fd313f28f0d838c83a2aa488934283\": rpc error: code = NotFound desc = could not find container \"9de20d9f01f9beef813e3a740e1a1ee8d4fd313f28f0d838c83a2aa488934283\": container with ID starting with 9de20d9f01f9beef813e3a740e1a1ee8d4fd313f28f0d838c83a2aa488934283 not found: ID does not exist" Oct 30 09:53:29.747301 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:29.747270 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-0ab35b/e2e-usage-sim-e02533-5d4ff8cf6d-455dp"] Oct 30 09:53:29.757008 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:29.756979 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-extmod-shr-user-0ab35b/e2e-usage-sim-e02533-5d4ff8cf6d-455dp"] Oct 30 09:53:29.877198 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:29.877109 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e32372d7-a2c3-44d4-9872-0ac6c3091bb9" path="/var/lib/kubelet/pods/e32372d7-a2c3-44d4-9872-0ac6c3091bb9/volumes" Oct 30 09:53:29.989696 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:29.989661 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-f76d4tcz6"] Oct 30 09:53:29.990498 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:29.990483 4562 cpu_manager.go:414] "containerMap: removing container" podUID="e32372d7-a2c3-44d4-9872-0ac6c3091bb9" containerName="sim" Oct 30 09:53:29.990548 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:29.990499 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e32372d7-a2c3-44d4-9872-0ac6c3091bb9" containerName="sim" Oct 30 09:53:29.990671 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:29.990662 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="e32372d7-a2c3-44d4-9872-0ac6c3091bb9" containerName="sim" Oct 30 09:53:29.995880 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:29.995860 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-f76d4tcz6" Oct 30 09:53:30.013747 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:30.013722 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-88cbc7\"/\"agc-cred-568a5da76981\"" Oct 30 09:53:30.013928 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:30.013806 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-88cbc7\"/\"routing-overlay\"" Oct 30 09:53:30.014623 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:30.014597 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-f76d4tcz6"] Oct 30 09:53:30.021219 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:30.021200 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-88cbc7\"/\"payload-processing-external-model-plugins-e2e-extmod-a-88cbc7\"" Oct 30 09:53:30.021335 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:30.021233 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-88cbc7\"/\"openshift-service-ca.crt\"" Oct 30 09:53:30.021335 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:30.021296 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-88cbc7\"/\"payload-processing-external-model-e2e--185456a2-dockercfg-dfrl7\"" Oct 30 09:53:30.021488 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:30.021473 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-88cbc7\"/\"kube-root-ca.crt\"" Oct 30 09:53:30.158124 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:30.158019 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ce4307bc-f5bb-40ae-bb86-a1458732170b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-f76d4tcz6\" (UID: \"ce4307bc-f5bb-40ae-bb86-a1458732170b\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-f76d4tcz6" Oct 30 09:53:30.158314 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:30.158141 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ce4307bc-f5bb-40ae-bb86-a1458732170b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-f76d4tcz6\" (UID: \"ce4307bc-f5bb-40ae-bb86-a1458732170b\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-f76d4tcz6" Oct 30 09:53:30.158314 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:30.158186 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce4307bc-f5bb-40ae-bb86-a1458732170b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-f76d4tcz6\" (UID: \"ce4307bc-f5bb-40ae-bb86-a1458732170b\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-f76d4tcz6" Oct 30 09:53:30.159456 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:30.158641 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9lc7\" (UniqueName: \"kubernetes.io/projected/ce4307bc-f5bb-40ae-bb86-a1458732170b-kube-api-access-w9lc7\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-f76d4tcz6\" (UID: \"ce4307bc-f5bb-40ae-bb86-a1458732170b\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-f76d4tcz6" Oct 30 09:53:30.259951 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:30.259913 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ce4307bc-f5bb-40ae-bb86-a1458732170b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-f76d4tcz6\" (UID: \"ce4307bc-f5bb-40ae-bb86-a1458732170b\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-f76d4tcz6" Oct 30 09:53:30.260154 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:30.260005 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ce4307bc-f5bb-40ae-bb86-a1458732170b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-f76d4tcz6\" (UID: \"ce4307bc-f5bb-40ae-bb86-a1458732170b\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-f76d4tcz6" Oct 30 09:53:30.260154 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:30.260036 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce4307bc-f5bb-40ae-bb86-a1458732170b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-f76d4tcz6\" (UID: \"ce4307bc-f5bb-40ae-bb86-a1458732170b\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-f76d4tcz6" Oct 30 09:53:30.260299 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:30.260230 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9lc7\" (UniqueName: \"kubernetes.io/projected/ce4307bc-f5bb-40ae-bb86-a1458732170b-kube-api-access-w9lc7\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-f76d4tcz6\" (UID: \"ce4307bc-f5bb-40ae-bb86-a1458732170b\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-f76d4tcz6" Oct 30 09:53:30.260845 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:30.260819 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ce4307bc-f5bb-40ae-bb86-a1458732170b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-f76d4tcz6\" (UID: \"ce4307bc-f5bb-40ae-bb86-a1458732170b\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-f76d4tcz6" Oct 30 09:53:30.260988 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:30.260882 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce4307bc-f5bb-40ae-bb86-a1458732170b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-f76d4tcz6\" (UID: \"ce4307bc-f5bb-40ae-bb86-a1458732170b\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-f76d4tcz6" Oct 30 09:53:30.262643 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:30.262598 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ce4307bc-f5bb-40ae-bb86-a1458732170b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-f76d4tcz6\" (UID: \"ce4307bc-f5bb-40ae-bb86-a1458732170b\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-f76d4tcz6" Oct 30 09:53:30.281137 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:30.281114 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9lc7\" (UniqueName: \"kubernetes.io/projected/ce4307bc-f5bb-40ae-bb86-a1458732170b-kube-api-access-w9lc7\") pod \"payload-processing-external-model-e2e-extmod-a-88cbc7-f76d4tcz6\" (UID: \"ce4307bc-f5bb-40ae-bb86-a1458732170b\") " pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-f76d4tcz6" Oct 30 09:53:30.313938 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:30.313910 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-f76d4tcz6" Oct 30 09:53:30.462016 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:30.461978 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-881f83-6b6777d984-fh7q9"] Oct 30 09:53:30.494921 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:30.494886 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-881f83-c579c5979-vj4zx"] Oct 30 09:53:30.629619 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:30.620812 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-f76d4tcz6"] Oct 30 09:53:30.698277 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:30.698246 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-f76d4tcz6" event={"ID":"ce4307bc-f5bb-40ae-bb86-a1458732170b","Type":"ContainerStarted","Data":"c7ab7493e145d7382804e3aeea09d3eb1fad3d74ce8a31d12f392f91ba921a96"} Oct 30 09:53:31.709626 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:31.709585 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-f76d4tcz6" event={"ID":"ce4307bc-f5bb-40ae-bb86-a1458732170b","Type":"ContainerStarted","Data":"fd276fd039054ed4ad1ed2817cc9b31b841878380e15e9b9e7a24f6b4012e4e2"} Oct 30 09:53:31.710135 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:31.709677 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-f76d4tcz6" Oct 30 09:53:31.750036 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:31.749977 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-f76d4tcz6" podStartSLOduration=2.749964439 podStartE2EDuration="2.749964439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:53:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:53:31.748603322 +0000 UTC m=+3572.010164323" watchObservedRunningTime="2026-10-30 09:53:31.749964439 +0000 UTC m=+3572.011525431" Oct 30 09:53:35.462538 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:35.462492 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-881f83-6b6777d984-fh7q9" podUID="169915d0-2153-46a4-a1af-a9d1670af675" containerName="payload-processing" containerID="cri-o://b1ca48ea1c2ce89477c54e3598289c25838533a471f2196df231db129e4caef4" gracePeriod=25 Oct 30 09:53:35.495209 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:35.495156 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-881f83-c579c5979-vj4zx" podUID="59e94911-c9dc-4fbd-b813-e22c0bc51fed" containerName="payload-pre-processing" containerID="cri-o://9ce672cce42e4e9ff3d8e8979f2a7e3dc14320e27c07728e245401d27e1efa76" gracePeriod=25 Oct 30 09:53:40.282164 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:40.282125 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-7dc7rbmf2"] Oct 30 09:53:42.314306 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:42.314265 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-85fdzxq9s"] Oct 30 09:53:42.772302 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:42.772266 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-f76d4tcz6" Oct 30 09:53:43.812093 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:43.812052 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-881f83-c579c5979-vj4zx" podUID="59e94911-c9dc-4fbd-b813-e22c0bc51fed" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:53:44.894887 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:44.894850 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-881f83-6b6777d984-fh7q9" podUID="169915d0-2153-46a4-a1af-a9d1670af675" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:53:45.283393 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:45.283329 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-7dc7rbmf2" podUID="b1a4232c-bb1f-4fb4-921a-8ada7bc47d6c" containerName="payload-processing" containerID="cri-o://1fb3e96233b04078b673692c17b3584359b8e8cac26f254330a2de9ddf9b3953" gracePeriod=25 Oct 30 09:53:45.575422 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:45.575395 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-7dc7rbmf2" Oct 30 09:53:45.630166 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:45.630128 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1a4232c-bb1f-4fb4-921a-8ada7bc47d6c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b1a4232c-bb1f-4fb4-921a-8ada7bc47d6c-provider-credentials\") pod \"b1a4232c-bb1f-4fb4-921a-8ada7bc47d6c\" (UID: \"b1a4232c-bb1f-4fb4-921a-8ada7bc47d6c\") " Oct 30 09:53:45.630491 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:45.630464 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1a4232c-bb1f-4fb4-921a-8ada7bc47d6c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b1a4232c-bb1f-4fb4-921a-8ada7bc47d6c-routing-overlay-volume\") pod \"b1a4232c-bb1f-4fb4-921a-8ada7bc47d6c\" (UID: \"b1a4232c-bb1f-4fb4-921a-8ada7bc47d6c\") " Oct 30 09:53:45.630627 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:45.630538 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1a4232c-bb1f-4fb4-921a-8ada7bc47d6c-kube-api-access-7w5w5\" (UniqueName: \"kubernetes.io/projected/b1a4232c-bb1f-4fb4-921a-8ada7bc47d6c-kube-api-access-7w5w5\") pod \"b1a4232c-bb1f-4fb4-921a-8ada7bc47d6c\" (UID: \"b1a4232c-bb1f-4fb4-921a-8ada7bc47d6c\") " Oct 30 09:53:45.630742 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:45.630717 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1a4232c-bb1f-4fb4-921a-8ada7bc47d6c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1a4232c-bb1f-4fb4-921a-8ada7bc47d6c-plugins-config-volume\") pod \"b1a4232c-bb1f-4fb4-921a-8ada7bc47d6c\" (UID: \"b1a4232c-bb1f-4fb4-921a-8ada7bc47d6c\") " Oct 30 09:53:45.630932 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:45.630904 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1a4232c-bb1f-4fb4-921a-8ada7bc47d6c-routing-overlay-volume" pod "b1a4232c-bb1f-4fb4-921a-8ada7bc47d6c" (UID: "b1a4232c-bb1f-4fb4-921a-8ada7bc47d6c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:53:45.631484 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:45.631461 4562 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b1a4232c-bb1f-4fb4-921a-8ada7bc47d6c-routing-overlay-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:53:45.631841 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:45.631818 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1a4232c-bb1f-4fb4-921a-8ada7bc47d6c-plugins-config-volume" pod "b1a4232c-bb1f-4fb4-921a-8ada7bc47d6c" (UID: "b1a4232c-bb1f-4fb4-921a-8ada7bc47d6c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:53:45.633449 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:45.633375 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1a4232c-bb1f-4fb4-921a-8ada7bc47d6c-provider-credentials" pod "b1a4232c-bb1f-4fb4-921a-8ada7bc47d6c" (UID: "b1a4232c-bb1f-4fb4-921a-8ada7bc47d6c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:53:45.633557 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:45.633462 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1a4232c-bb1f-4fb4-921a-8ada7bc47d6c-kube-api-access-7w5w5" pod "b1a4232c-bb1f-4fb4-921a-8ada7bc47d6c" (UID: "b1a4232c-bb1f-4fb4-921a-8ada7bc47d6c"). InnerVolumeSpecName "kube-api-access-7w5w5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:53:45.733140 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:45.733101 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1a4232c-bb1f-4fb4-921a-8ada7bc47d6c-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:53:45.733140 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:45.733137 4562 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b1a4232c-bb1f-4fb4-921a-8ada7bc47d6c-provider-credentials\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:53:45.733140 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:45.733146 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7w5w5\" (UniqueName: \"kubernetes.io/projected/b1a4232c-bb1f-4fb4-921a-8ada7bc47d6c-kube-api-access-7w5w5\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:53:45.838957 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:45.838852 4562 generic.go:356] "Generic (PLEG): container finished" podID="b1a4232c-bb1f-4fb4-921a-8ada7bc47d6c" containerID="1fb3e96233b04078b673692c17b3584359b8e8cac26f254330a2de9ddf9b3953" exitCode=0 Oct 30 09:53:45.838957 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:45.838905 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-7dc7rbmf2" Oct 30 09:53:45.838957 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:45.838923 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-7dc7rbmf2" event={"ID":"b1a4232c-bb1f-4fb4-921a-8ada7bc47d6c","Type":"ContainerDied","Data":"1fb3e96233b04078b673692c17b3584359b8e8cac26f254330a2de9ddf9b3953"} Oct 30 09:53:45.839203 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:45.838990 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-7dc7rbmf2" event={"ID":"b1a4232c-bb1f-4fb4-921a-8ada7bc47d6c","Type":"ContainerDied","Data":"3aa3eda39dc98611400c9c54c879e99edb6c116f689cddf8f00fc68ad00e9840"} Oct 30 09:53:45.839203 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:45.839023 4562 scope.go:122] "RemoveContainer" containerID="1fb3e96233b04078b673692c17b3584359b8e8cac26f254330a2de9ddf9b3953" Oct 30 09:53:45.858183 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:45.858156 4562 scope.go:122] "RemoveContainer" containerID="1fb3e96233b04078b673692c17b3584359b8e8cac26f254330a2de9ddf9b3953" Oct 30 09:53:45.858582 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:53:45.858551 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fb3e96233b04078b673692c17b3584359b8e8cac26f254330a2de9ddf9b3953\": container with ID starting with 1fb3e96233b04078b673692c17b3584359b8e8cac26f254330a2de9ddf9b3953 not found: ID does not exist" containerID="1fb3e96233b04078b673692c17b3584359b8e8cac26f254330a2de9ddf9b3953" Oct 30 09:53:45.858696 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:45.858590 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fb3e96233b04078b673692c17b3584359b8e8cac26f254330a2de9ddf9b3953"} err="failed to get container status \"1fb3e96233b04078b673692c17b3584359b8e8cac26f254330a2de9ddf9b3953\": rpc error: code = NotFound desc = could not find container \"1fb3e96233b04078b673692c17b3584359b8e8cac26f254330a2de9ddf9b3953\": container with ID starting with 1fb3e96233b04078b673692c17b3584359b8e8cac26f254330a2de9ddf9b3953 not found: ID does not exist" Oct 30 09:53:45.899995 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:45.899960 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-7dc7rbmf2"] Oct 30 09:53:45.904455 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:45.904425 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-d60c68/payload-processing-external-model-e2e-extmod-a-d60c68-7dc7rbmf2"] Oct 30 09:53:47.321415 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:47.318555 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-85fdzxq9s" podUID="06051f1f-c3c6-4c87-8cd4-1cb7b6c4f170" containerName="payload-processing" containerID="cri-o://d9b90853d7460963a64fd012b39b8edeb59a925e121feaea142da39265370fbc" gracePeriod=25 Oct 30 09:53:47.935363 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:47.935316 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1a4232c-bb1f-4fb4-921a-8ada7bc47d6c" path="/var/lib/kubelet/pods/b1a4232c-bb1f-4fb4-921a-8ada7bc47d6c/volumes" Oct 30 09:53:48.024820 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:48.024781 4562 generic.go:356] "Generic (PLEG): container finished" podID="06051f1f-c3c6-4c87-8cd4-1cb7b6c4f170" containerID="d9b90853d7460963a64fd012b39b8edeb59a925e121feaea142da39265370fbc" exitCode=0 Oct 30 09:53:48.025302 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:48.025049 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-85fdzxq9s" event={"ID":"06051f1f-c3c6-4c87-8cd4-1cb7b6c4f170","Type":"ContainerDied","Data":"d9b90853d7460963a64fd012b39b8edeb59a925e121feaea142da39265370fbc"} Oct 30 09:53:48.168453 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:48.168066 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-85fdzxq9s" Oct 30 09:53:48.307225 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:48.307169 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06051f1f-c3c6-4c87-8cd4-1cb7b6c4f170-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06051f1f-c3c6-4c87-8cd4-1cb7b6c4f170-plugins-config-volume\") pod \"06051f1f-c3c6-4c87-8cd4-1cb7b6c4f170\" (UID: \"06051f1f-c3c6-4c87-8cd4-1cb7b6c4f170\") " Oct 30 09:53:48.307462 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:48.307256 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06051f1f-c3c6-4c87-8cd4-1cb7b6c4f170-provider-credentials\" (UniqueName: \"kubernetes.io/projected/06051f1f-c3c6-4c87-8cd4-1cb7b6c4f170-provider-credentials\") pod \"06051f1f-c3c6-4c87-8cd4-1cb7b6c4f170\" (UID: \"06051f1f-c3c6-4c87-8cd4-1cb7b6c4f170\") " Oct 30 09:53:48.307462 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:48.307329 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06051f1f-c3c6-4c87-8cd4-1cb7b6c4f170-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/06051f1f-c3c6-4c87-8cd4-1cb7b6c4f170-routing-overlay-volume\") pod \"06051f1f-c3c6-4c87-8cd4-1cb7b6c4f170\" (UID: \"06051f1f-c3c6-4c87-8cd4-1cb7b6c4f170\") " Oct 30 09:53:48.307617 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:48.307481 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06051f1f-c3c6-4c87-8cd4-1cb7b6c4f170-kube-api-access-l9rv7\" (UniqueName: \"kubernetes.io/projected/06051f1f-c3c6-4c87-8cd4-1cb7b6c4f170-kube-api-access-l9rv7\") pod \"06051f1f-c3c6-4c87-8cd4-1cb7b6c4f170\" (UID: \"06051f1f-c3c6-4c87-8cd4-1cb7b6c4f170\") " Oct 30 09:53:48.309158 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:48.309127 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06051f1f-c3c6-4c87-8cd4-1cb7b6c4f170-plugins-config-volume" pod "06051f1f-c3c6-4c87-8cd4-1cb7b6c4f170" (UID: "06051f1f-c3c6-4c87-8cd4-1cb7b6c4f170"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:53:48.309735 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:48.309711 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06051f1f-c3c6-4c87-8cd4-1cb7b6c4f170-routing-overlay-volume" pod "06051f1f-c3c6-4c87-8cd4-1cb7b6c4f170" (UID: "06051f1f-c3c6-4c87-8cd4-1cb7b6c4f170"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:53:48.320475 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:48.318037 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06051f1f-c3c6-4c87-8cd4-1cb7b6c4f170-provider-credentials" pod "06051f1f-c3c6-4c87-8cd4-1cb7b6c4f170" (UID: "06051f1f-c3c6-4c87-8cd4-1cb7b6c4f170"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:53:48.320475 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:48.318079 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06051f1f-c3c6-4c87-8cd4-1cb7b6c4f170-kube-api-access-l9rv7" pod "06051f1f-c3c6-4c87-8cd4-1cb7b6c4f170" (UID: "06051f1f-c3c6-4c87-8cd4-1cb7b6c4f170"). InnerVolumeSpecName "kube-api-access-l9rv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:53:48.448405 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:48.446625 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06051f1f-c3c6-4c87-8cd4-1cb7b6c4f170-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:53:48.448405 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:48.446664 4562 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/06051f1f-c3c6-4c87-8cd4-1cb7b6c4f170-provider-credentials\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:53:48.448405 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:48.446680 4562 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/06051f1f-c3c6-4c87-8cd4-1cb7b6c4f170-routing-overlay-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:53:48.448405 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:48.446696 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9rv7\" (UniqueName: \"kubernetes.io/projected/06051f1f-c3c6-4c87-8cd4-1cb7b6c4f170-kube-api-access-l9rv7\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:53:49.055418 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:49.045766 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-85fdzxq9s" Oct 30 09:53:49.093415 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:49.088433 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-85fdzxq9s" event={"ID":"06051f1f-c3c6-4c87-8cd4-1cb7b6c4f170","Type":"ContainerDied","Data":"22eb32d2a142179e86fc7b2de5f0493dbb93c8edea9b6405155935be6886f6c1"} Oct 30 09:53:49.093415 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:49.088531 4562 scope.go:122] "RemoveContainer" containerID="d9b90853d7460963a64fd012b39b8edeb59a925e121feaea142da39265370fbc" Oct 30 09:53:49.208428 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:49.206312 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-85fdzxq9s"] Oct 30 09:53:49.254416 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:49.251465 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-3d9b36/payload-processing-external-model-e2e-extmod-b-3d9b36-85fdzxq9s"] Oct 30 09:53:49.925491 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:49.924855 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06051f1f-c3c6-4c87-8cd4-1cb7b6c4f170" path="/var/lib/kubelet/pods/06051f1f-c3c6-4c87-8cd4-1cb7b6c4f170/volumes" Oct 30 09:53:50.323467 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:50.322742 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd"] Oct 30 09:53:50.323467 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:50.322944 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd" podUID="2fca872d-4848-4886-925d-e06a5bcc2526" containerName="istio-proxy" containerID="cri-o://0d8ed9c24ec64c9580dd8e6a22b23007907dc8e934b7fdfd476d2e8ec0ffc701" gracePeriod=30 Oct 30 09:53:51.570637 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:51.570600 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h"] Oct 30 09:53:51.571283 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:51.570869 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h" podUID="aea403b0-3184-48ef-8e1a-b1f91b56d703" containerName="istio-proxy" containerID="cri-o://7a0381a04c9a072371781cd7fb53195ead4de0500e0be0e06ed9bb0ab2fe83ac" gracePeriod=30 Oct 30 09:53:52.711458 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:52.711424 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-f76d4tcz6"] Oct 30 09:53:53.812414 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:53.812342 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-881f83-c579c5979-vj4zx" podUID="59e94911-c9dc-4fbd-b813-e22c0bc51fed" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:53:54.307476 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:54.307438 4562 patch_prober.go:29] interesting pod/e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 09:53:54.307674 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:54.307504 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd" podUID="2fca872d-4848-4886-925d-e06a5bcc2526" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 09:53:54.894043 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:54.894002 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-881f83-6b6777d984-fh7q9" podUID="169915d0-2153-46a4-a1af-a9d1670af675" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:53:55.631432 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:55.631144 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd" Oct 30 09:53:55.696775 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:55.696742 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fca872d-4848-4886-925d-e06a5bcc2526-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fca872d-4848-4886-925d-e06a5bcc2526-istio-data\") pod \"2fca872d-4848-4886-925d-e06a5bcc2526\" (UID: \"2fca872d-4848-4886-925d-e06a5bcc2526\") " Oct 30 09:53:55.697075 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:55.697054 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fca872d-4848-4886-925d-e06a5bcc2526-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fca872d-4848-4886-925d-e06a5bcc2526-credential-socket\") pod \"2fca872d-4848-4886-925d-e06a5bcc2526\" (UID: \"2fca872d-4848-4886-925d-e06a5bcc2526\") " Oct 30 09:53:55.697200 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:55.697187 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fca872d-4848-4886-925d-e06a5bcc2526-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fca872d-4848-4886-925d-e06a5bcc2526-istiod-ca-cert\") pod \"2fca872d-4848-4886-925d-e06a5bcc2526\" (UID: \"2fca872d-4848-4886-925d-e06a5bcc2526\") " Oct 30 09:53:55.697308 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:55.697295 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fca872d-4848-4886-925d-e06a5bcc2526-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fca872d-4848-4886-925d-e06a5bcc2526-workload-certs\") pod \"2fca872d-4848-4886-925d-e06a5bcc2526\" (UID: \"2fca872d-4848-4886-925d-e06a5bcc2526\") " Oct 30 09:53:55.697483 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:55.697468 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fca872d-4848-4886-925d-e06a5bcc2526-kube-api-access-882g7\" (UniqueName: \"kubernetes.io/projected/2fca872d-4848-4886-925d-e06a5bcc2526-kube-api-access-882g7\") pod \"2fca872d-4848-4886-925d-e06a5bcc2526\" (UID: \"2fca872d-4848-4886-925d-e06a5bcc2526\") " Oct 30 09:53:55.697705 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:55.697690 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2fca872d-4848-4886-925d-e06a5bcc2526-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fca872d-4848-4886-925d-e06a5bcc2526-istio-podinfo\") pod \"2fca872d-4848-4886-925d-e06a5bcc2526\" (UID: \"2fca872d-4848-4886-925d-e06a5bcc2526\") " Oct 30 09:53:55.697838 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:55.697824 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fca872d-4848-4886-925d-e06a5bcc2526-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fca872d-4848-4886-925d-e06a5bcc2526-workload-socket\") pod \"2fca872d-4848-4886-925d-e06a5bcc2526\" (UID: \"2fca872d-4848-4886-925d-e06a5bcc2526\") " Oct 30 09:53:55.698006 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:55.697991 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fca872d-4848-4886-925d-e06a5bcc2526-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fca872d-4848-4886-925d-e06a5bcc2526-istio-envoy\") pod \"2fca872d-4848-4886-925d-e06a5bcc2526\" (UID: \"2fca872d-4848-4886-925d-e06a5bcc2526\") " Oct 30 09:53:55.698202 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:55.698190 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fca872d-4848-4886-925d-e06a5bcc2526-istio-token\" (UniqueName: \"kubernetes.io/projected/2fca872d-4848-4886-925d-e06a5bcc2526-istio-token\") pod \"2fca872d-4848-4886-925d-e06a5bcc2526\" (UID: \"2fca872d-4848-4886-925d-e06a5bcc2526\") " Oct 30 09:53:55.703226 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:55.703078 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fca872d-4848-4886-925d-e06a5bcc2526-istio-token" pod "2fca872d-4848-4886-925d-e06a5bcc2526" (UID: "2fca872d-4848-4886-925d-e06a5bcc2526"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:53:55.703625 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:55.703360 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fca872d-4848-4886-925d-e06a5bcc2526-workload-socket" pod "2fca872d-4848-4886-925d-e06a5bcc2526" (UID: "2fca872d-4848-4886-925d-e06a5bcc2526"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:53:55.704549 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:55.704256 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2fca872d-4848-4886-925d-e06a5bcc2526-istio-podinfo" pod "2fca872d-4848-4886-925d-e06a5bcc2526" (UID: "2fca872d-4848-4886-925d-e06a5bcc2526"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:53:55.708048 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:55.707476 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fca872d-4848-4886-925d-e06a5bcc2526-workload-certs" pod "2fca872d-4848-4886-925d-e06a5bcc2526" (UID: "2fca872d-4848-4886-925d-e06a5bcc2526"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:53:55.709898 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:55.709450 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fca872d-4848-4886-925d-e06a5bcc2526-istio-data" pod "2fca872d-4848-4886-925d-e06a5bcc2526" (UID: "2fca872d-4848-4886-925d-e06a5bcc2526"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:53:55.710639 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:55.710260 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fca872d-4848-4886-925d-e06a5bcc2526-credential-socket" pod "2fca872d-4848-4886-925d-e06a5bcc2526" (UID: "2fca872d-4848-4886-925d-e06a5bcc2526"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:53:55.710639 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:55.710576 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fca872d-4848-4886-925d-e06a5bcc2526-istiod-ca-cert" pod "2fca872d-4848-4886-925d-e06a5bcc2526" (UID: "2fca872d-4848-4886-925d-e06a5bcc2526"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:53:55.712061 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:55.712026 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fca872d-4848-4886-925d-e06a5bcc2526-istio-envoy" pod "2fca872d-4848-4886-925d-e06a5bcc2526" (UID: "2fca872d-4848-4886-925d-e06a5bcc2526"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:53:55.714725 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:55.714657 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fca872d-4848-4886-925d-e06a5bcc2526-kube-api-access-882g7" pod "2fca872d-4848-4886-925d-e06a5bcc2526" (UID: "2fca872d-4848-4886-925d-e06a5bcc2526"). InnerVolumeSpecName "kube-api-access-882g7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:53:55.770701 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:55.770674 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-881f83-6b6777d984-fh7q9" Oct 30 09:53:55.816798 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:55.814775 4562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fca872d-4848-4886-925d-e06a5bcc2526-istio-envoy\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:53:55.816798 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:55.814812 4562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fca872d-4848-4886-925d-e06a5bcc2526-istio-token\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:53:55.816798 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:55.814826 4562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fca872d-4848-4886-925d-e06a5bcc2526-istio-data\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:53:55.816798 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:55.814839 4562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fca872d-4848-4886-925d-e06a5bcc2526-credential-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:53:55.816798 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:55.814853 4562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fca872d-4848-4886-925d-e06a5bcc2526-istiod-ca-cert\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:53:55.816798 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:55.814878 4562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fca872d-4848-4886-925d-e06a5bcc2526-workload-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:53:55.816798 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:55.814887 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-882g7\" (UniqueName: \"kubernetes.io/projected/2fca872d-4848-4886-925d-e06a5bcc2526-kube-api-access-882g7\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:53:55.816798 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:55.814896 4562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fca872d-4848-4886-925d-e06a5bcc2526-istio-podinfo\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:53:55.816798 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:55.814905 4562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fca872d-4848-4886-925d-e06a5bcc2526-workload-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:53:55.837317 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:55.837282 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-881f83-c579c5979-vj4zx" Oct 30 09:53:55.916711 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:55.916675 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59e94911-c9dc-4fbd-b813-e22c0bc51fed-kube-api-access-qjpb2\" (UniqueName: \"kubernetes.io/projected/59e94911-c9dc-4fbd-b813-e22c0bc51fed-kube-api-access-qjpb2\") pod \"59e94911-c9dc-4fbd-b813-e22c0bc51fed\" (UID: \"59e94911-c9dc-4fbd-b813-e22c0bc51fed\") " Oct 30 09:53:55.917213 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:55.916734 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59e94911-c9dc-4fbd-b813-e22c0bc51fed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59e94911-c9dc-4fbd-b813-e22c0bc51fed-plugins-config-volume\") pod \"59e94911-c9dc-4fbd-b813-e22c0bc51fed\" (UID: \"59e94911-c9dc-4fbd-b813-e22c0bc51fed\") " Oct 30 09:53:55.917213 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:55.916789 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/169915d0-2153-46a4-a1af-a9d1670af675-kube-api-access-qtp99\" (UniqueName: \"kubernetes.io/projected/169915d0-2153-46a4-a1af-a9d1670af675-kube-api-access-qtp99\") pod \"169915d0-2153-46a4-a1af-a9d1670af675\" (UID: \"169915d0-2153-46a4-a1af-a9d1670af675\") " Oct 30 09:53:55.917213 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:55.916824 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/169915d0-2153-46a4-a1af-a9d1670af675-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/169915d0-2153-46a4-a1af-a9d1670af675-plugins-config-volume\") pod \"169915d0-2153-46a4-a1af-a9d1670af675\" (UID: \"169915d0-2153-46a4-a1af-a9d1670af675\") " Oct 30 09:53:55.917213 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:55.917166 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59e94911-c9dc-4fbd-b813-e22c0bc51fed-plugins-config-volume" pod "59e94911-c9dc-4fbd-b813-e22c0bc51fed" (UID: "59e94911-c9dc-4fbd-b813-e22c0bc51fed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:53:55.917508 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:55.917320 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/169915d0-2153-46a4-a1af-a9d1670af675-plugins-config-volume" pod "169915d0-2153-46a4-a1af-a9d1670af675" (UID: "169915d0-2153-46a4-a1af-a9d1670af675"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:53:55.917869 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:55.917843 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59e94911-c9dc-4fbd-b813-e22c0bc51fed-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:53:55.917962 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:55.917874 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/169915d0-2153-46a4-a1af-a9d1670af675-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:53:55.919338 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:55.919286 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/169915d0-2153-46a4-a1af-a9d1670af675-kube-api-access-qtp99" pod "169915d0-2153-46a4-a1af-a9d1670af675" (UID: "169915d0-2153-46a4-a1af-a9d1670af675"). InnerVolumeSpecName "kube-api-access-qtp99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:53:55.919475 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:55.919360 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59e94911-c9dc-4fbd-b813-e22c0bc51fed-kube-api-access-qjpb2" pod "59e94911-c9dc-4fbd-b813-e22c0bc51fed" (UID: "59e94911-c9dc-4fbd-b813-e22c0bc51fed"). InnerVolumeSpecName "kube-api-access-qjpb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:53:56.019858 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:56.019823 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjpb2\" (UniqueName: \"kubernetes.io/projected/59e94911-c9dc-4fbd-b813-e22c0bc51fed-kube-api-access-qjpb2\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:53:56.019858 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:56.019855 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtp99\" (UniqueName: \"kubernetes.io/projected/169915d0-2153-46a4-a1af-a9d1670af675-kube-api-access-qtp99\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:53:56.351072 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:56.351035 4562 generic.go:356] "Generic (PLEG): container finished" podID="59e94911-c9dc-4fbd-b813-e22c0bc51fed" containerID="9ce672cce42e4e9ff3d8e8979f2a7e3dc14320e27c07728e245401d27e1efa76" exitCode=0 Oct 30 09:53:56.351278 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:56.351117 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-881f83-c579c5979-vj4zx" Oct 30 09:53:56.351278 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:56.351124 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-881f83-c579c5979-vj4zx" event={"ID":"59e94911-c9dc-4fbd-b813-e22c0bc51fed","Type":"ContainerDied","Data":"9ce672cce42e4e9ff3d8e8979f2a7e3dc14320e27c07728e245401d27e1efa76"} Oct 30 09:53:56.351278 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:56.351191 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-881f83-c579c5979-vj4zx" event={"ID":"59e94911-c9dc-4fbd-b813-e22c0bc51fed","Type":"ContainerDied","Data":"a88d51354e50f37782c23d614fd20ec0dd9b6c0d4071ed684ca2d34e078f49ad"} Oct 30 09:53:56.351278 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:56.351222 4562 scope.go:122] "RemoveContainer" containerID="9ce672cce42e4e9ff3d8e8979f2a7e3dc14320e27c07728e245401d27e1efa76" Oct 30 09:53:56.352934 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:56.352907 4562 generic.go:356] "Generic (PLEG): container finished" podID="2fca872d-4848-4886-925d-e06a5bcc2526" containerID="0d8ed9c24ec64c9580dd8e6a22b23007907dc8e934b7fdfd476d2e8ec0ffc701" exitCode=0 Oct 30 09:53:56.353064 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:56.352942 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd" event={"ID":"2fca872d-4848-4886-925d-e06a5bcc2526","Type":"ContainerDied","Data":"0d8ed9c24ec64c9580dd8e6a22b23007907dc8e934b7fdfd476d2e8ec0ffc701"} Oct 30 09:53:56.353064 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:56.352998 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd" event={"ID":"2fca872d-4848-4886-925d-e06a5bcc2526","Type":"ContainerDied","Data":"edaed962556cca9ef72cb59367ec9ad421426f62b31af1d95f1e386482ae2fe9"} Oct 30 09:53:56.353273 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:56.353249 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd" Oct 30 09:53:56.354736 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:56.354711 4562 generic.go:356] "Generic (PLEG): container finished" podID="169915d0-2153-46a4-a1af-a9d1670af675" containerID="b1ca48ea1c2ce89477c54e3598289c25838533a471f2196df231db129e4caef4" exitCode=0 Oct 30 09:53:56.354837 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:56.354740 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-881f83-6b6777d984-fh7q9" Oct 30 09:53:56.354837 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:56.354750 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-881f83-6b6777d984-fh7q9" event={"ID":"169915d0-2153-46a4-a1af-a9d1670af675","Type":"ContainerDied","Data":"b1ca48ea1c2ce89477c54e3598289c25838533a471f2196df231db129e4caef4"} Oct 30 09:53:56.354837 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:56.354798 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-881f83-6b6777d984-fh7q9" event={"ID":"169915d0-2153-46a4-a1af-a9d1670af675","Type":"ContainerDied","Data":"177af5e069cacc05a7eca6e290c570491f42cb69acae997e1c8d800a8e5bee99"} Oct 30 09:53:56.376705 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:56.376684 4562 scope.go:122] "RemoveContainer" containerID="9ce672cce42e4e9ff3d8e8979f2a7e3dc14320e27c07728e245401d27e1efa76" Oct 30 09:53:56.377160 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:53:56.377135 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ce672cce42e4e9ff3d8e8979f2a7e3dc14320e27c07728e245401d27e1efa76\": container with ID starting with 9ce672cce42e4e9ff3d8e8979f2a7e3dc14320e27c07728e245401d27e1efa76 not found: ID does not exist" containerID="9ce672cce42e4e9ff3d8e8979f2a7e3dc14320e27c07728e245401d27e1efa76" Oct 30 09:53:56.377246 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:56.377166 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ce672cce42e4e9ff3d8e8979f2a7e3dc14320e27c07728e245401d27e1efa76"} err="failed to get container status \"9ce672cce42e4e9ff3d8e8979f2a7e3dc14320e27c07728e245401d27e1efa76\": rpc error: code = NotFound desc = could not find container \"9ce672cce42e4e9ff3d8e8979f2a7e3dc14320e27c07728e245401d27e1efa76\": container with ID starting with 9ce672cce42e4e9ff3d8e8979f2a7e3dc14320e27c07728e245401d27e1efa76 not found: ID does not exist" Oct 30 09:53:56.377246 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:56.377183 4562 scope.go:122] "RemoveContainer" containerID="0d8ed9c24ec64c9580dd8e6a22b23007907dc8e934b7fdfd476d2e8ec0ffc701" Oct 30 09:53:56.387538 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:56.387503 4562 scope.go:122] "RemoveContainer" containerID="0d8ed9c24ec64c9580dd8e6a22b23007907dc8e934b7fdfd476d2e8ec0ffc701" Oct 30 09:53:56.387771 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:53:56.387750 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d8ed9c24ec64c9580dd8e6a22b23007907dc8e934b7fdfd476d2e8ec0ffc701\": container with ID starting with 0d8ed9c24ec64c9580dd8e6a22b23007907dc8e934b7fdfd476d2e8ec0ffc701 not found: ID does not exist" containerID="0d8ed9c24ec64c9580dd8e6a22b23007907dc8e934b7fdfd476d2e8ec0ffc701" Oct 30 09:53:56.387820 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:56.387776 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d8ed9c24ec64c9580dd8e6a22b23007907dc8e934b7fdfd476d2e8ec0ffc701"} err="failed to get container status \"0d8ed9c24ec64c9580dd8e6a22b23007907dc8e934b7fdfd476d2e8ec0ffc701\": rpc error: code = NotFound desc = could not find container \"0d8ed9c24ec64c9580dd8e6a22b23007907dc8e934b7fdfd476d2e8ec0ffc701\": container with ID starting with 0d8ed9c24ec64c9580dd8e6a22b23007907dc8e934b7fdfd476d2e8ec0ffc701 not found: ID does not exist" Oct 30 09:53:56.387820 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:56.387799 4562 scope.go:122] "RemoveContainer" containerID="b1ca48ea1c2ce89477c54e3598289c25838533a471f2196df231db129e4caef4" Oct 30 09:53:56.398308 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:56.398278 4562 scope.go:122] "RemoveContainer" containerID="b1ca48ea1c2ce89477c54e3598289c25838533a471f2196df231db129e4caef4" Oct 30 09:53:56.398693 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:53:56.398672 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1ca48ea1c2ce89477c54e3598289c25838533a471f2196df231db129e4caef4\": container with ID starting with b1ca48ea1c2ce89477c54e3598289c25838533a471f2196df231db129e4caef4 not found: ID does not exist" containerID="b1ca48ea1c2ce89477c54e3598289c25838533a471f2196df231db129e4caef4" Oct 30 09:53:56.398833 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:56.398699 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1ca48ea1c2ce89477c54e3598289c25838533a471f2196df231db129e4caef4"} err="failed to get container status \"b1ca48ea1c2ce89477c54e3598289c25838533a471f2196df231db129e4caef4\": rpc error: code = NotFound desc = could not find container \"b1ca48ea1c2ce89477c54e3598289c25838533a471f2196df231db129e4caef4\": container with ID starting with b1ca48ea1c2ce89477c54e3598289c25838533a471f2196df231db129e4caef4 not found: ID does not exist" Oct 30 09:53:56.440754 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:56.440713 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd"] Oct 30 09:53:56.452540 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:56.452510 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-58rfd"] Oct 30 09:53:56.478790 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:56.478761 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-881f83-6b6777d984-fh7q9"] Oct 30 09:53:56.483062 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:56.483042 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-881f83-6b6777d984-fh7q9"] Oct 30 09:53:56.517232 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:56.517206 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-881f83-c579c5979-vj4zx"] Oct 30 09:53:56.522525 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:56.522503 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-881f83-c579c5979-vj4zx"] Oct 30 09:53:56.887540 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:56.887493 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h" Oct 30 09:53:57.036970 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:57.036931 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aea403b0-3184-48ef-8e1a-b1f91b56d703-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aea403b0-3184-48ef-8e1a-b1f91b56d703-workload-certs\") pod \"aea403b0-3184-48ef-8e1a-b1f91b56d703\" (UID: \"aea403b0-3184-48ef-8e1a-b1f91b56d703\") " Oct 30 09:53:57.037493 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:57.036997 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aea403b0-3184-48ef-8e1a-b1f91b56d703-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aea403b0-3184-48ef-8e1a-b1f91b56d703-workload-socket\") pod \"aea403b0-3184-48ef-8e1a-b1f91b56d703\" (UID: \"aea403b0-3184-48ef-8e1a-b1f91b56d703\") " Oct 30 09:53:57.037493 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:57.037068 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aea403b0-3184-48ef-8e1a-b1f91b56d703-kube-api-access-sbvb2\" (UniqueName: \"kubernetes.io/projected/aea403b0-3184-48ef-8e1a-b1f91b56d703-kube-api-access-sbvb2\") pod \"aea403b0-3184-48ef-8e1a-b1f91b56d703\" (UID: \"aea403b0-3184-48ef-8e1a-b1f91b56d703\") " Oct 30 09:53:57.037493 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:57.037147 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aea403b0-3184-48ef-8e1a-b1f91b56d703-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aea403b0-3184-48ef-8e1a-b1f91b56d703-istio-envoy\") pod \"aea403b0-3184-48ef-8e1a-b1f91b56d703\" (UID: \"aea403b0-3184-48ef-8e1a-b1f91b56d703\") " Oct 30 09:53:57.037493 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:57.037190 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aea403b0-3184-48ef-8e1a-b1f91b56d703-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aea403b0-3184-48ef-8e1a-b1f91b56d703-credential-socket\") pod \"aea403b0-3184-48ef-8e1a-b1f91b56d703\" (UID: \"aea403b0-3184-48ef-8e1a-b1f91b56d703\") " Oct 30 09:53:57.037493 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:57.037236 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aea403b0-3184-48ef-8e1a-b1f91b56d703-istio-token\" (UniqueName: \"kubernetes.io/projected/aea403b0-3184-48ef-8e1a-b1f91b56d703-istio-token\") pod \"aea403b0-3184-48ef-8e1a-b1f91b56d703\" (UID: \"aea403b0-3184-48ef-8e1a-b1f91b56d703\") " Oct 30 09:53:57.037493 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:57.037267 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aea403b0-3184-48ef-8e1a-b1f91b56d703-workload-certs" pod "aea403b0-3184-48ef-8e1a-b1f91b56d703" (UID: "aea403b0-3184-48ef-8e1a-b1f91b56d703"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:53:57.037493 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:57.037330 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aea403b0-3184-48ef-8e1a-b1f91b56d703-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aea403b0-3184-48ef-8e1a-b1f91b56d703-istiod-ca-cert\") pod \"aea403b0-3184-48ef-8e1a-b1f91b56d703\" (UID: \"aea403b0-3184-48ef-8e1a-b1f91b56d703\") " Oct 30 09:53:57.037493 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:57.037359 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aea403b0-3184-48ef-8e1a-b1f91b56d703-workload-socket" pod "aea403b0-3184-48ef-8e1a-b1f91b56d703" (UID: "aea403b0-3184-48ef-8e1a-b1f91b56d703"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:53:57.038006 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:57.037622 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aea403b0-3184-48ef-8e1a-b1f91b56d703-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aea403b0-3184-48ef-8e1a-b1f91b56d703-istio-podinfo\") pod \"aea403b0-3184-48ef-8e1a-b1f91b56d703\" (UID: \"aea403b0-3184-48ef-8e1a-b1f91b56d703\") " Oct 30 09:53:57.038006 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:57.037695 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aea403b0-3184-48ef-8e1a-b1f91b56d703-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aea403b0-3184-48ef-8e1a-b1f91b56d703-istio-data\") pod \"aea403b0-3184-48ef-8e1a-b1f91b56d703\" (UID: \"aea403b0-3184-48ef-8e1a-b1f91b56d703\") " Oct 30 09:53:57.038006 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:57.037784 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aea403b0-3184-48ef-8e1a-b1f91b56d703-credential-socket" pod "aea403b0-3184-48ef-8e1a-b1f91b56d703" (UID: "aea403b0-3184-48ef-8e1a-b1f91b56d703"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:53:57.038890 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:57.037953 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aea403b0-3184-48ef-8e1a-b1f91b56d703-istiod-ca-cert" pod "aea403b0-3184-48ef-8e1a-b1f91b56d703" (UID: "aea403b0-3184-48ef-8e1a-b1f91b56d703"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:53:57.038890 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:57.038434 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aea403b0-3184-48ef-8e1a-b1f91b56d703-istio-data" pod "aea403b0-3184-48ef-8e1a-b1f91b56d703" (UID: "aea403b0-3184-48ef-8e1a-b1f91b56d703"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:53:57.039077 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:57.038995 4562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aea403b0-3184-48ef-8e1a-b1f91b56d703-workload-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:53:57.039077 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:57.039012 4562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aea403b0-3184-48ef-8e1a-b1f91b56d703-workload-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:53:57.039077 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:57.039026 4562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aea403b0-3184-48ef-8e1a-b1f91b56d703-credential-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:53:57.040084 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:57.040055 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aea403b0-3184-48ef-8e1a-b1f91b56d703-istio-envoy" pod "aea403b0-3184-48ef-8e1a-b1f91b56d703" (UID: "aea403b0-3184-48ef-8e1a-b1f91b56d703"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:53:57.040439 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:57.040412 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aea403b0-3184-48ef-8e1a-b1f91b56d703-kube-api-access-sbvb2" pod "aea403b0-3184-48ef-8e1a-b1f91b56d703" (UID: "aea403b0-3184-48ef-8e1a-b1f91b56d703"). InnerVolumeSpecName "kube-api-access-sbvb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:53:57.040544 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:57.040456 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aea403b0-3184-48ef-8e1a-b1f91b56d703-istio-token" pod "aea403b0-3184-48ef-8e1a-b1f91b56d703" (UID: "aea403b0-3184-48ef-8e1a-b1f91b56d703"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:53:57.040967 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:57.040948 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aea403b0-3184-48ef-8e1a-b1f91b56d703-istio-podinfo" pod "aea403b0-3184-48ef-8e1a-b1f91b56d703" (UID: "aea403b0-3184-48ef-8e1a-b1f91b56d703"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:53:57.141038 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:57.140956 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbvb2\" (UniqueName: \"kubernetes.io/projected/aea403b0-3184-48ef-8e1a-b1f91b56d703-kube-api-access-sbvb2\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:53:57.141038 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:57.140989 4562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aea403b0-3184-48ef-8e1a-b1f91b56d703-istio-envoy\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:53:57.141038 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:57.140998 4562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aea403b0-3184-48ef-8e1a-b1f91b56d703-istio-token\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:53:57.141038 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:57.141006 4562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aea403b0-3184-48ef-8e1a-b1f91b56d703-istiod-ca-cert\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:53:57.141038 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:57.141014 4562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aea403b0-3184-48ef-8e1a-b1f91b56d703-istio-podinfo\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:53:57.141038 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:57.141023 4562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aea403b0-3184-48ef-8e1a-b1f91b56d703-istio-data\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:53:57.364614 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:57.364581 4562 generic.go:356] "Generic (PLEG): container finished" podID="aea403b0-3184-48ef-8e1a-b1f91b56d703" containerID="7a0381a04c9a072371781cd7fb53195ead4de0500e0be0e06ed9bb0ab2fe83ac" exitCode=0 Oct 30 09:53:57.364818 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:57.364666 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h" Oct 30 09:53:57.364818 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:57.364710 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h" event={"ID":"aea403b0-3184-48ef-8e1a-b1f91b56d703","Type":"ContainerDied","Data":"7a0381a04c9a072371781cd7fb53195ead4de0500e0be0e06ed9bb0ab2fe83ac"} Oct 30 09:53:57.364818 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:57.364762 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h" event={"ID":"aea403b0-3184-48ef-8e1a-b1f91b56d703","Type":"ContainerDied","Data":"fe53ffb9bc03c96b43aadc258ba56121f56ccd776bf9bcbdb3191c5ace67f13e"} Oct 30 09:53:57.364818 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:57.364789 4562 scope.go:122] "RemoveContainer" containerID="7a0381a04c9a072371781cd7fb53195ead4de0500e0be0e06ed9bb0ab2fe83ac" Oct 30 09:53:57.377017 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:57.376995 4562 scope.go:122] "RemoveContainer" containerID="2b2612ef43ed39a643510f2582516db4e90bb96a434e3624eaaf66443eb33d37" Oct 30 09:53:57.391119 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:57.389016 4562 scope.go:122] "RemoveContainer" containerID="7a0381a04c9a072371781cd7fb53195ead4de0500e0be0e06ed9bb0ab2fe83ac" Oct 30 09:53:57.391119 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:53:57.390955 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a0381a04c9a072371781cd7fb53195ead4de0500e0be0e06ed9bb0ab2fe83ac\": container with ID starting with 7a0381a04c9a072371781cd7fb53195ead4de0500e0be0e06ed9bb0ab2fe83ac not found: ID does not exist" containerID="7a0381a04c9a072371781cd7fb53195ead4de0500e0be0e06ed9bb0ab2fe83ac" Oct 30 09:53:57.391119 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:57.390993 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a0381a04c9a072371781cd7fb53195ead4de0500e0be0e06ed9bb0ab2fe83ac"} err="failed to get container status \"7a0381a04c9a072371781cd7fb53195ead4de0500e0be0e06ed9bb0ab2fe83ac\": rpc error: code = NotFound desc = could not find container \"7a0381a04c9a072371781cd7fb53195ead4de0500e0be0e06ed9bb0ab2fe83ac\": container with ID starting with 7a0381a04c9a072371781cd7fb53195ead4de0500e0be0e06ed9bb0ab2fe83ac not found: ID does not exist" Oct 30 09:53:57.391119 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:57.391012 4562 scope.go:122] "RemoveContainer" containerID="2b2612ef43ed39a643510f2582516db4e90bb96a434e3624eaaf66443eb33d37" Oct 30 09:53:57.391372 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:53:57.391327 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b2612ef43ed39a643510f2582516db4e90bb96a434e3624eaaf66443eb33d37\": container with ID starting with 2b2612ef43ed39a643510f2582516db4e90bb96a434e3624eaaf66443eb33d37 not found: ID does not exist" containerID="2b2612ef43ed39a643510f2582516db4e90bb96a434e3624eaaf66443eb33d37" Oct 30 09:53:57.391372 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:57.391360 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b2612ef43ed39a643510f2582516db4e90bb96a434e3624eaaf66443eb33d37"} err="failed to get container status \"2b2612ef43ed39a643510f2582516db4e90bb96a434e3624eaaf66443eb33d37\": rpc error: code = NotFound desc = could not find container \"2b2612ef43ed39a643510f2582516db4e90bb96a434e3624eaaf66443eb33d37\": container with ID starting with 2b2612ef43ed39a643510f2582516db4e90bb96a434e3624eaaf66443eb33d37 not found: ID does not exist" Oct 30 09:53:57.421629 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:57.421598 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h"] Oct 30 09:53:57.434397 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:57.434359 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-881f83-openshift-default-7f7b84cbd9-6r85h"] Oct 30 09:53:57.712482 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:57.712439 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-f76d4tcz6" podUID="ce4307bc-f5bb-40ae-bb86-a1458732170b" containerName="payload-processing" containerID="cri-o://fd276fd039054ed4ad1ed2817cc9b31b841878380e15e9b9e7a24f6b4012e4e2" gracePeriod=25 Oct 30 09:53:57.878269 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:57.878227 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="169915d0-2153-46a4-a1af-a9d1670af675" path="/var/lib/kubelet/pods/169915d0-2153-46a4-a1af-a9d1670af675/volumes" Oct 30 09:53:57.878937 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:57.878913 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fca872d-4848-4886-925d-e06a5bcc2526" path="/var/lib/kubelet/pods/2fca872d-4848-4886-925d-e06a5bcc2526/volumes" Oct 30 09:53:57.879469 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:57.879444 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59e94911-c9dc-4fbd-b813-e22c0bc51fed" path="/var/lib/kubelet/pods/59e94911-c9dc-4fbd-b813-e22c0bc51fed/volumes" Oct 30 09:53:57.880514 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:57.880493 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aea403b0-3184-48ef-8e1a-b1f91b56d703" path="/var/lib/kubelet/pods/aea403b0-3184-48ef-8e1a-b1f91b56d703/volumes" Oct 30 09:53:58.073522 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:58.073492 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-f76d4tcz6" Oct 30 09:53:58.161683 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:58.161637 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce4307bc-f5bb-40ae-bb86-a1458732170b-kube-api-access-w9lc7\" (UniqueName: \"kubernetes.io/projected/ce4307bc-f5bb-40ae-bb86-a1458732170b-kube-api-access-w9lc7\") pod \"ce4307bc-f5bb-40ae-bb86-a1458732170b\" (UID: \"ce4307bc-f5bb-40ae-bb86-a1458732170b\") " Oct 30 09:53:58.161859 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:58.161763 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce4307bc-f5bb-40ae-bb86-a1458732170b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ce4307bc-f5bb-40ae-bb86-a1458732170b-routing-overlay-volume\") pod \"ce4307bc-f5bb-40ae-bb86-a1458732170b\" (UID: \"ce4307bc-f5bb-40ae-bb86-a1458732170b\") " Oct 30 09:53:58.161859 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:58.161852 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce4307bc-f5bb-40ae-bb86-a1458732170b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ce4307bc-f5bb-40ae-bb86-a1458732170b-provider-credentials\") pod \"ce4307bc-f5bb-40ae-bb86-a1458732170b\" (UID: \"ce4307bc-f5bb-40ae-bb86-a1458732170b\") " Oct 30 09:53:58.162017 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:58.161946 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce4307bc-f5bb-40ae-bb86-a1458732170b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce4307bc-f5bb-40ae-bb86-a1458732170b-plugins-config-volume\") pod \"ce4307bc-f5bb-40ae-bb86-a1458732170b\" (UID: \"ce4307bc-f5bb-40ae-bb86-a1458732170b\") " Oct 30 09:53:58.162252 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:58.162221 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce4307bc-f5bb-40ae-bb86-a1458732170b-routing-overlay-volume" pod "ce4307bc-f5bb-40ae-bb86-a1458732170b" (UID: "ce4307bc-f5bb-40ae-bb86-a1458732170b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:53:58.162708 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:58.162679 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce4307bc-f5bb-40ae-bb86-a1458732170b-plugins-config-volume" pod "ce4307bc-f5bb-40ae-bb86-a1458732170b" (UID: "ce4307bc-f5bb-40ae-bb86-a1458732170b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:53:58.162824 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:58.162736 4562 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ce4307bc-f5bb-40ae-bb86-a1458732170b-routing-overlay-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:53:58.164210 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:58.164179 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce4307bc-f5bb-40ae-bb86-a1458732170b-provider-credentials" pod "ce4307bc-f5bb-40ae-bb86-a1458732170b" (UID: "ce4307bc-f5bb-40ae-bb86-a1458732170b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:53:58.164210 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:58.164205 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce4307bc-f5bb-40ae-bb86-a1458732170b-kube-api-access-w9lc7" pod "ce4307bc-f5bb-40ae-bb86-a1458732170b" (UID: "ce4307bc-f5bb-40ae-bb86-a1458732170b"). InnerVolumeSpecName "kube-api-access-w9lc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:53:58.264233 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:58.264194 4562 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ce4307bc-f5bb-40ae-bb86-a1458732170b-provider-credentials\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:53:58.264233 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:58.264227 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce4307bc-f5bb-40ae-bb86-a1458732170b-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:53:58.264233 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:58.264237 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9lc7\" (UniqueName: \"kubernetes.io/projected/ce4307bc-f5bb-40ae-bb86-a1458732170b-kube-api-access-w9lc7\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:53:58.379073 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:58.378980 4562 generic.go:356] "Generic (PLEG): container finished" podID="ce4307bc-f5bb-40ae-bb86-a1458732170b" containerID="fd276fd039054ed4ad1ed2817cc9b31b841878380e15e9b9e7a24f6b4012e4e2" exitCode=0 Oct 30 09:53:58.379073 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:58.379028 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-f76d4tcz6" Oct 30 09:53:58.379073 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:58.379064 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-f76d4tcz6" event={"ID":"ce4307bc-f5bb-40ae-bb86-a1458732170b","Type":"ContainerDied","Data":"fd276fd039054ed4ad1ed2817cc9b31b841878380e15e9b9e7a24f6b4012e4e2"} Oct 30 09:53:58.379343 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:58.379116 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-f76d4tcz6" event={"ID":"ce4307bc-f5bb-40ae-bb86-a1458732170b","Type":"ContainerDied","Data":"c7ab7493e145d7382804e3aeea09d3eb1fad3d74ce8a31d12f392f91ba921a96"} Oct 30 09:53:58.379343 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:58.379139 4562 scope.go:122] "RemoveContainer" containerID="fd276fd039054ed4ad1ed2817cc9b31b841878380e15e9b9e7a24f6b4012e4e2" Oct 30 09:53:58.390278 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:58.390261 4562 scope.go:122] "RemoveContainer" containerID="fd276fd039054ed4ad1ed2817cc9b31b841878380e15e9b9e7a24f6b4012e4e2" Oct 30 09:53:58.390552 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:53:58.390532 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd276fd039054ed4ad1ed2817cc9b31b841878380e15e9b9e7a24f6b4012e4e2\": container with ID starting with fd276fd039054ed4ad1ed2817cc9b31b841878380e15e9b9e7a24f6b4012e4e2 not found: ID does not exist" containerID="fd276fd039054ed4ad1ed2817cc9b31b841878380e15e9b9e7a24f6b4012e4e2" Oct 30 09:53:58.390595 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:58.390561 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd276fd039054ed4ad1ed2817cc9b31b841878380e15e9b9e7a24f6b4012e4e2"} err="failed to get container status \"fd276fd039054ed4ad1ed2817cc9b31b841878380e15e9b9e7a24f6b4012e4e2\": rpc error: code = NotFound desc = could not find container \"fd276fd039054ed4ad1ed2817cc9b31b841878380e15e9b9e7a24f6b4012e4e2\": container with ID starting with fd276fd039054ed4ad1ed2817cc9b31b841878380e15e9b9e7a24f6b4012e4e2 not found: ID does not exist" Oct 30 09:53:58.433568 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:58.433534 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-f76d4tcz6"] Oct 30 09:53:58.445743 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:58.445712 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-88cbc7/payload-processing-external-model-e2e-extmod-a-88cbc7-f76d4tcz6"] Oct 30 09:53:58.597470 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:53:58.597429 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:53:59.877559 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:53:59.877526 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce4307bc-f5bb-40ae-bb86-a1458732170b" path="/var/lib/kubelet/pods/ce4307bc-f5bb-40ae-bb86-a1458732170b/volumes" Oct 30 09:54:00.530944 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:00.530918 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6q7n_1f51db5a-5720-4dea-ad02-f336b35d43b1/ovn-acl-logging/0.log" Oct 30 09:54:00.542684 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:00.542662 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6q7n_1f51db5a-5720-4dea-ad02-f336b35d43b1/ovn-acl-logging/0.log" Oct 30 09:54:07.720327 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:07.720288 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d60c68-865c467cd-zgnz8"] Oct 30 09:54:07.753412 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:07.753369 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d60c68-768c8ccddc-lwqlw"] Oct 30 09:54:12.721109 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:12.721052 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-d60c68-865c467cd-zgnz8" podUID="7f4a9934-3725-4b3c-82e4-900bdcd57e18" containerName="payload-processing" containerID="cri-o://41265d488eca01c4c12a6c7584c2a4a2407e36ffeb1c488881db0dc2d7dcb43e" gracePeriod=25 Oct 30 09:54:12.754613 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:12.754522 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d60c68-768c8ccddc-lwqlw" podUID="c15af94f-dcf8-47ab-a39a-34de624afe00" containerName="payload-pre-processing" containerID="cri-o://6295fec76ca889f375a96f5a82295815b4376dd43fa7eff3948a57e9c69f0464" gracePeriod=25 Oct 30 09:54:15.074499 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:15.074450 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-d60c68-865c467cd-zgnz8" podUID="7f4a9934-3725-4b3c-82e4-900bdcd57e18" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:54:15.074945 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:15.074543 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d60c68-768c8ccddc-lwqlw" podUID="c15af94f-dcf8-47ab-a39a-34de624afe00" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:54:25.074764 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:25.074720 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-d60c68-865c467cd-zgnz8" podUID="7f4a9934-3725-4b3c-82e4-900bdcd57e18" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:54:25.075193 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:25.074728 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d60c68-768c8ccddc-lwqlw" podUID="c15af94f-dcf8-47ab-a39a-34de624afe00" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:54:28.599006 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:54:28.598969 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:54:32.223184 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:32.223149 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-82ccfd/test-model-82ccfd-kserve-77684cd675-n8rmr"] Oct 30 09:54:32.224003 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:32.223980 4562 cpu_manager.go:414] "containerMap: removing container" podUID="b1a4232c-bb1f-4fb4-921a-8ada7bc47d6c" containerName="payload-processing" Oct 30 09:54:32.224003 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:32.224003 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1a4232c-bb1f-4fb4-921a-8ada7bc47d6c" containerName="payload-processing" Oct 30 09:54:32.224189 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:32.224034 4562 cpu_manager.go:414] "containerMap: removing container" podUID="169915d0-2153-46a4-a1af-a9d1670af675" containerName="payload-processing" Oct 30 09:54:32.224189 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:32.224041 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="169915d0-2153-46a4-a1af-a9d1670af675" containerName="payload-processing" Oct 30 09:54:32.224189 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:32.224077 4562 cpu_manager.go:414] "containerMap: removing container" podUID="ce4307bc-f5bb-40ae-bb86-a1458732170b" containerName="payload-processing" Oct 30 09:54:32.224189 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:32.224096 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce4307bc-f5bb-40ae-bb86-a1458732170b" containerName="payload-processing" Oct 30 09:54:32.224189 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:32.224107 4562 cpu_manager.go:414] "containerMap: removing container" podUID="aea403b0-3184-48ef-8e1a-b1f91b56d703" containerName="istio-proxy" Oct 30 09:54:32.224189 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:32.224114 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aea403b0-3184-48ef-8e1a-b1f91b56d703" containerName="istio-proxy" Oct 30 09:54:32.224189 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:32.224128 4562 cpu_manager.go:414] "containerMap: removing container" podUID="06051f1f-c3c6-4c87-8cd4-1cb7b6c4f170" containerName="payload-processing" Oct 30 09:54:32.224189 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:32.224135 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06051f1f-c3c6-4c87-8cd4-1cb7b6c4f170" containerName="payload-processing" Oct 30 09:54:32.224189 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:32.224167 4562 cpu_manager.go:414] "containerMap: removing container" podUID="59e94911-c9dc-4fbd-b813-e22c0bc51fed" containerName="payload-pre-processing" Oct 30 09:54:32.224189 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:32.224174 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59e94911-c9dc-4fbd-b813-e22c0bc51fed" containerName="payload-pre-processing" Oct 30 09:54:32.224189 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:32.224188 4562 cpu_manager.go:414] "containerMap: removing container" podUID="2fca872d-4848-4886-925d-e06a5bcc2526" containerName="istio-proxy" Oct 30 09:54:32.224189 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:32.224194 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fca872d-4848-4886-925d-e06a5bcc2526" containerName="istio-proxy" Oct 30 09:54:32.224956 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:32.224395 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce4307bc-f5bb-40ae-bb86-a1458732170b" containerName="payload-processing" Oct 30 09:54:32.224956 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:32.224411 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="aea403b0-3184-48ef-8e1a-b1f91b56d703" containerName="istio-proxy" Oct 30 09:54:32.224956 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:32.224419 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="06051f1f-c3c6-4c87-8cd4-1cb7b6c4f170" containerName="payload-processing" Oct 30 09:54:32.224956 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:32.224429 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="59e94911-c9dc-4fbd-b813-e22c0bc51fed" containerName="payload-pre-processing" Oct 30 09:54:32.224956 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:32.224436 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fca872d-4848-4886-925d-e06a5bcc2526" containerName="istio-proxy" Oct 30 09:54:32.224956 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:32.224444 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="aea403b0-3184-48ef-8e1a-b1f91b56d703" containerName="istio-proxy" Oct 30 09:54:32.224956 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:32.224453 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1a4232c-bb1f-4fb4-921a-8ada7bc47d6c" containerName="payload-processing" Oct 30 09:54:32.224956 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:32.224462 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="169915d0-2153-46a4-a1af-a9d1670af675" containerName="payload-processing" Oct 30 09:54:32.228607 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:32.228586 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-82ccfd/test-model-82ccfd-kserve-77684cd675-n8rmr" Oct 30 09:54:32.249475 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:32.246264 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-82ccfd\"/\"kube-root-ca.crt\"" Oct 30 09:54:32.249475 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:32.247172 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-82ccfd/test-model-82ccfd-kserve-77684cd675-n8rmr"] Oct 30 09:54:32.253860 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:32.253812 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-82ccfd\"/\"openshift-service-ca.crt\"" Oct 30 09:54:32.259443 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:32.259345 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-82ccfd\"/\"test-model-82ccfd-kserve-self-signed-certs\"" Oct 30 09:54:32.259616 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:32.259509 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-82ccfd\"/\"default-dockercfg-22trq\"" Oct 30 09:54:32.356735 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:32.356697 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj9jp\" (UniqueName: \"kubernetes.io/projected/47f3324c-84c0-4bee-bcfd-88bff68a5e10-kube-api-access-sj9jp\") pod \"test-model-82ccfd-kserve-77684cd675-n8rmr\" (UID: \"47f3324c-84c0-4bee-bcfd-88bff68a5e10\") " pod="ai-tenant-e2e-inf-a-82ccfd/test-model-82ccfd-kserve-77684cd675-n8rmr" Oct 30 09:54:32.356971 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:32.356841 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47f3324c-84c0-4bee-bcfd-88bff68a5e10-tls-certs\") pod \"test-model-82ccfd-kserve-77684cd675-n8rmr\" (UID: \"47f3324c-84c0-4bee-bcfd-88bff68a5e10\") " pod="ai-tenant-e2e-inf-a-82ccfd/test-model-82ccfd-kserve-77684cd675-n8rmr" Oct 30 09:54:32.356971 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:32.356956 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47f3324c-84c0-4bee-bcfd-88bff68a5e10-model-cache\") pod \"test-model-82ccfd-kserve-77684cd675-n8rmr\" (UID: \"47f3324c-84c0-4bee-bcfd-88bff68a5e10\") " pod="ai-tenant-e2e-inf-a-82ccfd/test-model-82ccfd-kserve-77684cd675-n8rmr" Oct 30 09:54:32.357105 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:32.357091 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47f3324c-84c0-4bee-bcfd-88bff68a5e10-home\") pod \"test-model-82ccfd-kserve-77684cd675-n8rmr\" (UID: \"47f3324c-84c0-4bee-bcfd-88bff68a5e10\") " pod="ai-tenant-e2e-inf-a-82ccfd/test-model-82ccfd-kserve-77684cd675-n8rmr" Oct 30 09:54:32.357228 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:32.357202 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47f3324c-84c0-4bee-bcfd-88bff68a5e10-dshm\") pod \"test-model-82ccfd-kserve-77684cd675-n8rmr\" (UID: \"47f3324c-84c0-4bee-bcfd-88bff68a5e10\") " pod="ai-tenant-e2e-inf-a-82ccfd/test-model-82ccfd-kserve-77684cd675-n8rmr" Oct 30 09:54:32.357362 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:32.357300 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47f3324c-84c0-4bee-bcfd-88bff68a5e10-tmp-dir\") pod \"test-model-82ccfd-kserve-77684cd675-n8rmr\" (UID: \"47f3324c-84c0-4bee-bcfd-88bff68a5e10\") " pod="ai-tenant-e2e-inf-a-82ccfd/test-model-82ccfd-kserve-77684cd675-n8rmr" Oct 30 09:54:32.464420 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:32.461507 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj9jp\" (UniqueName: \"kubernetes.io/projected/47f3324c-84c0-4bee-bcfd-88bff68a5e10-kube-api-access-sj9jp\") pod \"test-model-82ccfd-kserve-77684cd675-n8rmr\" (UID: \"47f3324c-84c0-4bee-bcfd-88bff68a5e10\") " pod="ai-tenant-e2e-inf-a-82ccfd/test-model-82ccfd-kserve-77684cd675-n8rmr" Oct 30 09:54:32.464420 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:32.461643 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47f3324c-84c0-4bee-bcfd-88bff68a5e10-tls-certs\") pod \"test-model-82ccfd-kserve-77684cd675-n8rmr\" (UID: \"47f3324c-84c0-4bee-bcfd-88bff68a5e10\") " pod="ai-tenant-e2e-inf-a-82ccfd/test-model-82ccfd-kserve-77684cd675-n8rmr" Oct 30 09:54:32.464420 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:32.461709 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47f3324c-84c0-4bee-bcfd-88bff68a5e10-model-cache\") pod \"test-model-82ccfd-kserve-77684cd675-n8rmr\" (UID: \"47f3324c-84c0-4bee-bcfd-88bff68a5e10\") " pod="ai-tenant-e2e-inf-a-82ccfd/test-model-82ccfd-kserve-77684cd675-n8rmr" Oct 30 09:54:32.464420 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:32.461814 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47f3324c-84c0-4bee-bcfd-88bff68a5e10-home\") pod \"test-model-82ccfd-kserve-77684cd675-n8rmr\" (UID: \"47f3324c-84c0-4bee-bcfd-88bff68a5e10\") " pod="ai-tenant-e2e-inf-a-82ccfd/test-model-82ccfd-kserve-77684cd675-n8rmr" Oct 30 09:54:32.464420 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:32.461887 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47f3324c-84c0-4bee-bcfd-88bff68a5e10-dshm\") pod \"test-model-82ccfd-kserve-77684cd675-n8rmr\" (UID: \"47f3324c-84c0-4bee-bcfd-88bff68a5e10\") " pod="ai-tenant-e2e-inf-a-82ccfd/test-model-82ccfd-kserve-77684cd675-n8rmr" Oct 30 09:54:32.464420 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:32.461978 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47f3324c-84c0-4bee-bcfd-88bff68a5e10-tmp-dir\") pod \"test-model-82ccfd-kserve-77684cd675-n8rmr\" (UID: \"47f3324c-84c0-4bee-bcfd-88bff68a5e10\") " pod="ai-tenant-e2e-inf-a-82ccfd/test-model-82ccfd-kserve-77684cd675-n8rmr" Oct 30 09:54:32.464420 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:32.462490 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47f3324c-84c0-4bee-bcfd-88bff68a5e10-tmp-dir\") pod \"test-model-82ccfd-kserve-77684cd675-n8rmr\" (UID: \"47f3324c-84c0-4bee-bcfd-88bff68a5e10\") " pod="ai-tenant-e2e-inf-a-82ccfd/test-model-82ccfd-kserve-77684cd675-n8rmr" Oct 30 09:54:32.464420 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:32.463812 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47f3324c-84c0-4bee-bcfd-88bff68a5e10-model-cache\") pod \"test-model-82ccfd-kserve-77684cd675-n8rmr\" (UID: \"47f3324c-84c0-4bee-bcfd-88bff68a5e10\") " pod="ai-tenant-e2e-inf-a-82ccfd/test-model-82ccfd-kserve-77684cd675-n8rmr" Oct 30 09:54:32.464989 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:32.464552 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47f3324c-84c0-4bee-bcfd-88bff68a5e10-home\") pod \"test-model-82ccfd-kserve-77684cd675-n8rmr\" (UID: \"47f3324c-84c0-4bee-bcfd-88bff68a5e10\") " pod="ai-tenant-e2e-inf-a-82ccfd/test-model-82ccfd-kserve-77684cd675-n8rmr" Oct 30 09:54:32.496474 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:32.494706 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47f3324c-84c0-4bee-bcfd-88bff68a5e10-dshm\") pod \"test-model-82ccfd-kserve-77684cd675-n8rmr\" (UID: \"47f3324c-84c0-4bee-bcfd-88bff68a5e10\") " pod="ai-tenant-e2e-inf-a-82ccfd/test-model-82ccfd-kserve-77684cd675-n8rmr" Oct 30 09:54:32.517410 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:32.517274 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47f3324c-84c0-4bee-bcfd-88bff68a5e10-tls-certs\") pod \"test-model-82ccfd-kserve-77684cd675-n8rmr\" (UID: \"47f3324c-84c0-4bee-bcfd-88bff68a5e10\") " pod="ai-tenant-e2e-inf-a-82ccfd/test-model-82ccfd-kserve-77684cd675-n8rmr" Oct 30 09:54:32.544770 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:32.544049 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj9jp\" (UniqueName: \"kubernetes.io/projected/47f3324c-84c0-4bee-bcfd-88bff68a5e10-kube-api-access-sj9jp\") pod \"test-model-82ccfd-kserve-77684cd675-n8rmr\" (UID: \"47f3324c-84c0-4bee-bcfd-88bff68a5e10\") " pod="ai-tenant-e2e-inf-a-82ccfd/test-model-82ccfd-kserve-77684cd675-n8rmr" Oct 30 09:54:32.549450 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:32.548967 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-82ccfd/test-model-82ccfd-kserve-77684cd675-n8rmr" Oct 30 09:54:32.882794 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:32.882753 4562 generic.go:356] "Generic (PLEG): container finished" podID="7f4a9934-3725-4b3c-82e4-900bdcd57e18" containerID="41265d488eca01c4c12a6c7584c2a4a2407e36ffeb1c488881db0dc2d7dcb43e" exitCode=0 Oct 30 09:54:32.882952 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:32.882804 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d60c68-865c467cd-zgnz8" event={"ID":"7f4a9934-3725-4b3c-82e4-900bdcd57e18","Type":"ContainerDied","Data":"41265d488eca01c4c12a6c7584c2a4a2407e36ffeb1c488881db0dc2d7dcb43e"} Oct 30 09:54:32.884534 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:32.884510 4562 generic.go:356] "Generic (PLEG): container finished" podID="c15af94f-dcf8-47ab-a39a-34de624afe00" containerID="6295fec76ca889f375a96f5a82295815b4376dd43fa7eff3948a57e9c69f0464" exitCode=0 Oct 30 09:54:32.884686 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:32.884556 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d60c68-768c8ccddc-lwqlw" event={"ID":"c15af94f-dcf8-47ab-a39a-34de624afe00","Type":"ContainerDied","Data":"6295fec76ca889f375a96f5a82295815b4376dd43fa7eff3948a57e9c69f0464"} Oct 30 09:54:32.897090 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:54:32.897045 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod47f3324c_84c0_4bee_bcfd_88bff68a5e10.slice/crio-4a8a51471582553534f6f296614e07ee22be708bae1fc6757855d3ada6fdf5d1 WatchSource:0}: Error finding container 4a8a51471582553534f6f296614e07ee22be708bae1fc6757855d3ada6fdf5d1: Status 404 returned error can't find the container with id 4a8a51471582553534f6f296614e07ee22be708bae1fc6757855d3ada6fdf5d1 Oct 30 09:54:32.900810 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:32.900780 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-82ccfd/test-model-82ccfd-kserve-77684cd675-n8rmr"] Oct 30 09:54:33.043314 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:33.041526 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d60c68-865c467cd-zgnz8" Oct 30 09:54:33.108086 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:33.108058 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d60c68-768c8ccddc-lwqlw" Oct 30 09:54:33.193064 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:33.193027 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f4a9934-3725-4b3c-82e4-900bdcd57e18-kube-api-access-6bsxn\" (UniqueName: \"kubernetes.io/projected/7f4a9934-3725-4b3c-82e4-900bdcd57e18-kube-api-access-6bsxn\") pod \"7f4a9934-3725-4b3c-82e4-900bdcd57e18\" (UID: \"7f4a9934-3725-4b3c-82e4-900bdcd57e18\") " Oct 30 09:54:33.193284 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:33.193092 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f4a9934-3725-4b3c-82e4-900bdcd57e18-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f4a9934-3725-4b3c-82e4-900bdcd57e18-plugins-config-volume\") pod \"7f4a9934-3725-4b3c-82e4-900bdcd57e18\" (UID: \"7f4a9934-3725-4b3c-82e4-900bdcd57e18\") " Oct 30 09:54:33.193284 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:33.193161 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c15af94f-dcf8-47ab-a39a-34de624afe00-kube-api-access-xtq8l\" (UniqueName: \"kubernetes.io/projected/c15af94f-dcf8-47ab-a39a-34de624afe00-kube-api-access-xtq8l\") pod \"c15af94f-dcf8-47ab-a39a-34de624afe00\" (UID: \"c15af94f-dcf8-47ab-a39a-34de624afe00\") " Oct 30 09:54:33.193460 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:33.193323 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c15af94f-dcf8-47ab-a39a-34de624afe00-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c15af94f-dcf8-47ab-a39a-34de624afe00-plugins-config-volume\") pod \"c15af94f-dcf8-47ab-a39a-34de624afe00\" (UID: \"c15af94f-dcf8-47ab-a39a-34de624afe00\") " Oct 30 09:54:33.193864 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:33.193800 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f4a9934-3725-4b3c-82e4-900bdcd57e18-plugins-config-volume" pod "7f4a9934-3725-4b3c-82e4-900bdcd57e18" (UID: "7f4a9934-3725-4b3c-82e4-900bdcd57e18"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:54:33.193864 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:33.193834 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c15af94f-dcf8-47ab-a39a-34de624afe00-plugins-config-volume" pod "c15af94f-dcf8-47ab-a39a-34de624afe00" (UID: "c15af94f-dcf8-47ab-a39a-34de624afe00"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:54:33.194131 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:33.194109 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f4a9934-3725-4b3c-82e4-900bdcd57e18-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:54:33.194213 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:33.194139 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c15af94f-dcf8-47ab-a39a-34de624afe00-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:54:33.195551 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:33.195521 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f4a9934-3725-4b3c-82e4-900bdcd57e18-kube-api-access-6bsxn" pod "7f4a9934-3725-4b3c-82e4-900bdcd57e18" (UID: "7f4a9934-3725-4b3c-82e4-900bdcd57e18"). InnerVolumeSpecName "kube-api-access-6bsxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:54:33.196102 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:33.196079 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c15af94f-dcf8-47ab-a39a-34de624afe00-kube-api-access-xtq8l" pod "c15af94f-dcf8-47ab-a39a-34de624afe00" (UID: "c15af94f-dcf8-47ab-a39a-34de624afe00"). InnerVolumeSpecName "kube-api-access-xtq8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:54:33.295630 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:33.295545 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bsxn\" (UniqueName: \"kubernetes.io/projected/7f4a9934-3725-4b3c-82e4-900bdcd57e18-kube-api-access-6bsxn\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:54:33.295630 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:33.295574 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtq8l\" (UniqueName: \"kubernetes.io/projected/c15af94f-dcf8-47ab-a39a-34de624afe00-kube-api-access-xtq8l\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:54:33.893744 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:33.893718 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d60c68-865c467cd-zgnz8" Oct 30 09:54:33.893894 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:33.893717 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d60c68-865c467cd-zgnz8" event={"ID":"7f4a9934-3725-4b3c-82e4-900bdcd57e18","Type":"ContainerDied","Data":"ccd92cc8595dd180a67bb62cc41ec345eaa754864dcdc2f9e83cb407d2c63749"} Oct 30 09:54:33.893894 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:33.893848 4562 scope.go:122] "RemoveContainer" containerID="41265d488eca01c4c12a6c7584c2a4a2407e36ffeb1c488881db0dc2d7dcb43e" Oct 30 09:54:33.901011 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:33.900976 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-82ccfd/test-model-82ccfd-kserve-77684cd675-n8rmr" event={"ID":"47f3324c-84c0-4bee-bcfd-88bff68a5e10","Type":"ContainerStarted","Data":"e83148d5c124e7e33be60c708dcb74d5d35a6604e22b23261827c2e9b03af8a7"} Oct 30 09:54:33.901173 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:33.901025 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-82ccfd/test-model-82ccfd-kserve-77684cd675-n8rmr" event={"ID":"47f3324c-84c0-4bee-bcfd-88bff68a5e10","Type":"ContainerStarted","Data":"4a8a51471582553534f6f296614e07ee22be708bae1fc6757855d3ada6fdf5d1"} Oct 30 09:54:33.902674 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:33.902649 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d60c68-768c8ccddc-lwqlw" Oct 30 09:54:33.902674 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:33.902664 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d60c68-768c8ccddc-lwqlw" event={"ID":"c15af94f-dcf8-47ab-a39a-34de624afe00","Type":"ContainerDied","Data":"1d256fff24b4039178ce1cc9ce62ebe814047ee3e424950db6800a8a12220dfc"} Oct 30 09:54:33.907532 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:33.907515 4562 scope.go:122] "RemoveContainer" containerID="6295fec76ca889f375a96f5a82295815b4376dd43fa7eff3948a57e9c69f0464" Oct 30 09:54:33.952238 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:33.952169 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-82ccfd/test-model-82ccfd-kserve-77684cd675-n8rmr" podStartSLOduration=1.952154661 podStartE2EDuration="1.952154661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:54:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:54:33.947710898 +0000 UTC m=+3634.209271888" watchObservedRunningTime="2026-10-30 09:54:33.952154661 +0000 UTC m=+3634.213715649" Oct 30 09:54:33.987230 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:33.987183 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d60c68-865c467cd-zgnz8"] Oct 30 09:54:33.991333 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:33.991299 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d60c68-865c467cd-zgnz8"] Oct 30 09:54:34.024723 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:34.024690 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d60c68-768c8ccddc-lwqlw"] Oct 30 09:54:34.028806 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:34.028777 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d60c68-768c8ccddc-lwqlw"] Oct 30 09:54:34.549948 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:34.549912 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-82ccfd/test-model-82ccfd-kserve-77684cd675-n8rmr" Oct 30 09:54:35.876620 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:35.876585 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f4a9934-3725-4b3c-82e4-900bdcd57e18" path="/var/lib/kubelet/pods/7f4a9934-3725-4b3c-82e4-900bdcd57e18/volumes" Oct 30 09:54:35.877054 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:35.877040 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c15af94f-dcf8-47ab-a39a-34de624afe00" path="/var/lib/kubelet/pods/c15af94f-dcf8-47ab-a39a-34de624afe00/volumes" Oct 30 09:54:42.549682 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:42.549645 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-82ccfd/test-model-82ccfd-kserve-77684cd675-n8rmr" Oct 30 09:54:42.562421 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:42.562374 4562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-82ccfd/test-model-82ccfd-kserve-77684cd675-n8rmr" Oct 30 09:54:43.035431 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:43.032914 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-82ccfd/test-model-82ccfd-kserve-77684cd675-n8rmr" Oct 30 09:54:57.606603 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:57.606570 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-fc7ee7-6b565b88bd-c4tht"] Oct 30 09:54:57.636777 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:54:57.636741 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fc7ee7-5dd775bb8c-7fcnp"] Oct 30 09:54:58.598541 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:54:58.598504 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:55:01.657414 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:01.657335 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-3d9b36-5cf56c76bd-c976p"] Oct 30 09:55:01.692301 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:01.692254 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-3d9b36-5bcbf74867-k9hvd"] Oct 30 09:55:02.607738 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:02.607696 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fc7ee7-6b565b88bd-c4tht" podUID="e574e901-6a15-4cb8-9a91-0a96e5347b07" containerName="payload-processing" containerID="cri-o://46c6d2c076349127a537c7ed7c9953d4b8cc85d63ef2bfc05ce36161a72bf1ba" gracePeriod=25 Oct 30 09:55:02.637366 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:02.637328 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fc7ee7-5dd775bb8c-7fcnp" podUID="4a4b5a98-b1b2-41ff-9130-1f2367ccff29" containerName="payload-pre-processing" containerID="cri-o://7d49268029e774a7174387f49e4bb2fc72eaf6b3d44cc2182aaaa68d5c5c5a0c" gracePeriod=25 Oct 30 09:55:06.658571 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:06.658521 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-3d9b36-5cf56c76bd-c976p" podUID="d696e344-8d35-4896-84dc-9a4dc225f291" containerName="payload-processing" containerID="cri-o://38db6254771c4c209929a8a6e6928f07fddcbedfc000d88ec0a81e643716de4d" gracePeriod=25 Oct 30 09:55:06.692772 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:06.692738 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3d9b36-5bcbf74867-k9hvd" podUID="c0dbb750-1c20-41a5-b33f-5784a1f683d9" containerName="payload-pre-processing" containerID="cri-o://ba2b86702a749f28cc495af83ae66f51e321838468d53bb897e6adf50741d459" gracePeriod=25 Oct 30 09:55:07.284185 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:07.284148 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d66d22-7bccc4d786-z6qjp"] Oct 30 09:55:11.287465 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:11.287428 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fc7ee7-5dd775bb8c-7fcnp" podUID="4a4b5a98-b1b2-41ff-9130-1f2367ccff29" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:55:11.287872 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:11.287416 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fc7ee7-6b565b88bd-c4tht" podUID="e574e901-6a15-4cb8-9a91-0a96e5347b07" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:55:12.284733 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:12.284687 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-d66d22-7bccc4d786-z6qjp" podUID="c8073ce8-37bf-4ad9-9311-d17bdc00fbb0" containerName="payload-processing" containerID="cri-o://76fecc19b8e65f4c5b62f01ae82c02e62b583d33352272559248ee08fa36b3fd" gracePeriod=25 Oct 30 09:55:13.046870 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:13.046833 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-3d9b36-5cf56c76bd-c976p" podUID="d696e344-8d35-4896-84dc-9a4dc225f291" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:55:14.053435 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:14.053396 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3d9b36-5bcbf74867-k9hvd" podUID="c0dbb750-1c20-41a5-b33f-5784a1f683d9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:55:18.725289 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:18.725248 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-d66d22-7bccc4d786-z6qjp" podUID="c8073ce8-37bf-4ad9-9311-d17bdc00fbb0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:55:21.286853 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:21.286809 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fc7ee7-5dd775bb8c-7fcnp" podUID="4a4b5a98-b1b2-41ff-9130-1f2367ccff29" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:55:21.287276 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:21.286820 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fc7ee7-6b565b88bd-c4tht" podUID="e574e901-6a15-4cb8-9a91-0a96e5347b07" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:55:22.952493 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:22.950425 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fc7ee7-6b565b88bd-c4tht" Oct 30 09:55:22.993057 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:22.993022 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fc7ee7-5dd775bb8c-7fcnp" Oct 30 09:55:23.046990 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:23.046948 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-3d9b36-5cf56c76bd-c976p" podUID="d696e344-8d35-4896-84dc-9a4dc225f291" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:55:23.052494 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:23.052465 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a4b5a98-b1b2-41ff-9130-1f2367ccff29-kube-api-access-n6qvb\" (UniqueName: \"kubernetes.io/projected/4a4b5a98-b1b2-41ff-9130-1f2367ccff29-kube-api-access-n6qvb\") pod \"4a4b5a98-b1b2-41ff-9130-1f2367ccff29\" (UID: \"4a4b5a98-b1b2-41ff-9130-1f2367ccff29\") " Oct 30 09:55:23.052863 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:23.052839 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e574e901-6a15-4cb8-9a91-0a96e5347b07-kube-api-access-z9d75\" (UniqueName: \"kubernetes.io/projected/e574e901-6a15-4cb8-9a91-0a96e5347b07-kube-api-access-z9d75\") pod \"e574e901-6a15-4cb8-9a91-0a96e5347b07\" (UID: \"e574e901-6a15-4cb8-9a91-0a96e5347b07\") " Oct 30 09:55:23.053114 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:23.053096 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e574e901-6a15-4cb8-9a91-0a96e5347b07-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e574e901-6a15-4cb8-9a91-0a96e5347b07-plugins-config-volume\") pod \"e574e901-6a15-4cb8-9a91-0a96e5347b07\" (UID: \"e574e901-6a15-4cb8-9a91-0a96e5347b07\") " Oct 30 09:55:23.053716 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:23.053366 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a4b5a98-b1b2-41ff-9130-1f2367ccff29-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a4b5a98-b1b2-41ff-9130-1f2367ccff29-plugins-config-volume\") pod \"4a4b5a98-b1b2-41ff-9130-1f2367ccff29\" (UID: \"4a4b5a98-b1b2-41ff-9130-1f2367ccff29\") " Oct 30 09:55:23.053838 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:23.053515 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e574e901-6a15-4cb8-9a91-0a96e5347b07-plugins-config-volume" pod "e574e901-6a15-4cb8-9a91-0a96e5347b07" (UID: "e574e901-6a15-4cb8-9a91-0a96e5347b07"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:55:23.053838 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:23.053670 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a4b5a98-b1b2-41ff-9130-1f2367ccff29-plugins-config-volume" pod "4a4b5a98-b1b2-41ff-9130-1f2367ccff29" (UID: "4a4b5a98-b1b2-41ff-9130-1f2367ccff29"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:55:23.054426 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:23.054366 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e574e901-6a15-4cb8-9a91-0a96e5347b07-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:55:23.054557 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:23.054524 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a4b5a98-b1b2-41ff-9130-1f2367ccff29-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:55:23.057239 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:23.057210 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e574e901-6a15-4cb8-9a91-0a96e5347b07-kube-api-access-z9d75" pod "e574e901-6a15-4cb8-9a91-0a96e5347b07" (UID: "e574e901-6a15-4cb8-9a91-0a96e5347b07"). InnerVolumeSpecName "kube-api-access-z9d75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:55:23.059112 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:23.059084 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a4b5a98-b1b2-41ff-9130-1f2367ccff29-kube-api-access-n6qvb" pod "4a4b5a98-b1b2-41ff-9130-1f2367ccff29" (UID: "4a4b5a98-b1b2-41ff-9130-1f2367ccff29"). InnerVolumeSpecName "kube-api-access-n6qvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:55:23.155855 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:23.155771 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6qvb\" (UniqueName: \"kubernetes.io/projected/4a4b5a98-b1b2-41ff-9130-1f2367ccff29-kube-api-access-n6qvb\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:55:23.155855 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:23.155800 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9d75\" (UniqueName: \"kubernetes.io/projected/e574e901-6a15-4cb8-9a91-0a96e5347b07-kube-api-access-z9d75\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:55:23.573876 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:23.573841 4562 generic.go:356] "Generic (PLEG): container finished" podID="4a4b5a98-b1b2-41ff-9130-1f2367ccff29" containerID="7d49268029e774a7174387f49e4bb2fc72eaf6b3d44cc2182aaaa68d5c5c5a0c" exitCode=0 Oct 30 09:55:23.573876 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:23.573883 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fc7ee7-5dd775bb8c-7fcnp" Oct 30 09:55:23.574185 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:23.573922 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fc7ee7-5dd775bb8c-7fcnp" event={"ID":"4a4b5a98-b1b2-41ff-9130-1f2367ccff29","Type":"ContainerDied","Data":"7d49268029e774a7174387f49e4bb2fc72eaf6b3d44cc2182aaaa68d5c5c5a0c"} Oct 30 09:55:23.574185 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:23.573985 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fc7ee7-5dd775bb8c-7fcnp" event={"ID":"4a4b5a98-b1b2-41ff-9130-1f2367ccff29","Type":"ContainerDied","Data":"0da2b03de5e455f63613327f13467ef9f77db26a63837b557cc1a2b6d3a3ff22"} Oct 30 09:55:23.574185 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:23.574015 4562 scope.go:122] "RemoveContainer" containerID="7d49268029e774a7174387f49e4bb2fc72eaf6b3d44cc2182aaaa68d5c5c5a0c" Oct 30 09:55:23.575613 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:23.575589 4562 generic.go:356] "Generic (PLEG): container finished" podID="e574e901-6a15-4cb8-9a91-0a96e5347b07" containerID="46c6d2c076349127a537c7ed7c9953d4b8cc85d63ef2bfc05ce36161a72bf1ba" exitCode=0 Oct 30 09:55:23.575741 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:23.575641 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fc7ee7-6b565b88bd-c4tht" Oct 30 09:55:23.575741 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:23.575700 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fc7ee7-6b565b88bd-c4tht" event={"ID":"e574e901-6a15-4cb8-9a91-0a96e5347b07","Type":"ContainerDied","Data":"46c6d2c076349127a537c7ed7c9953d4b8cc85d63ef2bfc05ce36161a72bf1ba"} Oct 30 09:55:23.575847 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:23.575753 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fc7ee7-6b565b88bd-c4tht" event={"ID":"e574e901-6a15-4cb8-9a91-0a96e5347b07","Type":"ContainerDied","Data":"651dd78e7ab41f9cb4609d89c7d50b067146e05fbac2340199639319f3f893d0"} Oct 30 09:55:23.585461 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:23.585443 4562 scope.go:122] "RemoveContainer" containerID="7d49268029e774a7174387f49e4bb2fc72eaf6b3d44cc2182aaaa68d5c5c5a0c" Oct 30 09:55:23.585728 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:55:23.585711 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d49268029e774a7174387f49e4bb2fc72eaf6b3d44cc2182aaaa68d5c5c5a0c\": container with ID starting with 7d49268029e774a7174387f49e4bb2fc72eaf6b3d44cc2182aaaa68d5c5c5a0c not found: ID does not exist" containerID="7d49268029e774a7174387f49e4bb2fc72eaf6b3d44cc2182aaaa68d5c5c5a0c" Oct 30 09:55:23.585787 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:23.585735 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d49268029e774a7174387f49e4bb2fc72eaf6b3d44cc2182aaaa68d5c5c5a0c"} err="failed to get container status \"7d49268029e774a7174387f49e4bb2fc72eaf6b3d44cc2182aaaa68d5c5c5a0c\": rpc error: code = NotFound desc = could not find container \"7d49268029e774a7174387f49e4bb2fc72eaf6b3d44cc2182aaaa68d5c5c5a0c\": container with ID starting with 7d49268029e774a7174387f49e4bb2fc72eaf6b3d44cc2182aaaa68d5c5c5a0c not found: ID does not exist" Oct 30 09:55:23.585787 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:23.585750 4562 scope.go:122] "RemoveContainer" containerID="46c6d2c076349127a537c7ed7c9953d4b8cc85d63ef2bfc05ce36161a72bf1ba" Oct 30 09:55:23.595700 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:23.595678 4562 scope.go:122] "RemoveContainer" containerID="46c6d2c076349127a537c7ed7c9953d4b8cc85d63ef2bfc05ce36161a72bf1ba" Oct 30 09:55:23.595942 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:55:23.595926 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46c6d2c076349127a537c7ed7c9953d4b8cc85d63ef2bfc05ce36161a72bf1ba\": container with ID starting with 46c6d2c076349127a537c7ed7c9953d4b8cc85d63ef2bfc05ce36161a72bf1ba not found: ID does not exist" containerID="46c6d2c076349127a537c7ed7c9953d4b8cc85d63ef2bfc05ce36161a72bf1ba" Oct 30 09:55:23.596011 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:23.595948 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46c6d2c076349127a537c7ed7c9953d4b8cc85d63ef2bfc05ce36161a72bf1ba"} err="failed to get container status \"46c6d2c076349127a537c7ed7c9953d4b8cc85d63ef2bfc05ce36161a72bf1ba\": rpc error: code = NotFound desc = could not find container \"46c6d2c076349127a537c7ed7c9953d4b8cc85d63ef2bfc05ce36161a72bf1ba\": container with ID starting with 46c6d2c076349127a537c7ed7c9953d4b8cc85d63ef2bfc05ce36161a72bf1ba not found: ID does not exist" Oct 30 09:55:23.628220 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:23.628197 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fc7ee7-5dd775bb8c-7fcnp"] Oct 30 09:55:23.641546 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:23.641520 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fc7ee7-5dd775bb8c-7fcnp"] Oct 30 09:55:23.675724 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:23.675695 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-fc7ee7-6b565b88bd-c4tht"] Oct 30 09:55:23.680915 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:23.680889 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-fc7ee7-6b565b88bd-c4tht"] Oct 30 09:55:23.877284 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:23.877206 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a4b5a98-b1b2-41ff-9130-1f2367ccff29" path="/var/lib/kubelet/pods/4a4b5a98-b1b2-41ff-9130-1f2367ccff29/volumes" Oct 30 09:55:23.877709 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:23.877682 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e574e901-6a15-4cb8-9a91-0a96e5347b07" path="/var/lib/kubelet/pods/e574e901-6a15-4cb8-9a91-0a96e5347b07/volumes" Oct 30 09:55:24.053404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:24.053356 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3d9b36-5bcbf74867-k9hvd" podUID="c0dbb750-1c20-41a5-b33f-5784a1f683d9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:55:27.008807 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:27.008779 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-3d9b36-5cf56c76bd-c976p" Oct 30 09:55:27.063110 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:27.063081 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3d9b36-5bcbf74867-k9hvd" Oct 30 09:55:27.168410 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:27.166534 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0dbb750-1c20-41a5-b33f-5784a1f683d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0dbb750-1c20-41a5-b33f-5784a1f683d9-plugins-config-volume\") pod \"c0dbb750-1c20-41a5-b33f-5784a1f683d9\" (UID: \"c0dbb750-1c20-41a5-b33f-5784a1f683d9\") " Oct 30 09:55:27.168410 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:27.166804 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d696e344-8d35-4896-84dc-9a4dc225f291-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d696e344-8d35-4896-84dc-9a4dc225f291-plugins-config-volume\") pod \"d696e344-8d35-4896-84dc-9a4dc225f291\" (UID: \"d696e344-8d35-4896-84dc-9a4dc225f291\") " Oct 30 09:55:27.168410 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:27.166873 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d696e344-8d35-4896-84dc-9a4dc225f291-kube-api-access-q55n9\" (UniqueName: \"kubernetes.io/projected/d696e344-8d35-4896-84dc-9a4dc225f291-kube-api-access-q55n9\") pod \"d696e344-8d35-4896-84dc-9a4dc225f291\" (UID: \"d696e344-8d35-4896-84dc-9a4dc225f291\") " Oct 30 09:55:27.168410 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:27.166979 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0dbb750-1c20-41a5-b33f-5784a1f683d9-kube-api-access-vnx5k\" (UniqueName: \"kubernetes.io/projected/c0dbb750-1c20-41a5-b33f-5784a1f683d9-kube-api-access-vnx5k\") pod \"c0dbb750-1c20-41a5-b33f-5784a1f683d9\" (UID: \"c0dbb750-1c20-41a5-b33f-5784a1f683d9\") " Oct 30 09:55:27.204014 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:27.203872 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0dbb750-1c20-41a5-b33f-5784a1f683d9-plugins-config-volume" pod "c0dbb750-1c20-41a5-b33f-5784a1f683d9" (UID: "c0dbb750-1c20-41a5-b33f-5784a1f683d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:55:27.204635 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:27.204602 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d696e344-8d35-4896-84dc-9a4dc225f291-plugins-config-volume" pod "d696e344-8d35-4896-84dc-9a4dc225f291" (UID: "d696e344-8d35-4896-84dc-9a4dc225f291"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:55:27.225662 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:27.205225 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0dbb750-1c20-41a5-b33f-5784a1f683d9-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:55:27.225662 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:27.205261 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d696e344-8d35-4896-84dc-9a4dc225f291-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:55:27.287727 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:27.287678 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d696e344-8d35-4896-84dc-9a4dc225f291-kube-api-access-q55n9" pod "d696e344-8d35-4896-84dc-9a4dc225f291" (UID: "d696e344-8d35-4896-84dc-9a4dc225f291"). InnerVolumeSpecName "kube-api-access-q55n9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:55:27.300256 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:27.299561 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0dbb750-1c20-41a5-b33f-5784a1f683d9-kube-api-access-vnx5k" pod "c0dbb750-1c20-41a5-b33f-5784a1f683d9" (UID: "c0dbb750-1c20-41a5-b33f-5784a1f683d9"). InnerVolumeSpecName "kube-api-access-vnx5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:55:27.308434 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:27.307254 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q55n9\" (UniqueName: \"kubernetes.io/projected/d696e344-8d35-4896-84dc-9a4dc225f291-kube-api-access-q55n9\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:55:27.308434 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:27.307291 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnx5k\" (UniqueName: \"kubernetes.io/projected/c0dbb750-1c20-41a5-b33f-5784a1f683d9-kube-api-access-vnx5k\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:55:27.610546 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:27.610438 4562 generic.go:356] "Generic (PLEG): container finished" podID="d696e344-8d35-4896-84dc-9a4dc225f291" containerID="38db6254771c4c209929a8a6e6928f07fddcbedfc000d88ec0a81e643716de4d" exitCode=0 Oct 30 09:55:27.610546 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:27.610494 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-3d9b36-5cf56c76bd-c976p" Oct 30 09:55:27.610858 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:27.610530 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-3d9b36-5cf56c76bd-c976p" event={"ID":"d696e344-8d35-4896-84dc-9a4dc225f291","Type":"ContainerDied","Data":"38db6254771c4c209929a8a6e6928f07fddcbedfc000d88ec0a81e643716de4d"} Oct 30 09:55:27.610858 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:27.610606 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-3d9b36-5cf56c76bd-c976p" event={"ID":"d696e344-8d35-4896-84dc-9a4dc225f291","Type":"ContainerDied","Data":"57e38bc4eefd3a27221220af629641496de6f07577ce1e83b200de0c0427e191"} Oct 30 09:55:27.610858 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:27.610636 4562 scope.go:122] "RemoveContainer" containerID="38db6254771c4c209929a8a6e6928f07fddcbedfc000d88ec0a81e643716de4d" Oct 30 09:55:27.611986 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:27.611963 4562 generic.go:356] "Generic (PLEG): container finished" podID="c0dbb750-1c20-41a5-b33f-5784a1f683d9" containerID="ba2b86702a749f28cc495af83ae66f51e321838468d53bb897e6adf50741d459" exitCode=0 Oct 30 09:55:27.612114 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:27.611993 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3d9b36-5bcbf74867-k9hvd" event={"ID":"c0dbb750-1c20-41a5-b33f-5784a1f683d9","Type":"ContainerDied","Data":"ba2b86702a749f28cc495af83ae66f51e321838468d53bb897e6adf50741d459"} Oct 30 09:55:27.612114 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:27.612022 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3d9b36-5bcbf74867-k9hvd" event={"ID":"c0dbb750-1c20-41a5-b33f-5784a1f683d9","Type":"ContainerDied","Data":"804e66c52e1df89f4644f08468788a5ae2148a079e5c80e3b00d48ec101da041"} Oct 30 09:55:27.612114 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:27.612073 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3d9b36-5bcbf74867-k9hvd" Oct 30 09:55:27.622483 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:27.622358 4562 scope.go:122] "RemoveContainer" containerID="38db6254771c4c209929a8a6e6928f07fddcbedfc000d88ec0a81e643716de4d" Oct 30 09:55:27.622722 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:55:27.622692 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38db6254771c4c209929a8a6e6928f07fddcbedfc000d88ec0a81e643716de4d\": container with ID starting with 38db6254771c4c209929a8a6e6928f07fddcbedfc000d88ec0a81e643716de4d not found: ID does not exist" containerID="38db6254771c4c209929a8a6e6928f07fddcbedfc000d88ec0a81e643716de4d" Oct 30 09:55:27.622838 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:27.622728 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38db6254771c4c209929a8a6e6928f07fddcbedfc000d88ec0a81e643716de4d"} err="failed to get container status \"38db6254771c4c209929a8a6e6928f07fddcbedfc000d88ec0a81e643716de4d\": rpc error: code = NotFound desc = could not find container \"38db6254771c4c209929a8a6e6928f07fddcbedfc000d88ec0a81e643716de4d\": container with ID starting with 38db6254771c4c209929a8a6e6928f07fddcbedfc000d88ec0a81e643716de4d not found: ID does not exist" Oct 30 09:55:27.622838 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:27.622749 4562 scope.go:122] "RemoveContainer" containerID="ba2b86702a749f28cc495af83ae66f51e321838468d53bb897e6adf50741d459" Oct 30 09:55:27.638125 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:27.638099 4562 scope.go:122] "RemoveContainer" containerID="ba2b86702a749f28cc495af83ae66f51e321838468d53bb897e6adf50741d459" Oct 30 09:55:27.638526 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:55:27.638496 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba2b86702a749f28cc495af83ae66f51e321838468d53bb897e6adf50741d459\": container with ID starting with ba2b86702a749f28cc495af83ae66f51e321838468d53bb897e6adf50741d459 not found: ID does not exist" containerID="ba2b86702a749f28cc495af83ae66f51e321838468d53bb897e6adf50741d459" Oct 30 09:55:27.638641 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:27.638536 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba2b86702a749f28cc495af83ae66f51e321838468d53bb897e6adf50741d459"} err="failed to get container status \"ba2b86702a749f28cc495af83ae66f51e321838468d53bb897e6adf50741d459\": rpc error: code = NotFound desc = could not find container \"ba2b86702a749f28cc495af83ae66f51e321838468d53bb897e6adf50741d459\": container with ID starting with ba2b86702a749f28cc495af83ae66f51e321838468d53bb897e6adf50741d459 not found: ID does not exist" Oct 30 09:55:27.665276 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:27.665232 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-3d9b36-5bcbf74867-k9hvd"] Oct 30 09:55:27.678679 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:27.678645 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-3d9b36-5bcbf74867-k9hvd"] Oct 30 09:55:27.702824 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:27.702784 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-3d9b36-5cf56c76bd-c976p"] Oct 30 09:55:27.707453 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:27.707423 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-3d9b36-5cf56c76bd-c976p"] Oct 30 09:55:27.876763 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:27.876686 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0dbb750-1c20-41a5-b33f-5784a1f683d9" path="/var/lib/kubelet/pods/c0dbb750-1c20-41a5-b33f-5784a1f683d9/volumes" Oct 30 09:55:27.877193 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:27.877179 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d696e344-8d35-4896-84dc-9a4dc225f291" path="/var/lib/kubelet/pods/d696e344-8d35-4896-84dc-9a4dc225f291/volumes" Oct 30 09:55:28.599685 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:55:28.599576 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:55:28.724578 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:28.724546 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-d66d22-7bccc4d786-z6qjp" podUID="c8073ce8-37bf-4ad9-9311-d17bdc00fbb0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:55:32.582515 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:32.580404 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-3d9b36-59cf455fb5-qhp85"] Oct 30 09:55:32.582515 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:32.580661 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3d9b36-59cf455fb5-qhp85" podUID="0ecc2c25-ad9a-438b-9b2c-ab6e98905576" containerName="maas-api" containerID="cri-o://6c84dd2037088c60cac8c86cda595e0be39816f97cdb74ead68954a2b4046f5f" gracePeriod=30 Oct 30 09:55:32.600999 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:32.600968 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-d66d22-7bccc4d786-z6qjp" Oct 30 09:55:32.653237 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:32.653196 4562 generic.go:356] "Generic (PLEG): container finished" podID="c8073ce8-37bf-4ad9-9311-d17bdc00fbb0" containerID="76fecc19b8e65f4c5b62f01ae82c02e62b583d33352272559248ee08fa36b3fd" exitCode=0 Oct 30 09:55:32.653454 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:32.653401 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-d66d22-7bccc4d786-z6qjp" Oct 30 09:55:32.653454 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:32.653374 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-d66d22-7bccc4d786-z6qjp" event={"ID":"c8073ce8-37bf-4ad9-9311-d17bdc00fbb0","Type":"ContainerDied","Data":"76fecc19b8e65f4c5b62f01ae82c02e62b583d33352272559248ee08fa36b3fd"} Oct 30 09:55:32.653596 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:32.653482 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-d66d22-7bccc4d786-z6qjp" event={"ID":"c8073ce8-37bf-4ad9-9311-d17bdc00fbb0","Type":"ContainerDied","Data":"8a8d201e1ea4841d5bf9e95fe38aee8c7442bf47f90c7d333d055f5760c472a1"} Oct 30 09:55:32.653596 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:32.653513 4562 scope.go:122] "RemoveContainer" containerID="76fecc19b8e65f4c5b62f01ae82c02e62b583d33352272559248ee08fa36b3fd" Oct 30 09:55:32.685661 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:32.685626 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8073ce8-37bf-4ad9-9311-d17bdc00fbb0-kube-api-access-qgc4v\" (UniqueName: \"kubernetes.io/projected/c8073ce8-37bf-4ad9-9311-d17bdc00fbb0-kube-api-access-qgc4v\") pod \"c8073ce8-37bf-4ad9-9311-d17bdc00fbb0\" (UID: \"c8073ce8-37bf-4ad9-9311-d17bdc00fbb0\") " Oct 30 09:55:32.685869 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:32.685707 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8073ce8-37bf-4ad9-9311-d17bdc00fbb0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8073ce8-37bf-4ad9-9311-d17bdc00fbb0-plugins-config-volume\") pod \"c8073ce8-37bf-4ad9-9311-d17bdc00fbb0\" (UID: \"c8073ce8-37bf-4ad9-9311-d17bdc00fbb0\") " Oct 30 09:55:32.686158 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:32.686132 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8073ce8-37bf-4ad9-9311-d17bdc00fbb0-plugins-config-volume" pod "c8073ce8-37bf-4ad9-9311-d17bdc00fbb0" (UID: "c8073ce8-37bf-4ad9-9311-d17bdc00fbb0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:55:32.686577 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:32.686556 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8073ce8-37bf-4ad9-9311-d17bdc00fbb0-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:55:32.687928 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:32.687897 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8073ce8-37bf-4ad9-9311-d17bdc00fbb0-kube-api-access-qgc4v" pod "c8073ce8-37bf-4ad9-9311-d17bdc00fbb0" (UID: "c8073ce8-37bf-4ad9-9311-d17bdc00fbb0"). InnerVolumeSpecName "kube-api-access-qgc4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:55:32.711674 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:32.711649 4562 scope.go:122] "RemoveContainer" containerID="76fecc19b8e65f4c5b62f01ae82c02e62b583d33352272559248ee08fa36b3fd" Oct 30 09:55:32.712063 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:55:32.712028 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76fecc19b8e65f4c5b62f01ae82c02e62b583d33352272559248ee08fa36b3fd\": container with ID starting with 76fecc19b8e65f4c5b62f01ae82c02e62b583d33352272559248ee08fa36b3fd not found: ID does not exist" containerID="76fecc19b8e65f4c5b62f01ae82c02e62b583d33352272559248ee08fa36b3fd" Oct 30 09:55:32.712164 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:32.712077 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76fecc19b8e65f4c5b62f01ae82c02e62b583d33352272559248ee08fa36b3fd"} err="failed to get container status \"76fecc19b8e65f4c5b62f01ae82c02e62b583d33352272559248ee08fa36b3fd\": rpc error: code = NotFound desc = could not find container \"76fecc19b8e65f4c5b62f01ae82c02e62b583d33352272559248ee08fa36b3fd\": container with ID starting with 76fecc19b8e65f4c5b62f01ae82c02e62b583d33352272559248ee08fa36b3fd not found: ID does not exist" Oct 30 09:55:32.788299 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:32.788259 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgc4v\" (UniqueName: \"kubernetes.io/projected/c8073ce8-37bf-4ad9-9311-d17bdc00fbb0-kube-api-access-qgc4v\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:55:32.889998 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:32.875823 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3d9b36-59cf455fb5-qhp85" Oct 30 09:55:32.891456 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:32.890664 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ecc2c25-ad9a-438b-9b2c-ab6e98905576-kube-api-access-wbdrc\" (UniqueName: \"kubernetes.io/projected/0ecc2c25-ad9a-438b-9b2c-ab6e98905576-kube-api-access-wbdrc\") pod \"0ecc2c25-ad9a-438b-9b2c-ab6e98905576\" (UID: \"0ecc2c25-ad9a-438b-9b2c-ab6e98905576\") " Oct 30 09:55:32.891456 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:32.890732 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ecc2c25-ad9a-438b-9b2c-ab6e98905576-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ecc2c25-ad9a-438b-9b2c-ab6e98905576-openshift-service-ca\") pod \"0ecc2c25-ad9a-438b-9b2c-ab6e98905576\" (UID: \"0ecc2c25-ad9a-438b-9b2c-ab6e98905576\") " Oct 30 09:55:32.891456 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:32.890907 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ecc2c25-ad9a-438b-9b2c-ab6e98905576-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ecc2c25-ad9a-438b-9b2c-ab6e98905576-maas-api-tls\") pod \"0ecc2c25-ad9a-438b-9b2c-ab6e98905576\" (UID: \"0ecc2c25-ad9a-438b-9b2c-ab6e98905576\") " Oct 30 09:55:32.891456 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:32.890957 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ecc2c25-ad9a-438b-9b2c-ab6e98905576-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ecc2c25-ad9a-438b-9b2c-ab6e98905576-metrics-certs\") pod \"0ecc2c25-ad9a-438b-9b2c-ab6e98905576\" (UID: \"0ecc2c25-ad9a-438b-9b2c-ab6e98905576\") " Oct 30 09:55:32.892759 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:32.892735 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ecc2c25-ad9a-438b-9b2c-ab6e98905576-openshift-service-ca" pod "0ecc2c25-ad9a-438b-9b2c-ab6e98905576" (UID: "0ecc2c25-ad9a-438b-9b2c-ab6e98905576"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:55:32.907647 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:32.907607 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ecc2c25-ad9a-438b-9b2c-ab6e98905576-maas-api-tls" pod "0ecc2c25-ad9a-438b-9b2c-ab6e98905576" (UID: "0ecc2c25-ad9a-438b-9b2c-ab6e98905576"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:55:32.932232 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:32.932187 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ecc2c25-ad9a-438b-9b2c-ab6e98905576-kube-api-access-wbdrc" pod "0ecc2c25-ad9a-438b-9b2c-ab6e98905576" (UID: "0ecc2c25-ad9a-438b-9b2c-ab6e98905576"). InnerVolumeSpecName "kube-api-access-wbdrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:55:32.932460 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:32.932290 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ecc2c25-ad9a-438b-9b2c-ab6e98905576-metrics-certs" pod "0ecc2c25-ad9a-438b-9b2c-ab6e98905576" (UID: "0ecc2c25-ad9a-438b-9b2c-ab6e98905576"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:55:32.996407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:32.994165 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbdrc\" (UniqueName: \"kubernetes.io/projected/0ecc2c25-ad9a-438b-9b2c-ab6e98905576-kube-api-access-wbdrc\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:55:32.996407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:32.994201 4562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ecc2c25-ad9a-438b-9b2c-ab6e98905576-openshift-service-ca\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:55:32.996407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:32.994217 4562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ecc2c25-ad9a-438b-9b2c-ab6e98905576-maas-api-tls\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:55:32.996407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:32.994231 4562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ecc2c25-ad9a-438b-9b2c-ab6e98905576-metrics-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:55:33.028520 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:33.028478 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d66d22-7bccc4d786-z6qjp"] Oct 30 09:55:33.043408 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:33.042226 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d66d22-7bccc4d786-z6qjp"] Oct 30 09:55:33.663502 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:33.663311 4562 generic.go:356] "Generic (PLEG): container finished" podID="0ecc2c25-ad9a-438b-9b2c-ab6e98905576" containerID="6c84dd2037088c60cac8c86cda595e0be39816f97cdb74ead68954a2b4046f5f" exitCode=0 Oct 30 09:55:33.663502 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:33.663361 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3d9b36-59cf455fb5-qhp85" Oct 30 09:55:33.663502 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:33.663410 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3d9b36-59cf455fb5-qhp85" event={"ID":"0ecc2c25-ad9a-438b-9b2c-ab6e98905576","Type":"ContainerDied","Data":"6c84dd2037088c60cac8c86cda595e0be39816f97cdb74ead68954a2b4046f5f"} Oct 30 09:55:33.663502 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:33.663470 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3d9b36-59cf455fb5-qhp85" event={"ID":"0ecc2c25-ad9a-438b-9b2c-ab6e98905576","Type":"ContainerDied","Data":"d82ec7a2bc52df9d8f2b1e71de89e6b55b7e3c71ed20ce5e10068f7742a8e866"} Oct 30 09:55:33.663502 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:33.663505 4562 scope.go:122] "RemoveContainer" containerID="6c84dd2037088c60cac8c86cda595e0be39816f97cdb74ead68954a2b4046f5f" Oct 30 09:55:33.674546 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:33.674528 4562 scope.go:122] "RemoveContainer" containerID="6c84dd2037088c60cac8c86cda595e0be39816f97cdb74ead68954a2b4046f5f" Oct 30 09:55:33.674809 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:55:33.674787 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c84dd2037088c60cac8c86cda595e0be39816f97cdb74ead68954a2b4046f5f\": container with ID starting with 6c84dd2037088c60cac8c86cda595e0be39816f97cdb74ead68954a2b4046f5f not found: ID does not exist" containerID="6c84dd2037088c60cac8c86cda595e0be39816f97cdb74ead68954a2b4046f5f" Oct 30 09:55:33.674866 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:33.674818 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c84dd2037088c60cac8c86cda595e0be39816f97cdb74ead68954a2b4046f5f"} err="failed to get container status \"6c84dd2037088c60cac8c86cda595e0be39816f97cdb74ead68954a2b4046f5f\": rpc error: code = NotFound desc = could not find container \"6c84dd2037088c60cac8c86cda595e0be39816f97cdb74ead68954a2b4046f5f\": container with ID starting with 6c84dd2037088c60cac8c86cda595e0be39816f97cdb74ead68954a2b4046f5f not found: ID does not exist" Oct 30 09:55:33.717704 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:33.717671 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-3d9b36-59cf455fb5-qhp85"] Oct 30 09:55:33.730699 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:33.730666 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-3d9b36-59cf455fb5-qhp85"] Oct 30 09:55:33.877042 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:33.877011 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ecc2c25-ad9a-438b-9b2c-ab6e98905576" path="/var/lib/kubelet/pods/0ecc2c25-ad9a-438b-9b2c-ab6e98905576/volumes" Oct 30 09:55:33.877487 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:33.877471 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8073ce8-37bf-4ad9-9311-d17bdc00fbb0" path="/var/lib/kubelet/pods/c8073ce8-37bf-4ad9-9311-d17bdc00fbb0/volumes" Oct 30 09:55:36.398339 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:36.398303 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d66d22-f86b5c5b-7wsft"] Oct 30 09:55:36.398887 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:36.398573 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d66d22-f86b5c5b-7wsft" podUID="17acd011-7a83-4ac2-9689-3a5568b7f94b" containerName="maas-api" containerID="cri-o://673152ac959d04641228d6e3ac7decbec9deafda8237ca6e86d095729fc16e58" gracePeriod=30 Oct 30 09:55:36.771568 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:36.768850 4562 generic.go:356] "Generic (PLEG): container finished" podID="17acd011-7a83-4ac2-9689-3a5568b7f94b" containerID="673152ac959d04641228d6e3ac7decbec9deafda8237ca6e86d095729fc16e58" exitCode=0 Oct 30 09:55:36.771568 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:36.768899 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d66d22-f86b5c5b-7wsft" event={"ID":"17acd011-7a83-4ac2-9689-3a5568b7f94b","Type":"ContainerDied","Data":"673152ac959d04641228d6e3ac7decbec9deafda8237ca6e86d095729fc16e58"} Oct 30 09:55:36.900682 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:36.900648 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d66d22-f86b5c5b-7wsft" Oct 30 09:55:37.008406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:37.005925 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17acd011-7a83-4ac2-9689-3a5568b7f94b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/17acd011-7a83-4ac2-9689-3a5568b7f94b-metrics-certs\") pod \"17acd011-7a83-4ac2-9689-3a5568b7f94b\" (UID: \"17acd011-7a83-4ac2-9689-3a5568b7f94b\") " Oct 30 09:55:37.008406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:37.006048 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17acd011-7a83-4ac2-9689-3a5568b7f94b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17acd011-7a83-4ac2-9689-3a5568b7f94b-maas-api-tls\") pod \"17acd011-7a83-4ac2-9689-3a5568b7f94b\" (UID: \"17acd011-7a83-4ac2-9689-3a5568b7f94b\") " Oct 30 09:55:37.008406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:37.006277 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17acd011-7a83-4ac2-9689-3a5568b7f94b-kube-api-access-4xstl\" (UniqueName: \"kubernetes.io/projected/17acd011-7a83-4ac2-9689-3a5568b7f94b-kube-api-access-4xstl\") pod \"17acd011-7a83-4ac2-9689-3a5568b7f94b\" (UID: \"17acd011-7a83-4ac2-9689-3a5568b7f94b\") " Oct 30 09:55:37.008406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:37.006308 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17acd011-7a83-4ac2-9689-3a5568b7f94b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17acd011-7a83-4ac2-9689-3a5568b7f94b-openshift-service-ca\") pod \"17acd011-7a83-4ac2-9689-3a5568b7f94b\" (UID: \"17acd011-7a83-4ac2-9689-3a5568b7f94b\") " Oct 30 09:55:37.021289 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:37.008840 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17acd011-7a83-4ac2-9689-3a5568b7f94b-openshift-service-ca" pod "17acd011-7a83-4ac2-9689-3a5568b7f94b" (UID: "17acd011-7a83-4ac2-9689-3a5568b7f94b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:55:37.029244 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:37.029164 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17acd011-7a83-4ac2-9689-3a5568b7f94b-metrics-certs" pod "17acd011-7a83-4ac2-9689-3a5568b7f94b" (UID: "17acd011-7a83-4ac2-9689-3a5568b7f94b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:55:37.029876 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:37.029845 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17acd011-7a83-4ac2-9689-3a5568b7f94b-maas-api-tls" pod "17acd011-7a83-4ac2-9689-3a5568b7f94b" (UID: "17acd011-7a83-4ac2-9689-3a5568b7f94b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:55:37.031560 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:37.031521 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17acd011-7a83-4ac2-9689-3a5568b7f94b-kube-api-access-4xstl" pod "17acd011-7a83-4ac2-9689-3a5568b7f94b" (UID: "17acd011-7a83-4ac2-9689-3a5568b7f94b"). InnerVolumeSpecName "kube-api-access-4xstl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:55:37.118415 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:37.109579 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xstl\" (UniqueName: \"kubernetes.io/projected/17acd011-7a83-4ac2-9689-3a5568b7f94b-kube-api-access-4xstl\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:55:37.118415 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:37.109624 4562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17acd011-7a83-4ac2-9689-3a5568b7f94b-openshift-service-ca\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:55:37.118415 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:37.109639 4562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17acd011-7a83-4ac2-9689-3a5568b7f94b-metrics-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:55:37.118415 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:37.109651 4562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17acd011-7a83-4ac2-9689-3a5568b7f94b-maas-api-tls\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:55:37.779170 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:37.779135 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d66d22-f86b5c5b-7wsft" Oct 30 09:55:37.779170 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:37.779157 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d66d22-f86b5c5b-7wsft" event={"ID":"17acd011-7a83-4ac2-9689-3a5568b7f94b","Type":"ContainerDied","Data":"e6d270ba86d4a9f86b2379a538531df2d26c184ee1534432eca2df77746b9044"} Oct 30 09:55:37.779888 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:37.779229 4562 scope.go:122] "RemoveContainer" containerID="673152ac959d04641228d6e3ac7decbec9deafda8237ca6e86d095729fc16e58" Oct 30 09:55:37.865433 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:37.863753 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d66d22-f86b5c5b-7wsft"] Oct 30 09:55:37.936559 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:37.936527 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d66d22-f86b5c5b-7wsft"] Oct 30 09:55:39.876711 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:39.876677 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17acd011-7a83-4ac2-9689-3a5568b7f94b" path="/var/lib/kubelet/pods/17acd011-7a83-4ac2-9689-3a5568b7f94b/volumes" Oct 30 09:55:55.054545 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:55:55.054504 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-304459-69f5bf967f-l4kvg"] Oct 30 09:55:58.598495 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:55:58.598457 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:56:00.055188 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:00.055144 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-304459-69f5bf967f-l4kvg" podUID="a75406b8-14aa-4fc1-975e-620b911cb9b7" containerName="payload-pre-processing" containerID="cri-o://f85f9e1f90c5f66cac6e0e729237d2a15487fcb905a1919cef7b40bc14a45c09" gracePeriod=25 Oct 30 09:56:00.390065 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:00.390037 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-304459-69f5bf967f-l4kvg" Oct 30 09:56:00.480006 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:00.479974 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a75406b8-14aa-4fc1-975e-620b911cb9b7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a75406b8-14aa-4fc1-975e-620b911cb9b7-plugins-config-volume\") pod \"a75406b8-14aa-4fc1-975e-620b911cb9b7\" (UID: \"a75406b8-14aa-4fc1-975e-620b911cb9b7\") " Oct 30 09:56:00.480224 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:00.480152 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a75406b8-14aa-4fc1-975e-620b911cb9b7-kube-api-access-v9942\" (UniqueName: \"kubernetes.io/projected/a75406b8-14aa-4fc1-975e-620b911cb9b7-kube-api-access-v9942\") pod \"a75406b8-14aa-4fc1-975e-620b911cb9b7\" (UID: \"a75406b8-14aa-4fc1-975e-620b911cb9b7\") " Oct 30 09:56:00.480434 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:00.480404 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a75406b8-14aa-4fc1-975e-620b911cb9b7-plugins-config-volume" pod "a75406b8-14aa-4fc1-975e-620b911cb9b7" (UID: "a75406b8-14aa-4fc1-975e-620b911cb9b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:56:00.480844 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:00.480816 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a75406b8-14aa-4fc1-975e-620b911cb9b7-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:56:00.482566 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:00.482543 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a75406b8-14aa-4fc1-975e-620b911cb9b7-kube-api-access-v9942" pod "a75406b8-14aa-4fc1-975e-620b911cb9b7" (UID: "a75406b8-14aa-4fc1-975e-620b911cb9b7"). InnerVolumeSpecName "kube-api-access-v9942". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:56:00.582703 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:00.582669 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9942\" (UniqueName: \"kubernetes.io/projected/a75406b8-14aa-4fc1-975e-620b911cb9b7-kube-api-access-v9942\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:56:00.968887 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:00.968849 4562 generic.go:356] "Generic (PLEG): container finished" podID="a75406b8-14aa-4fc1-975e-620b911cb9b7" containerID="f85f9e1f90c5f66cac6e0e729237d2a15487fcb905a1919cef7b40bc14a45c09" exitCode=0 Oct 30 09:56:00.969067 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:00.968936 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-304459-69f5bf967f-l4kvg" event={"ID":"a75406b8-14aa-4fc1-975e-620b911cb9b7","Type":"ContainerDied","Data":"f85f9e1f90c5f66cac6e0e729237d2a15487fcb905a1919cef7b40bc14a45c09"} Oct 30 09:56:00.969067 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:00.968976 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-304459-69f5bf967f-l4kvg" event={"ID":"a75406b8-14aa-4fc1-975e-620b911cb9b7","Type":"ContainerDied","Data":"75c2ed182902b5a92aa05f04afc974f4d6757be9a41ecd689543dd85f7e438a2"} Oct 30 09:56:00.969067 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:00.968995 4562 scope.go:122] "RemoveContainer" containerID="f85f9e1f90c5f66cac6e0e729237d2a15487fcb905a1919cef7b40bc14a45c09" Oct 30 09:56:00.969175 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:00.969119 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-304459-69f5bf967f-l4kvg" Oct 30 09:56:00.982027 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:00.982006 4562 scope.go:122] "RemoveContainer" containerID="f85f9e1f90c5f66cac6e0e729237d2a15487fcb905a1919cef7b40bc14a45c09" Oct 30 09:56:00.982321 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:56:00.982293 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f85f9e1f90c5f66cac6e0e729237d2a15487fcb905a1919cef7b40bc14a45c09\": container with ID starting with f85f9e1f90c5f66cac6e0e729237d2a15487fcb905a1919cef7b40bc14a45c09 not found: ID does not exist" containerID="f85f9e1f90c5f66cac6e0e729237d2a15487fcb905a1919cef7b40bc14a45c09" Oct 30 09:56:00.982463 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:00.982325 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f85f9e1f90c5f66cac6e0e729237d2a15487fcb905a1919cef7b40bc14a45c09"} err="failed to get container status \"f85f9e1f90c5f66cac6e0e729237d2a15487fcb905a1919cef7b40bc14a45c09\": rpc error: code = NotFound desc = could not find container \"f85f9e1f90c5f66cac6e0e729237d2a15487fcb905a1919cef7b40bc14a45c09\": container with ID starting with f85f9e1f90c5f66cac6e0e729237d2a15487fcb905a1919cef7b40bc14a45c09 not found: ID does not exist" Oct 30 09:56:01.027016 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:01.026983 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-304459-69f5bf967f-l4kvg"] Oct 30 09:56:01.036700 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:01.036666 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-304459-69f5bf967f-l4kvg"] Oct 30 09:56:01.877510 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:01.877472 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a75406b8-14aa-4fc1-975e-620b911cb9b7" path="/var/lib/kubelet/pods/a75406b8-14aa-4fc1-975e-620b911cb9b7/volumes" Oct 30 09:56:10.449107 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:10.449071 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-aa2fdf-799786b6bf-vtjc9"] Oct 30 09:56:10.449737 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:10.449363 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-aa2fdf-799786b6bf-vtjc9" podUID="37fad2d6-d1ba-4440-bf0f-0f46c12987fb" containerName="maas-api" containerID="cri-o://19600e41907c2749e4a4e3d197702cc014c643bd306b2c7de963218a81156f9f" gracePeriod=30 Oct 30 09:56:11.012428 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:11.011996 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb1a2a-5d7444785b-h7zjw"] Oct 30 09:56:11.012634 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:11.012442 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb1a2a-5d7444785b-h7zjw" podUID="7a163089-8bc6-4fde-afbb-7970c8ae3a08" containerName="maas-api" containerID="cri-o://116268acfca5119aac2a934f021d97f6f5ad58b1259c3406dd473cfffd34e180" gracePeriod=30 Oct 30 09:56:11.033224 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:11.031991 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-aa2fdf-799786b6bf-vtjc9" Oct 30 09:56:11.133426 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:11.133115 4562 generic.go:356] "Generic (PLEG): container finished" podID="37fad2d6-d1ba-4440-bf0f-0f46c12987fb" containerID="19600e41907c2749e4a4e3d197702cc014c643bd306b2c7de963218a81156f9f" exitCode=0 Oct 30 09:56:11.133426 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:11.133287 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-aa2fdf-799786b6bf-vtjc9" event={"ID":"37fad2d6-d1ba-4440-bf0f-0f46c12987fb","Type":"ContainerDied","Data":"19600e41907c2749e4a4e3d197702cc014c643bd306b2c7de963218a81156f9f"} Oct 30 09:56:11.133426 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:11.133334 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-aa2fdf-799786b6bf-vtjc9" event={"ID":"37fad2d6-d1ba-4440-bf0f-0f46c12987fb","Type":"ContainerDied","Data":"4597e9a969cc8229029c87e5dba6999d2ac77208c3112462139c8187519be431"} Oct 30 09:56:11.133426 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:11.133361 4562 scope.go:122] "RemoveContainer" containerID="19600e41907c2749e4a4e3d197702cc014c643bd306b2c7de963218a81156f9f" Oct 30 09:56:11.135377 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:11.133515 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-aa2fdf-799786b6bf-vtjc9" Oct 30 09:56:11.150227 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:11.150151 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37fad2d6-d1ba-4440-bf0f-0f46c12987fb-kube-api-access-xkhhk\" (UniqueName: \"kubernetes.io/projected/37fad2d6-d1ba-4440-bf0f-0f46c12987fb-kube-api-access-xkhhk\") pod \"37fad2d6-d1ba-4440-bf0f-0f46c12987fb\" (UID: \"37fad2d6-d1ba-4440-bf0f-0f46c12987fb\") " Oct 30 09:56:11.150472 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:11.150306 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37fad2d6-d1ba-4440-bf0f-0f46c12987fb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/37fad2d6-d1ba-4440-bf0f-0f46c12987fb-metrics-certs\") pod \"37fad2d6-d1ba-4440-bf0f-0f46c12987fb\" (UID: \"37fad2d6-d1ba-4440-bf0f-0f46c12987fb\") " Oct 30 09:56:11.150580 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:11.150474 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37fad2d6-d1ba-4440-bf0f-0f46c12987fb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37fad2d6-d1ba-4440-bf0f-0f46c12987fb-openshift-service-ca\") pod \"37fad2d6-d1ba-4440-bf0f-0f46c12987fb\" (UID: \"37fad2d6-d1ba-4440-bf0f-0f46c12987fb\") " Oct 30 09:56:11.150580 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:11.150509 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37fad2d6-d1ba-4440-bf0f-0f46c12987fb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37fad2d6-d1ba-4440-bf0f-0f46c12987fb-maas-api-tls\") pod \"37fad2d6-d1ba-4440-bf0f-0f46c12987fb\" (UID: \"37fad2d6-d1ba-4440-bf0f-0f46c12987fb\") " Oct 30 09:56:11.153434 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:11.150923 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37fad2d6-d1ba-4440-bf0f-0f46c12987fb-openshift-service-ca" pod "37fad2d6-d1ba-4440-bf0f-0f46c12987fb" (UID: "37fad2d6-d1ba-4440-bf0f-0f46c12987fb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:56:11.153434 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:11.151556 4562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37fad2d6-d1ba-4440-bf0f-0f46c12987fb-openshift-service-ca\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:56:11.155614 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:11.155584 4562 scope.go:122] "RemoveContainer" containerID="19600e41907c2749e4a4e3d197702cc014c643bd306b2c7de963218a81156f9f" Oct 30 09:56:11.156202 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:56:11.156171 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19600e41907c2749e4a4e3d197702cc014c643bd306b2c7de963218a81156f9f\": container with ID starting with 19600e41907c2749e4a4e3d197702cc014c643bd306b2c7de963218a81156f9f not found: ID does not exist" containerID="19600e41907c2749e4a4e3d197702cc014c643bd306b2c7de963218a81156f9f" Oct 30 09:56:11.156297 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:11.156212 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19600e41907c2749e4a4e3d197702cc014c643bd306b2c7de963218a81156f9f"} err="failed to get container status \"19600e41907c2749e4a4e3d197702cc014c643bd306b2c7de963218a81156f9f\": rpc error: code = NotFound desc = could not find container \"19600e41907c2749e4a4e3d197702cc014c643bd306b2c7de963218a81156f9f\": container with ID starting with 19600e41907c2749e4a4e3d197702cc014c643bd306b2c7de963218a81156f9f not found: ID does not exist" Oct 30 09:56:11.157475 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:11.157374 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37fad2d6-d1ba-4440-bf0f-0f46c12987fb-metrics-certs" pod "37fad2d6-d1ba-4440-bf0f-0f46c12987fb" (UID: "37fad2d6-d1ba-4440-bf0f-0f46c12987fb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:56:11.159503 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:11.159476 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37fad2d6-d1ba-4440-bf0f-0f46c12987fb-kube-api-access-xkhhk" pod "37fad2d6-d1ba-4440-bf0f-0f46c12987fb" (UID: "37fad2d6-d1ba-4440-bf0f-0f46c12987fb"). InnerVolumeSpecName "kube-api-access-xkhhk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:56:11.160622 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:11.160572 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37fad2d6-d1ba-4440-bf0f-0f46c12987fb-maas-api-tls" pod "37fad2d6-d1ba-4440-bf0f-0f46c12987fb" (UID: "37fad2d6-d1ba-4440-bf0f-0f46c12987fb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:56:11.253967 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:11.253922 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkhhk\" (UniqueName: \"kubernetes.io/projected/37fad2d6-d1ba-4440-bf0f-0f46c12987fb-kube-api-access-xkhhk\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:56:11.253967 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:11.253968 4562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37fad2d6-d1ba-4440-bf0f-0f46c12987fb-metrics-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:56:11.254232 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:11.253985 4562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37fad2d6-d1ba-4440-bf0f-0f46c12987fb-maas-api-tls\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:56:11.495519 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:11.495491 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb1a2a-5d7444785b-h7zjw" Oct 30 09:56:11.539604 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:11.539564 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-aa2fdf-799786b6bf-vtjc9"] Oct 30 09:56:11.545868 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:11.545827 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-aa2fdf-799786b6bf-vtjc9"] Oct 30 09:56:11.671232 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:11.671197 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a163089-8bc6-4fde-afbb-7970c8ae3a08-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a163089-8bc6-4fde-afbb-7970c8ae3a08-metrics-certs\") pod \"7a163089-8bc6-4fde-afbb-7970c8ae3a08\" (UID: \"7a163089-8bc6-4fde-afbb-7970c8ae3a08\") " Oct 30 09:56:11.671473 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:11.671268 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a163089-8bc6-4fde-afbb-7970c8ae3a08-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a163089-8bc6-4fde-afbb-7970c8ae3a08-openshift-service-ca\") pod \"7a163089-8bc6-4fde-afbb-7970c8ae3a08\" (UID: \"7a163089-8bc6-4fde-afbb-7970c8ae3a08\") " Oct 30 09:56:11.671567 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:11.671468 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a163089-8bc6-4fde-afbb-7970c8ae3a08-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a163089-8bc6-4fde-afbb-7970c8ae3a08-maas-api-tls\") pod \"7a163089-8bc6-4fde-afbb-7970c8ae3a08\" (UID: \"7a163089-8bc6-4fde-afbb-7970c8ae3a08\") " Oct 30 09:56:11.671567 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:11.671542 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a163089-8bc6-4fde-afbb-7970c8ae3a08-kube-api-access-9cvg6\" (UniqueName: \"kubernetes.io/projected/7a163089-8bc6-4fde-afbb-7970c8ae3a08-kube-api-access-9cvg6\") pod \"7a163089-8bc6-4fde-afbb-7970c8ae3a08\" (UID: \"7a163089-8bc6-4fde-afbb-7970c8ae3a08\") " Oct 30 09:56:11.671683 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:11.671646 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a163089-8bc6-4fde-afbb-7970c8ae3a08-openshift-service-ca" pod "7a163089-8bc6-4fde-afbb-7970c8ae3a08" (UID: "7a163089-8bc6-4fde-afbb-7970c8ae3a08"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:56:11.672038 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:11.672021 4562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a163089-8bc6-4fde-afbb-7970c8ae3a08-openshift-service-ca\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:56:11.673459 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:11.673374 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a163089-8bc6-4fde-afbb-7970c8ae3a08-metrics-certs" pod "7a163089-8bc6-4fde-afbb-7970c8ae3a08" (UID: "7a163089-8bc6-4fde-afbb-7970c8ae3a08"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:56:11.673582 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:11.673565 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a163089-8bc6-4fde-afbb-7970c8ae3a08-kube-api-access-9cvg6" pod "7a163089-8bc6-4fde-afbb-7970c8ae3a08" (UID: "7a163089-8bc6-4fde-afbb-7970c8ae3a08"). InnerVolumeSpecName "kube-api-access-9cvg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:56:11.673706 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:11.673660 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a163089-8bc6-4fde-afbb-7970c8ae3a08-maas-api-tls" pod "7a163089-8bc6-4fde-afbb-7970c8ae3a08" (UID: "7a163089-8bc6-4fde-afbb-7970c8ae3a08"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:56:11.774249 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:11.774209 4562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a163089-8bc6-4fde-afbb-7970c8ae3a08-metrics-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:56:11.774249 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:11.774241 4562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a163089-8bc6-4fde-afbb-7970c8ae3a08-maas-api-tls\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:56:11.774249 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:11.774251 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cvg6\" (UniqueName: \"kubernetes.io/projected/7a163089-8bc6-4fde-afbb-7970c8ae3a08-kube-api-access-9cvg6\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:56:11.896408 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:11.879020 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37fad2d6-d1ba-4440-bf0f-0f46c12987fb" path="/var/lib/kubelet/pods/37fad2d6-d1ba-4440-bf0f-0f46c12987fb/volumes" Oct 30 09:56:12.142723 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:12.142606 4562 generic.go:356] "Generic (PLEG): container finished" podID="7a163089-8bc6-4fde-afbb-7970c8ae3a08" containerID="116268acfca5119aac2a934f021d97f6f5ad58b1259c3406dd473cfffd34e180" exitCode=0 Oct 30 09:56:12.142892 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:12.142752 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb1a2a-5d7444785b-h7zjw" event={"ID":"7a163089-8bc6-4fde-afbb-7970c8ae3a08","Type":"ContainerDied","Data":"116268acfca5119aac2a934f021d97f6f5ad58b1259c3406dd473cfffd34e180"} Oct 30 09:56:12.142892 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:12.142811 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb1a2a-5d7444785b-h7zjw" event={"ID":"7a163089-8bc6-4fde-afbb-7970c8ae3a08","Type":"ContainerDied","Data":"8c8b8ce21f8bacbd78fe4ba1aa67f024900d87123cdd238867f8036bc0f148fe"} Oct 30 09:56:12.142892 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:12.142841 4562 scope.go:122] "RemoveContainer" containerID="116268acfca5119aac2a934f021d97f6f5ad58b1259c3406dd473cfffd34e180" Oct 30 09:56:12.143042 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:12.142955 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb1a2a-5d7444785b-h7zjw" Oct 30 09:56:12.155443 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:12.155419 4562 scope.go:122] "RemoveContainer" containerID="116268acfca5119aac2a934f021d97f6f5ad58b1259c3406dd473cfffd34e180" Oct 30 09:56:12.155773 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:56:12.155746 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"116268acfca5119aac2a934f021d97f6f5ad58b1259c3406dd473cfffd34e180\": container with ID starting with 116268acfca5119aac2a934f021d97f6f5ad58b1259c3406dd473cfffd34e180 not found: ID does not exist" containerID="116268acfca5119aac2a934f021d97f6f5ad58b1259c3406dd473cfffd34e180" Oct 30 09:56:12.155841 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:12.155786 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"116268acfca5119aac2a934f021d97f6f5ad58b1259c3406dd473cfffd34e180"} err="failed to get container status \"116268acfca5119aac2a934f021d97f6f5ad58b1259c3406dd473cfffd34e180\": rpc error: code = NotFound desc = could not find container \"116268acfca5119aac2a934f021d97f6f5ad58b1259c3406dd473cfffd34e180\": container with ID starting with 116268acfca5119aac2a934f021d97f6f5ad58b1259c3406dd473cfffd34e180 not found: ID does not exist" Oct 30 09:56:12.198545 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:12.198507 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb1a2a-5d7444785b-h7zjw"] Oct 30 09:56:12.202405 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:12.202360 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb1a2a-5d7444785b-h7zjw"] Oct 30 09:56:13.877637 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:13.877228 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a163089-8bc6-4fde-afbb-7970c8ae3a08" path="/var/lib/kubelet/pods/7a163089-8bc6-4fde-afbb-7970c8ae3a08/volumes" Oct 30 09:56:18.661735 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:18.659892 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr"] Oct 30 09:56:18.661735 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:18.660620 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr" podUID="310de27f-d65f-4c73-840a-df4b5f6aed8b" containerName="istio-proxy" containerID="cri-o://d8892b8b3bcc9a80af360f1d79970a7a39fd4141ac28bba34b334f125530bbc6" gracePeriod=30 Oct 30 09:56:18.672934 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:18.672902 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67"] Oct 30 09:56:18.673303 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:18.673267 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67" podUID="821c88fc-b72b-4104-97f3-e3528428498e" containerName="istio-proxy" containerID="cri-o://c21f8960ca996a0e9b070c9a4bcd15ff37dfd7532869991b5332e577f3cc37e4" gracePeriod=30 Oct 30 09:56:23.620523 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:23.620484 4562 patch_prober.go:29] interesting pod/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 09:56:23.621024 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:23.620544 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67" podUID="821c88fc-b72b-4104-97f3-e3528428498e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 09:56:23.977765 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:23.977731 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67" Oct 30 09:56:24.015462 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.015433 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr" Oct 30 09:56:24.123633 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.123592 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-c5447d/distinct-2-w1-c5447d-kserve-676749d548-xlmsf"] Oct 30 09:56:24.146784 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.146751 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/821c88fc-b72b-4104-97f3-e3528428498e-istio-token\" (UniqueName: \"kubernetes.io/projected/821c88fc-b72b-4104-97f3-e3528428498e-istio-token\") pod \"821c88fc-b72b-4104-97f3-e3528428498e\" (UID: \"821c88fc-b72b-4104-97f3-e3528428498e\") " Oct 30 09:56:24.147027 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.146808 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/821c88fc-b72b-4104-97f3-e3528428498e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/821c88fc-b72b-4104-97f3-e3528428498e-istio-data\") pod \"821c88fc-b72b-4104-97f3-e3528428498e\" (UID: \"821c88fc-b72b-4104-97f3-e3528428498e\") " Oct 30 09:56:24.147027 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.146881 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/821c88fc-b72b-4104-97f3-e3528428498e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/821c88fc-b72b-4104-97f3-e3528428498e-istiod-ca-cert\") pod \"821c88fc-b72b-4104-97f3-e3528428498e\" (UID: \"821c88fc-b72b-4104-97f3-e3528428498e\") " Oct 30 09:56:24.147027 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.146906 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/821c88fc-b72b-4104-97f3-e3528428498e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/821c88fc-b72b-4104-97f3-e3528428498e-workload-socket\") pod \"821c88fc-b72b-4104-97f3-e3528428498e\" (UID: \"821c88fc-b72b-4104-97f3-e3528428498e\") " Oct 30 09:56:24.147027 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.146948 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/821c88fc-b72b-4104-97f3-e3528428498e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/821c88fc-b72b-4104-97f3-e3528428498e-istio-envoy\") pod \"821c88fc-b72b-4104-97f3-e3528428498e\" (UID: \"821c88fc-b72b-4104-97f3-e3528428498e\") " Oct 30 09:56:24.147027 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.147005 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/310de27f-d65f-4c73-840a-df4b5f6aed8b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/310de27f-d65f-4c73-840a-df4b5f6aed8b-credential-socket\") pod \"310de27f-d65f-4c73-840a-df4b5f6aed8b\" (UID: \"310de27f-d65f-4c73-840a-df4b5f6aed8b\") " Oct 30 09:56:24.147321 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.147045 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/310de27f-d65f-4c73-840a-df4b5f6aed8b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/310de27f-d65f-4c73-840a-df4b5f6aed8b-istio-data\") pod \"310de27f-d65f-4c73-840a-df4b5f6aed8b\" (UID: \"310de27f-d65f-4c73-840a-df4b5f6aed8b\") " Oct 30 09:56:24.147321 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.147123 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/821c88fc-b72b-4104-97f3-e3528428498e-kube-api-access-9rr2x\" (UniqueName: \"kubernetes.io/projected/821c88fc-b72b-4104-97f3-e3528428498e-kube-api-access-9rr2x\") pod \"821c88fc-b72b-4104-97f3-e3528428498e\" (UID: \"821c88fc-b72b-4104-97f3-e3528428498e\") " Oct 30 09:56:24.147321 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.147158 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/310de27f-d65f-4c73-840a-df4b5f6aed8b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/310de27f-d65f-4c73-840a-df4b5f6aed8b-istio-envoy\") pod \"310de27f-d65f-4c73-840a-df4b5f6aed8b\" (UID: \"310de27f-d65f-4c73-840a-df4b5f6aed8b\") " Oct 30 09:56:24.147321 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.147214 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/310de27f-d65f-4c73-840a-df4b5f6aed8b-kube-api-access-gttjf\" (UniqueName: \"kubernetes.io/projected/310de27f-d65f-4c73-840a-df4b5f6aed8b-kube-api-access-gttjf\") pod \"310de27f-d65f-4c73-840a-df4b5f6aed8b\" (UID: \"310de27f-d65f-4c73-840a-df4b5f6aed8b\") " Oct 30 09:56:24.147321 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.147262 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/310de27f-d65f-4c73-840a-df4b5f6aed8b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/310de27f-d65f-4c73-840a-df4b5f6aed8b-workload-certs\") pod \"310de27f-d65f-4c73-840a-df4b5f6aed8b\" (UID: \"310de27f-d65f-4c73-840a-df4b5f6aed8b\") " Oct 30 09:56:24.147321 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.147300 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/310de27f-d65f-4c73-840a-df4b5f6aed8b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/310de27f-d65f-4c73-840a-df4b5f6aed8b-istio-podinfo\") pod \"310de27f-d65f-4c73-840a-df4b5f6aed8b\" (UID: \"310de27f-d65f-4c73-840a-df4b5f6aed8b\") " Oct 30 09:56:24.147706 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.147350 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/310de27f-d65f-4c73-840a-df4b5f6aed8b-istio-token\" (UniqueName: \"kubernetes.io/projected/310de27f-d65f-4c73-840a-df4b5f6aed8b-istio-token\") pod \"310de27f-d65f-4c73-840a-df4b5f6aed8b\" (UID: \"310de27f-d65f-4c73-840a-df4b5f6aed8b\") " Oct 30 09:56:24.147706 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.147426 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/821c88fc-b72b-4104-97f3-e3528428498e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/821c88fc-b72b-4104-97f3-e3528428498e-credential-socket\") pod \"821c88fc-b72b-4104-97f3-e3528428498e\" (UID: \"821c88fc-b72b-4104-97f3-e3528428498e\") " Oct 30 09:56:24.147706 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.147467 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/310de27f-d65f-4c73-840a-df4b5f6aed8b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/310de27f-d65f-4c73-840a-df4b5f6aed8b-istiod-ca-cert\") pod \"310de27f-d65f-4c73-840a-df4b5f6aed8b\" (UID: \"310de27f-d65f-4c73-840a-df4b5f6aed8b\") " Oct 30 09:56:24.147706 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.147508 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/821c88fc-b72b-4104-97f3-e3528428498e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/821c88fc-b72b-4104-97f3-e3528428498e-istio-podinfo\") pod \"821c88fc-b72b-4104-97f3-e3528428498e\" (UID: \"821c88fc-b72b-4104-97f3-e3528428498e\") " Oct 30 09:56:24.147706 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.147571 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/821c88fc-b72b-4104-97f3-e3528428498e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/821c88fc-b72b-4104-97f3-e3528428498e-workload-certs\") pod \"821c88fc-b72b-4104-97f3-e3528428498e\" (UID: \"821c88fc-b72b-4104-97f3-e3528428498e\") " Oct 30 09:56:24.147706 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.147610 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/310de27f-d65f-4c73-840a-df4b5f6aed8b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/310de27f-d65f-4c73-840a-df4b5f6aed8b-workload-socket\") pod \"310de27f-d65f-4c73-840a-df4b5f6aed8b\" (UID: \"310de27f-d65f-4c73-840a-df4b5f6aed8b\") " Oct 30 09:56:24.148502 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.148470 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/310de27f-d65f-4c73-840a-df4b5f6aed8b-workload-socket" pod "310de27f-d65f-4c73-840a-df4b5f6aed8b" (UID: "310de27f-d65f-4c73-840a-df4b5f6aed8b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:56:24.149348 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.148632 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/821c88fc-b72b-4104-97f3-e3528428498e-workload-socket" pod "821c88fc-b72b-4104-97f3-e3528428498e" (UID: "821c88fc-b72b-4104-97f3-e3528428498e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:56:24.149348 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.148713 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/821c88fc-b72b-4104-97f3-e3528428498e-istio-data" pod "821c88fc-b72b-4104-97f3-e3528428498e" (UID: "821c88fc-b72b-4104-97f3-e3528428498e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:56:24.149585 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.148762 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/310de27f-d65f-4c73-840a-df4b5f6aed8b-workload-certs" pod "310de27f-d65f-4c73-840a-df4b5f6aed8b" (UID: "310de27f-d65f-4c73-840a-df4b5f6aed8b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:56:24.149585 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.149509 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/310de27f-d65f-4c73-840a-df4b5f6aed8b-credential-socket" pod "310de27f-d65f-4c73-840a-df4b5f6aed8b" (UID: "310de27f-d65f-4c73-840a-df4b5f6aed8b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:56:24.149711 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.149649 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/310de27f-d65f-4c73-840a-df4b5f6aed8b-istiod-ca-cert" pod "310de27f-d65f-4c73-840a-df4b5f6aed8b" (UID: "310de27f-d65f-4c73-840a-df4b5f6aed8b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:56:24.149773 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.148884 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/821c88fc-b72b-4104-97f3-e3528428498e-credential-socket" pod "821c88fc-b72b-4104-97f3-e3528428498e" (UID: "821c88fc-b72b-4104-97f3-e3528428498e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:56:24.149773 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.149061 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/821c88fc-b72b-4104-97f3-e3528428498e-workload-certs" pod "821c88fc-b72b-4104-97f3-e3528428498e" (UID: "821c88fc-b72b-4104-97f3-e3528428498e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:56:24.149773 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.148984 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/821c88fc-b72b-4104-97f3-e3528428498e-istiod-ca-cert" pod "821c88fc-b72b-4104-97f3-e3528428498e" (UID: "821c88fc-b72b-4104-97f3-e3528428498e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:56:24.149773 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.149235 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/310de27f-d65f-4c73-840a-df4b5f6aed8b-istio-data" pod "310de27f-d65f-4c73-840a-df4b5f6aed8b" (UID: "310de27f-d65f-4c73-840a-df4b5f6aed8b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:56:24.151313 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.151268 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/821c88fc-b72b-4104-97f3-e3528428498e-istio-envoy" pod "821c88fc-b72b-4104-97f3-e3528428498e" (UID: "821c88fc-b72b-4104-97f3-e3528428498e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:56:24.151611 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.151578 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/310de27f-d65f-4c73-840a-df4b5f6aed8b-kube-api-access-gttjf" pod "310de27f-d65f-4c73-840a-df4b5f6aed8b" (UID: "310de27f-d65f-4c73-840a-df4b5f6aed8b"). InnerVolumeSpecName "kube-api-access-gttjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:56:24.151719 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.151643 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/310de27f-d65f-4c73-840a-df4b5f6aed8b-istio-podinfo" pod "310de27f-d65f-4c73-840a-df4b5f6aed8b" (UID: "310de27f-d65f-4c73-840a-df4b5f6aed8b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:56:24.151779 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.151736 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/821c88fc-b72b-4104-97f3-e3528428498e-istio-token" pod "821c88fc-b72b-4104-97f3-e3528428498e" (UID: "821c88fc-b72b-4104-97f3-e3528428498e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:56:24.151968 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.151945 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/821c88fc-b72b-4104-97f3-e3528428498e-kube-api-access-9rr2x" pod "821c88fc-b72b-4104-97f3-e3528428498e" (UID: "821c88fc-b72b-4104-97f3-e3528428498e"). InnerVolumeSpecName "kube-api-access-9rr2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:56:24.152147 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.152128 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/310de27f-d65f-4c73-840a-df4b5f6aed8b-istio-token" pod "310de27f-d65f-4c73-840a-df4b5f6aed8b" (UID: "310de27f-d65f-4c73-840a-df4b5f6aed8b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:56:24.152207 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.152126 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/310de27f-d65f-4c73-840a-df4b5f6aed8b-istio-envoy" pod "310de27f-d65f-4c73-840a-df4b5f6aed8b" (UID: "310de27f-d65f-4c73-840a-df4b5f6aed8b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:56:24.152207 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.152157 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/821c88fc-b72b-4104-97f3-e3528428498e-istio-podinfo" pod "821c88fc-b72b-4104-97f3-e3528428498e" (UID: "821c88fc-b72b-4104-97f3-e3528428498e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:56:24.250038 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.249995 4562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/821c88fc-b72b-4104-97f3-e3528428498e-istio-token\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:56:24.250038 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.250029 4562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/821c88fc-b72b-4104-97f3-e3528428498e-istio-data\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:56:24.250038 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.250040 4562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/821c88fc-b72b-4104-97f3-e3528428498e-istiod-ca-cert\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:56:24.250038 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.250049 4562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/821c88fc-b72b-4104-97f3-e3528428498e-workload-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:56:24.250446 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.250057 4562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/821c88fc-b72b-4104-97f3-e3528428498e-istio-envoy\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:56:24.250446 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.250066 4562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/310de27f-d65f-4c73-840a-df4b5f6aed8b-credential-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:56:24.250446 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.250076 4562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/310de27f-d65f-4c73-840a-df4b5f6aed8b-istio-data\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:56:24.250446 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.250085 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rr2x\" (UniqueName: \"kubernetes.io/projected/821c88fc-b72b-4104-97f3-e3528428498e-kube-api-access-9rr2x\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:56:24.250446 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.250094 4562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/310de27f-d65f-4c73-840a-df4b5f6aed8b-istio-envoy\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:56:24.250446 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.250103 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gttjf\" (UniqueName: \"kubernetes.io/projected/310de27f-d65f-4c73-840a-df4b5f6aed8b-kube-api-access-gttjf\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:56:24.250446 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.250124 4562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/310de27f-d65f-4c73-840a-df4b5f6aed8b-workload-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:56:24.250446 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.250132 4562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/310de27f-d65f-4c73-840a-df4b5f6aed8b-istio-podinfo\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:56:24.250446 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.250140 4562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/310de27f-d65f-4c73-840a-df4b5f6aed8b-istio-token\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:56:24.250446 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.250147 4562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/821c88fc-b72b-4104-97f3-e3528428498e-credential-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:56:24.250446 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.250155 4562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/310de27f-d65f-4c73-840a-df4b5f6aed8b-istiod-ca-cert\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:56:24.250446 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.250165 4562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/821c88fc-b72b-4104-97f3-e3528428498e-istio-podinfo\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:56:24.250446 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.250173 4562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/821c88fc-b72b-4104-97f3-e3528428498e-workload-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:56:24.250446 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.250181 4562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/310de27f-d65f-4c73-840a-df4b5f6aed8b-workload-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:56:24.296736 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.296700 4562 generic.go:356] "Generic (PLEG): container finished" podID="310de27f-d65f-4c73-840a-df4b5f6aed8b" containerID="d8892b8b3bcc9a80af360f1d79970a7a39fd4141ac28bba34b334f125530bbc6" exitCode=0 Oct 30 09:56:24.296941 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.296757 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr" Oct 30 09:56:24.296941 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.296779 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr" event={"ID":"310de27f-d65f-4c73-840a-df4b5f6aed8b","Type":"ContainerDied","Data":"d8892b8b3bcc9a80af360f1d79970a7a39fd4141ac28bba34b334f125530bbc6"} Oct 30 09:56:24.296941 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.296830 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr" event={"ID":"310de27f-d65f-4c73-840a-df4b5f6aed8b","Type":"ContainerDied","Data":"24111d419a0b7c991fa69badd17d365a23bda3446a53c90f013d4b7774d04215"} Oct 30 09:56:24.296941 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.296851 4562 scope.go:122] "RemoveContainer" containerID="d8892b8b3bcc9a80af360f1d79970a7a39fd4141ac28bba34b334f125530bbc6" Oct 30 09:56:24.298994 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.298914 4562 generic.go:356] "Generic (PLEG): container finished" podID="821c88fc-b72b-4104-97f3-e3528428498e" containerID="c21f8960ca996a0e9b070c9a4bcd15ff37dfd7532869991b5332e577f3cc37e4" exitCode=0 Oct 30 09:56:24.298994 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.298948 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67" event={"ID":"821c88fc-b72b-4104-97f3-e3528428498e","Type":"ContainerDied","Data":"c21f8960ca996a0e9b070c9a4bcd15ff37dfd7532869991b5332e577f3cc37e4"} Oct 30 09:56:24.298994 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.298983 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67" Oct 30 09:56:24.299255 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.299018 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67" event={"ID":"821c88fc-b72b-4104-97f3-e3528428498e","Type":"ContainerDied","Data":"aeca570386888e4772d4c5095fa92650e30fd03967abe132e4cf436ea9e67e50"} Oct 30 09:56:24.311021 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.310815 4562 scope.go:122] "RemoveContainer" containerID="d8892b8b3bcc9a80af360f1d79970a7a39fd4141ac28bba34b334f125530bbc6" Oct 30 09:56:24.311261 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:56:24.311232 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8892b8b3bcc9a80af360f1d79970a7a39fd4141ac28bba34b334f125530bbc6\": container with ID starting with d8892b8b3bcc9a80af360f1d79970a7a39fd4141ac28bba34b334f125530bbc6 not found: ID does not exist" containerID="d8892b8b3bcc9a80af360f1d79970a7a39fd4141ac28bba34b334f125530bbc6" Oct 30 09:56:24.311455 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.311427 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8892b8b3bcc9a80af360f1d79970a7a39fd4141ac28bba34b334f125530bbc6"} err="failed to get container status \"d8892b8b3bcc9a80af360f1d79970a7a39fd4141ac28bba34b334f125530bbc6\": rpc error: code = NotFound desc = could not find container \"d8892b8b3bcc9a80af360f1d79970a7a39fd4141ac28bba34b334f125530bbc6\": container with ID starting with d8892b8b3bcc9a80af360f1d79970a7a39fd4141ac28bba34b334f125530bbc6 not found: ID does not exist" Oct 30 09:56:24.311612 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.311597 4562 scope.go:122] "RemoveContainer" containerID="c21f8960ca996a0e9b070c9a4bcd15ff37dfd7532869991b5332e577f3cc37e4" Oct 30 09:56:24.324786 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.324760 4562 scope.go:122] "RemoveContainer" containerID="adb2c7f68c8ef0c8e496656d9db26ad4174a5e4dc87e308101da44dc9998967b" Oct 30 09:56:24.341890 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.339438 4562 scope.go:122] "RemoveContainer" containerID="c21f8960ca996a0e9b070c9a4bcd15ff37dfd7532869991b5332e577f3cc37e4" Oct 30 09:56:24.341890 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:56:24.339943 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c21f8960ca996a0e9b070c9a4bcd15ff37dfd7532869991b5332e577f3cc37e4\": container with ID starting with c21f8960ca996a0e9b070c9a4bcd15ff37dfd7532869991b5332e577f3cc37e4 not found: ID does not exist" containerID="c21f8960ca996a0e9b070c9a4bcd15ff37dfd7532869991b5332e577f3cc37e4" Oct 30 09:56:24.341890 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.339985 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c21f8960ca996a0e9b070c9a4bcd15ff37dfd7532869991b5332e577f3cc37e4"} err="failed to get container status \"c21f8960ca996a0e9b070c9a4bcd15ff37dfd7532869991b5332e577f3cc37e4\": rpc error: code = NotFound desc = could not find container \"c21f8960ca996a0e9b070c9a4bcd15ff37dfd7532869991b5332e577f3cc37e4\": container with ID starting with c21f8960ca996a0e9b070c9a4bcd15ff37dfd7532869991b5332e577f3cc37e4 not found: ID does not exist" Oct 30 09:56:24.341890 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.340010 4562 scope.go:122] "RemoveContainer" containerID="adb2c7f68c8ef0c8e496656d9db26ad4174a5e4dc87e308101da44dc9998967b" Oct 30 09:56:24.341890 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:56:24.340329 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adb2c7f68c8ef0c8e496656d9db26ad4174a5e4dc87e308101da44dc9998967b\": container with ID starting with adb2c7f68c8ef0c8e496656d9db26ad4174a5e4dc87e308101da44dc9998967b not found: ID does not exist" containerID="adb2c7f68c8ef0c8e496656d9db26ad4174a5e4dc87e308101da44dc9998967b" Oct 30 09:56:24.341890 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.340370 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adb2c7f68c8ef0c8e496656d9db26ad4174a5e4dc87e308101da44dc9998967b"} err="failed to get container status \"adb2c7f68c8ef0c8e496656d9db26ad4174a5e4dc87e308101da44dc9998967b\": rpc error: code = NotFound desc = could not find container \"adb2c7f68c8ef0c8e496656d9db26ad4174a5e4dc87e308101da44dc9998967b\": container with ID starting with adb2c7f68c8ef0c8e496656d9db26ad4174a5e4dc87e308101da44dc9998967b not found: ID does not exist" Oct 30 09:56:24.367432 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.367396 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67"] Oct 30 09:56:24.375933 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.375900 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-aa2fdf-openshift-default-7585c7b855-cbr67"] Oct 30 09:56:24.410675 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.410586 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr"] Oct 30 09:56:24.418277 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:24.418248 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb1a2a-openshift-default-6bf8c78cbf-r6cgr"] Oct 30 09:56:25.875714 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:25.875672 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="310de27f-d65f-4c73-840a-df4b5f6aed8b" path="/var/lib/kubelet/pods/310de27f-d65f-4c73-840a-df4b5f6aed8b/volumes" Oct 30 09:56:25.876319 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:25.876300 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="821c88fc-b72b-4104-97f3-e3528428498e" path="/var/lib/kubelet/pods/821c88fc-b72b-4104-97f3-e3528428498e/volumes" Oct 30 09:56:28.598889 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:56:28.598851 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:56:29.659618 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:29.659577 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-c5447d-7bb465d9f8-89mpq"] Oct 30 09:56:30.502359 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:30.502306 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0dc148-7b74d7b55-zzv9d"] Oct 30 09:56:32.177941 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:32.177903 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh"] Oct 30 09:56:32.178474 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:32.178097 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh" podUID="d7f74510-23f8-4f0b-af8e-6c9861215203" containerName="istio-proxy" containerID="cri-o://d19181cb5caa03a0eca73cae1ddd5f62d68d096a3eaed7827dfec612c8f21545" gracePeriod=30 Oct 30 09:56:34.660147 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:34.660101 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-c5447d-7bb465d9f8-89mpq" podUID="3e125f79-421e-4ac0-8f37-8bbd5f430c9a" containerName="payload-processing" containerID="cri-o://a5a857705f309283aa0246600a32e3f0faac2040dbb6855e9854236da8ff5e6d" gracePeriod=25 Oct 30 09:56:35.503500 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:35.503429 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0dc148-7b74d7b55-zzv9d" podUID="fa734120-4c82-426f-947c-1a36330750ba" containerName="payload-pre-processing" containerID="cri-o://88e862975b14935fa9dc419ea24f2e4f04494605d9049bbc0ddb76838445f0c5" gracePeriod=25 Oct 30 09:56:37.459371 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:37.459017 4562 generic.go:356] "Generic (PLEG): container finished" podID="d7f74510-23f8-4f0b-af8e-6c9861215203" containerID="d19181cb5caa03a0eca73cae1ddd5f62d68d096a3eaed7827dfec612c8f21545" exitCode=0 Oct 30 09:56:37.459371 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:37.459228 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh" event={"ID":"d7f74510-23f8-4f0b-af8e-6c9861215203","Type":"ContainerDied","Data":"d19181cb5caa03a0eca73cae1ddd5f62d68d096a3eaed7827dfec612c8f21545"} Oct 30 09:56:37.499769 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:37.499740 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh" Oct 30 09:56:37.640186 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:37.640153 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7f74510-23f8-4f0b-af8e-6c9861215203-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7f74510-23f8-4f0b-af8e-6c9861215203-istio-data\") pod \"d7f74510-23f8-4f0b-af8e-6c9861215203\" (UID: \"d7f74510-23f8-4f0b-af8e-6c9861215203\") " Oct 30 09:56:37.640360 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:37.640239 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7f74510-23f8-4f0b-af8e-6c9861215203-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7f74510-23f8-4f0b-af8e-6c9861215203-istio-envoy\") pod \"d7f74510-23f8-4f0b-af8e-6c9861215203\" (UID: \"d7f74510-23f8-4f0b-af8e-6c9861215203\") " Oct 30 09:56:37.640360 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:37.640290 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7f74510-23f8-4f0b-af8e-6c9861215203-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f74510-23f8-4f0b-af8e-6c9861215203-workload-socket\") pod \"d7f74510-23f8-4f0b-af8e-6c9861215203\" (UID: \"d7f74510-23f8-4f0b-af8e-6c9861215203\") " Oct 30 09:56:37.640360 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:37.640333 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7f74510-23f8-4f0b-af8e-6c9861215203-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7f74510-23f8-4f0b-af8e-6c9861215203-workload-certs\") pod \"d7f74510-23f8-4f0b-af8e-6c9861215203\" (UID: \"d7f74510-23f8-4f0b-af8e-6c9861215203\") " Oct 30 09:56:37.640585 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:37.640370 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7f74510-23f8-4f0b-af8e-6c9861215203-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7f74510-23f8-4f0b-af8e-6c9861215203-istiod-ca-cert\") pod \"d7f74510-23f8-4f0b-af8e-6c9861215203\" (UID: \"d7f74510-23f8-4f0b-af8e-6c9861215203\") " Oct 30 09:56:37.640585 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:37.640462 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d7f74510-23f8-4f0b-af8e-6c9861215203-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7f74510-23f8-4f0b-af8e-6c9861215203-istio-podinfo\") pod \"d7f74510-23f8-4f0b-af8e-6c9861215203\" (UID: \"d7f74510-23f8-4f0b-af8e-6c9861215203\") " Oct 30 09:56:37.640585 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:37.640492 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7f74510-23f8-4f0b-af8e-6c9861215203-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f74510-23f8-4f0b-af8e-6c9861215203-credential-socket\") pod \"d7f74510-23f8-4f0b-af8e-6c9861215203\" (UID: \"d7f74510-23f8-4f0b-af8e-6c9861215203\") " Oct 30 09:56:37.640585 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:37.640578 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7f74510-23f8-4f0b-af8e-6c9861215203-kube-api-access-9wl4g\" (UniqueName: \"kubernetes.io/projected/d7f74510-23f8-4f0b-af8e-6c9861215203-kube-api-access-9wl4g\") pod \"d7f74510-23f8-4f0b-af8e-6c9861215203\" (UID: \"d7f74510-23f8-4f0b-af8e-6c9861215203\") " Oct 30 09:56:37.640807 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:37.640486 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7f74510-23f8-4f0b-af8e-6c9861215203-istio-data" pod "d7f74510-23f8-4f0b-af8e-6c9861215203" (UID: "d7f74510-23f8-4f0b-af8e-6c9861215203"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:56:37.640807 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:37.640702 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7f74510-23f8-4f0b-af8e-6c9861215203-istio-token\" (UniqueName: \"kubernetes.io/projected/d7f74510-23f8-4f0b-af8e-6c9861215203-istio-token\") pod \"d7f74510-23f8-4f0b-af8e-6c9861215203\" (UID: \"d7f74510-23f8-4f0b-af8e-6c9861215203\") " Oct 30 09:56:37.640807 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:37.640736 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7f74510-23f8-4f0b-af8e-6c9861215203-workload-certs" pod "d7f74510-23f8-4f0b-af8e-6c9861215203" (UID: "d7f74510-23f8-4f0b-af8e-6c9861215203"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:56:37.640807 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:37.640750 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7f74510-23f8-4f0b-af8e-6c9861215203-workload-socket" pod "d7f74510-23f8-4f0b-af8e-6c9861215203" (UID: "d7f74510-23f8-4f0b-af8e-6c9861215203"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:56:37.641036 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:37.640845 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7f74510-23f8-4f0b-af8e-6c9861215203-credential-socket" pod "d7f74510-23f8-4f0b-af8e-6c9861215203" (UID: "d7f74510-23f8-4f0b-af8e-6c9861215203"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:56:37.641036 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:37.640913 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7f74510-23f8-4f0b-af8e-6c9861215203-istiod-ca-cert" pod "d7f74510-23f8-4f0b-af8e-6c9861215203" (UID: "d7f74510-23f8-4f0b-af8e-6c9861215203"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:56:37.641413 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:37.641364 4562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7f74510-23f8-4f0b-af8e-6c9861215203-istio-data\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:56:37.641531 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:37.641421 4562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f74510-23f8-4f0b-af8e-6c9861215203-workload-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:56:37.641531 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:37.641441 4562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7f74510-23f8-4f0b-af8e-6c9861215203-workload-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:56:37.641531 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:37.641455 4562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7f74510-23f8-4f0b-af8e-6c9861215203-istiod-ca-cert\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:56:37.641531 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:37.641468 4562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f74510-23f8-4f0b-af8e-6c9861215203-credential-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:56:37.642660 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:37.642634 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7f74510-23f8-4f0b-af8e-6c9861215203-istio-envoy" pod "d7f74510-23f8-4f0b-af8e-6c9861215203" (UID: "d7f74510-23f8-4f0b-af8e-6c9861215203"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:56:37.642780 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:37.642761 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7f74510-23f8-4f0b-af8e-6c9861215203-istio-podinfo" pod "d7f74510-23f8-4f0b-af8e-6c9861215203" (UID: "d7f74510-23f8-4f0b-af8e-6c9861215203"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:56:37.643059 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:37.643041 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7f74510-23f8-4f0b-af8e-6c9861215203-istio-token" pod "d7f74510-23f8-4f0b-af8e-6c9861215203" (UID: "d7f74510-23f8-4f0b-af8e-6c9861215203"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:56:37.643165 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:37.643056 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7f74510-23f8-4f0b-af8e-6c9861215203-kube-api-access-9wl4g" pod "d7f74510-23f8-4f0b-af8e-6c9861215203" (UID: "d7f74510-23f8-4f0b-af8e-6c9861215203"). InnerVolumeSpecName "kube-api-access-9wl4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:56:37.743091 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:37.742998 4562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7f74510-23f8-4f0b-af8e-6c9861215203-istio-envoy\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:56:37.743091 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:37.743031 4562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7f74510-23f8-4f0b-af8e-6c9861215203-istio-podinfo\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:56:37.743091 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:37.743042 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wl4g\" (UniqueName: \"kubernetes.io/projected/d7f74510-23f8-4f0b-af8e-6c9861215203-kube-api-access-9wl4g\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:56:37.743091 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:37.743053 4562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7f74510-23f8-4f0b-af8e-6c9861215203-istio-token\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:56:38.472669 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:38.472634 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh" Oct 30 09:56:38.472669 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:38.472649 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh" event={"ID":"d7f74510-23f8-4f0b-af8e-6c9861215203","Type":"ContainerDied","Data":"14d4851afc2e113871d21b4ac0c00de9a6d70016014dedf28437f4bb5cc179bf"} Oct 30 09:56:38.473295 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:38.472724 4562 scope.go:122] "RemoveContainer" containerID="d19181cb5caa03a0eca73cae1ddd5f62d68d096a3eaed7827dfec612c8f21545" Oct 30 09:56:38.570408 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:38.568110 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh"] Oct 30 09:56:38.577495 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:38.574022 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-304459-openshift-default-6885974d55-6v5bh"] Oct 30 09:56:39.147226 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:39.147186 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-c5447d/distinct-2-w1-c5447d-kserve-676749d548-xlmsf" podUID="c9ff897f-c520-4d0f-8820-e185b1d47dac" containerName="main" containerID="cri-o://7e3c502cda9d264cb042b3daa3853f5bb421e93190b324b1a7429fba1287a936" gracePeriod=45 Oct 30 09:56:39.438494 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:39.438466 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c5447d/distinct-2-w1-c5447d-kserve-676749d548-xlmsf" Oct 30 09:56:39.477974 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:39.477936 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9ff897f-c520-4d0f-8820-e185b1d47dac-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9ff897f-c520-4d0f-8820-e185b1d47dac-model-cache\") pod \"c9ff897f-c520-4d0f-8820-e185b1d47dac\" (UID: \"c9ff897f-c520-4d0f-8820-e185b1d47dac\") " Oct 30 09:56:39.478502 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:39.478076 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9ff897f-c520-4d0f-8820-e185b1d47dac-kube-api-access-vrn25\" (UniqueName: \"kubernetes.io/projected/c9ff897f-c520-4d0f-8820-e185b1d47dac-kube-api-access-vrn25\") pod \"c9ff897f-c520-4d0f-8820-e185b1d47dac\" (UID: \"c9ff897f-c520-4d0f-8820-e185b1d47dac\") " Oct 30 09:56:39.478502 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:39.478233 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9ff897f-c520-4d0f-8820-e185b1d47dac-tls-certs\" (UniqueName: \"kubernetes.io/secret/c9ff897f-c520-4d0f-8820-e185b1d47dac-tls-certs\") pod \"c9ff897f-c520-4d0f-8820-e185b1d47dac\" (UID: \"c9ff897f-c520-4d0f-8820-e185b1d47dac\") " Oct 30 09:56:39.478502 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:39.478272 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9ff897f-c520-4d0f-8820-e185b1d47dac-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9ff897f-c520-4d0f-8820-e185b1d47dac-dshm\") pod \"c9ff897f-c520-4d0f-8820-e185b1d47dac\" (UID: \"c9ff897f-c520-4d0f-8820-e185b1d47dac\") " Oct 30 09:56:39.478502 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:39.478289 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9ff897f-c520-4d0f-8820-e185b1d47dac-model-cache" pod "c9ff897f-c520-4d0f-8820-e185b1d47dac" (UID: "c9ff897f-c520-4d0f-8820-e185b1d47dac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:56:39.478502 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:39.478315 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9ff897f-c520-4d0f-8820-e185b1d47dac-home\" (UniqueName: \"kubernetes.io/empty-dir/c9ff897f-c520-4d0f-8820-e185b1d47dac-home\") pod \"c9ff897f-c520-4d0f-8820-e185b1d47dac\" (UID: \"c9ff897f-c520-4d0f-8820-e185b1d47dac\") " Oct 30 09:56:39.478803 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:39.478609 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9ff897f-c520-4d0f-8820-e185b1d47dac-home" pod "c9ff897f-c520-4d0f-8820-e185b1d47dac" (UID: "c9ff897f-c520-4d0f-8820-e185b1d47dac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:56:39.480058 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:39.479000 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9ff897f-c520-4d0f-8820-e185b1d47dac-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9ff897f-c520-4d0f-8820-e185b1d47dac-tmp-dir\") pod \"c9ff897f-c520-4d0f-8820-e185b1d47dac\" (UID: \"c9ff897f-c520-4d0f-8820-e185b1d47dac\") " Oct 30 09:56:39.480058 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:39.479656 4562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9ff897f-c520-4d0f-8820-e185b1d47dac-home\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:56:39.480058 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:39.479680 4562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9ff897f-c520-4d0f-8820-e185b1d47dac-model-cache\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:56:39.480058 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:39.479966 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9ff897f-c520-4d0f-8820-e185b1d47dac-tmp-dir" pod "c9ff897f-c520-4d0f-8820-e185b1d47dac" (UID: "c9ff897f-c520-4d0f-8820-e185b1d47dac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:56:39.481267 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:39.481242 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9ff897f-c520-4d0f-8820-e185b1d47dac-kube-api-access-vrn25" pod "c9ff897f-c520-4d0f-8820-e185b1d47dac" (UID: "c9ff897f-c520-4d0f-8820-e185b1d47dac"). InnerVolumeSpecName "kube-api-access-vrn25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:56:39.481644 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:39.481222 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9ff897f-c520-4d0f-8820-e185b1d47dac-dshm" pod "c9ff897f-c520-4d0f-8820-e185b1d47dac" (UID: "c9ff897f-c520-4d0f-8820-e185b1d47dac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:56:39.497439 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:39.483690 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9ff897f-c520-4d0f-8820-e185b1d47dac-tls-certs" pod "c9ff897f-c520-4d0f-8820-e185b1d47dac" (UID: "c9ff897f-c520-4d0f-8820-e185b1d47dac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:56:39.497439 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:39.486925 4562 generic.go:356] "Generic (PLEG): container finished" podID="c9ff897f-c520-4d0f-8820-e185b1d47dac" containerID="7e3c502cda9d264cb042b3daa3853f5bb421e93190b324b1a7429fba1287a936" exitCode=0 Oct 30 09:56:39.497439 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:39.487077 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c5447d/distinct-2-w1-c5447d-kserve-676749d548-xlmsf" event={"ID":"c9ff897f-c520-4d0f-8820-e185b1d47dac","Type":"ContainerDied","Data":"7e3c502cda9d264cb042b3daa3853f5bb421e93190b324b1a7429fba1287a936"} Oct 30 09:56:39.497439 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:39.487111 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c5447d/distinct-2-w1-c5447d-kserve-676749d548-xlmsf" event={"ID":"c9ff897f-c520-4d0f-8820-e185b1d47dac","Type":"ContainerDied","Data":"3da253e25b21f02545ef82e53cc6b4634ac1783ad3e5f397e6af1a58ffebb4a2"} Oct 30 09:56:39.497439 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:39.487133 4562 scope.go:122] "RemoveContainer" containerID="7e3c502cda9d264cb042b3daa3853f5bb421e93190b324b1a7429fba1287a936" Oct 30 09:56:39.497439 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:39.487245 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c5447d/distinct-2-w1-c5447d-kserve-676749d548-xlmsf" Oct 30 09:56:39.525014 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:39.524885 4562 scope.go:122] "RemoveContainer" containerID="7e3c502cda9d264cb042b3daa3853f5bb421e93190b324b1a7429fba1287a936" Oct 30 09:56:39.525351 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:56:39.525325 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e3c502cda9d264cb042b3daa3853f5bb421e93190b324b1a7429fba1287a936\": container with ID starting with 7e3c502cda9d264cb042b3daa3853f5bb421e93190b324b1a7429fba1287a936 not found: ID does not exist" containerID="7e3c502cda9d264cb042b3daa3853f5bb421e93190b324b1a7429fba1287a936" Oct 30 09:56:39.525488 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:39.525363 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e3c502cda9d264cb042b3daa3853f5bb421e93190b324b1a7429fba1287a936"} err="failed to get container status \"7e3c502cda9d264cb042b3daa3853f5bb421e93190b324b1a7429fba1287a936\": rpc error: code = NotFound desc = could not find container \"7e3c502cda9d264cb042b3daa3853f5bb421e93190b324b1a7429fba1287a936\": container with ID starting with 7e3c502cda9d264cb042b3daa3853f5bb421e93190b324b1a7429fba1287a936 not found: ID does not exist" Oct 30 09:56:39.563190 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:39.563153 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-c5447d/distinct-2-w1-c5447d-kserve-676749d548-xlmsf"] Oct 30 09:56:39.569137 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:39.569097 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-c5447d/distinct-2-w1-c5447d-kserve-676749d548-xlmsf"] Oct 30 09:56:39.580975 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:39.580941 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrn25\" (UniqueName: \"kubernetes.io/projected/c9ff897f-c520-4d0f-8820-e185b1d47dac-kube-api-access-vrn25\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:56:39.580975 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:39.580971 4562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9ff897f-c520-4d0f-8820-e185b1d47dac-tls-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:56:39.580975 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:39.580982 4562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9ff897f-c520-4d0f-8820-e185b1d47dac-dshm\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:56:39.581286 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:39.580990 4562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9ff897f-c520-4d0f-8820-e185b1d47dac-tmp-dir\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:56:39.877466 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:39.877435 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9ff897f-c520-4d0f-8820-e185b1d47dac" path="/var/lib/kubelet/pods/c9ff897f-c520-4d0f-8820-e185b1d47dac/volumes" Oct 30 09:56:39.877967 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:39.877952 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7f74510-23f8-4f0b-af8e-6c9861215203" path="/var/lib/kubelet/pods/d7f74510-23f8-4f0b-af8e-6c9861215203/volumes" Oct 30 09:56:41.379473 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:41.379428 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-c5447d-7bb465d9f8-89mpq" podUID="3e125f79-421e-4ac0-8f37-8bbd5f430c9a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:56:42.272437 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:42.272401 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0dc148-7b74d7b55-zzv9d" podUID="fa734120-4c82-426f-947c-1a36330750ba" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:56:43.199166 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:43.199128 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7a213d-689cb5bf76-9b72m"] Oct 30 09:56:43.231319 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:43.231277 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7a213d-65fd87589f-rr7jz"] Oct 30 09:56:48.200098 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:48.200034 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7a213d-689cb5bf76-9b72m" podUID="8db504e0-4d30-41b1-91cb-12daa3e2ac43" containerName="payload-processing" containerID="cri-o://edcdbff1f4f7329212ff0cd68aed0bac37e8b0a25ea633b113a37429a4fa77b3" gracePeriod=25 Oct 30 09:56:48.232003 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:48.231957 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7a213d-65fd87589f-rr7jz" podUID="c33c6c07-06d5-4069-9eb8-2c176a66ab2a" containerName="payload-pre-processing" containerID="cri-o://a62b796d94f3c63ac074a903113ce6f9ab0a573ae27eea5c1888390cdc2dabf4" gracePeriod=25 Oct 30 09:56:51.379022 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:51.378980 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-c5447d-7bb465d9f8-89mpq" podUID="3e125f79-421e-4ac0-8f37-8bbd5f430c9a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:56:52.272264 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:52.272221 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0dc148-7b74d7b55-zzv9d" podUID="fa734120-4c82-426f-947c-1a36330750ba" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:56:55.029867 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:55.029839 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-c5447d-7bb465d9f8-89mpq" Oct 30 09:56:55.043709 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:55.043680 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e125f79-421e-4ac0-8f37-8bbd5f430c9a-kube-api-access-lnr5c\" (UniqueName: \"kubernetes.io/projected/3e125f79-421e-4ac0-8f37-8bbd5f430c9a-kube-api-access-lnr5c\") pod \"3e125f79-421e-4ac0-8f37-8bbd5f430c9a\" (UID: \"3e125f79-421e-4ac0-8f37-8bbd5f430c9a\") " Oct 30 09:56:55.043905 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:55.043890 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e125f79-421e-4ac0-8f37-8bbd5f430c9a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e125f79-421e-4ac0-8f37-8bbd5f430c9a-plugins-config-volume\") pod \"3e125f79-421e-4ac0-8f37-8bbd5f430c9a\" (UID: \"3e125f79-421e-4ac0-8f37-8bbd5f430c9a\") " Oct 30 09:56:55.044243 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:55.044217 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e125f79-421e-4ac0-8f37-8bbd5f430c9a-plugins-config-volume" pod "3e125f79-421e-4ac0-8f37-8bbd5f430c9a" (UID: "3e125f79-421e-4ac0-8f37-8bbd5f430c9a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:56:55.044467 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:55.044453 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e125f79-421e-4ac0-8f37-8bbd5f430c9a-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:56:55.045923 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:55.045891 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e125f79-421e-4ac0-8f37-8bbd5f430c9a-kube-api-access-lnr5c" pod "3e125f79-421e-4ac0-8f37-8bbd5f430c9a" (UID: "3e125f79-421e-4ac0-8f37-8bbd5f430c9a"). InnerVolumeSpecName "kube-api-access-lnr5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:56:55.145487 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:55.145449 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnr5c\" (UniqueName: \"kubernetes.io/projected/3e125f79-421e-4ac0-8f37-8bbd5f430c9a-kube-api-access-lnr5c\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:56:55.639570 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:55.639527 4562 generic.go:356] "Generic (PLEG): container finished" podID="3e125f79-421e-4ac0-8f37-8bbd5f430c9a" containerID="a5a857705f309283aa0246600a32e3f0faac2040dbb6855e9854236da8ff5e6d" exitCode=0 Oct 30 09:56:55.639768 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:55.639641 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-c5447d-7bb465d9f8-89mpq" Oct 30 09:56:55.639768 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:55.639657 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-c5447d-7bb465d9f8-89mpq" event={"ID":"3e125f79-421e-4ac0-8f37-8bbd5f430c9a","Type":"ContainerDied","Data":"a5a857705f309283aa0246600a32e3f0faac2040dbb6855e9854236da8ff5e6d"} Oct 30 09:56:55.639768 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:55.639724 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-c5447d-7bb465d9f8-89mpq" event={"ID":"3e125f79-421e-4ac0-8f37-8bbd5f430c9a","Type":"ContainerDied","Data":"74b8bf0b1db1fdbb647be777f62b784951276e406f1c6be7934a2c3dbd3bffef"} Oct 30 09:56:55.639768 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:55.639758 4562 scope.go:122] "RemoveContainer" containerID="a5a857705f309283aa0246600a32e3f0faac2040dbb6855e9854236da8ff5e6d" Oct 30 09:56:55.641562 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:55.641535 4562 generic.go:356] "Generic (PLEG): container finished" podID="fa734120-4c82-426f-947c-1a36330750ba" containerID="88e862975b14935fa9dc419ea24f2e4f04494605d9049bbc0ddb76838445f0c5" exitCode=0 Oct 30 09:56:55.641704 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:55.641579 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0dc148-7b74d7b55-zzv9d" event={"ID":"fa734120-4c82-426f-947c-1a36330750ba","Type":"ContainerDied","Data":"88e862975b14935fa9dc419ea24f2e4f04494605d9049bbc0ddb76838445f0c5"} Oct 30 09:56:55.653945 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:55.653924 4562 scope.go:122] "RemoveContainer" containerID="a5a857705f309283aa0246600a32e3f0faac2040dbb6855e9854236da8ff5e6d" Oct 30 09:56:55.654417 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:56:55.654364 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5a857705f309283aa0246600a32e3f0faac2040dbb6855e9854236da8ff5e6d\": container with ID starting with a5a857705f309283aa0246600a32e3f0faac2040dbb6855e9854236da8ff5e6d not found: ID does not exist" containerID="a5a857705f309283aa0246600a32e3f0faac2040dbb6855e9854236da8ff5e6d" Oct 30 09:56:55.654518 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:55.654413 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5a857705f309283aa0246600a32e3f0faac2040dbb6855e9854236da8ff5e6d"} err="failed to get container status \"a5a857705f309283aa0246600a32e3f0faac2040dbb6855e9854236da8ff5e6d\": rpc error: code = NotFound desc = could not find container \"a5a857705f309283aa0246600a32e3f0faac2040dbb6855e9854236da8ff5e6d\": container with ID starting with a5a857705f309283aa0246600a32e3f0faac2040dbb6855e9854236da8ff5e6d not found: ID does not exist" Oct 30 09:56:55.667208 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:55.667173 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7a213d-689cb5bf76-9b72m" podUID="8db504e0-4d30-41b1-91cb-12daa3e2ac43" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:56:55.667355 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:55.667209 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7a213d-65fd87589f-rr7jz" podUID="c33c6c07-06d5-4069-9eb8-2c176a66ab2a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:56:55.698327 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:55.698222 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-c5447d-7bb465d9f8-89mpq"] Oct 30 09:56:55.701987 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:55.701957 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-c5447d-7bb465d9f8-89mpq"] Oct 30 09:56:55.833786 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:55.833751 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0dc148-7b74d7b55-zzv9d" Oct 30 09:56:55.867478 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:55.867446 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa734120-4c82-426f-947c-1a36330750ba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa734120-4c82-426f-947c-1a36330750ba-plugins-config-volume\") pod \"fa734120-4c82-426f-947c-1a36330750ba\" (UID: \"fa734120-4c82-426f-947c-1a36330750ba\") " Oct 30 09:56:55.867478 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:55.867493 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa734120-4c82-426f-947c-1a36330750ba-kube-api-access-rskd2\" (UniqueName: \"kubernetes.io/projected/fa734120-4c82-426f-947c-1a36330750ba-kube-api-access-rskd2\") pod \"fa734120-4c82-426f-947c-1a36330750ba\" (UID: \"fa734120-4c82-426f-947c-1a36330750ba\") " Oct 30 09:56:55.867930 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:55.867903 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa734120-4c82-426f-947c-1a36330750ba-plugins-config-volume" pod "fa734120-4c82-426f-947c-1a36330750ba" (UID: "fa734120-4c82-426f-947c-1a36330750ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:56:55.869940 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:55.869906 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa734120-4c82-426f-947c-1a36330750ba-kube-api-access-rskd2" pod "fa734120-4c82-426f-947c-1a36330750ba" (UID: "fa734120-4c82-426f-947c-1a36330750ba"). InnerVolumeSpecName "kube-api-access-rskd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:56:55.885431 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:55.880963 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e125f79-421e-4ac0-8f37-8bbd5f430c9a" path="/var/lib/kubelet/pods/3e125f79-421e-4ac0-8f37-8bbd5f430c9a/volumes" Oct 30 09:56:55.969353 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:55.969317 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa734120-4c82-426f-947c-1a36330750ba-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:56:55.969353 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:55.969347 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rskd2\" (UniqueName: \"kubernetes.io/projected/fa734120-4c82-426f-947c-1a36330750ba-kube-api-access-rskd2\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:56:56.650809 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:56.650764 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0dc148-7b74d7b55-zzv9d" Oct 30 09:56:56.651260 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:56.650839 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0dc148-7b74d7b55-zzv9d" event={"ID":"fa734120-4c82-426f-947c-1a36330750ba","Type":"ContainerDied","Data":"b73cd36deecb1149a8656f1068d944d662ae1082c3168a75ccdd5a7538413c65"} Oct 30 09:56:56.651260 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:56.650902 4562 scope.go:122] "RemoveContainer" containerID="88e862975b14935fa9dc419ea24f2e4f04494605d9049bbc0ddb76838445f0c5" Oct 30 09:56:56.704842 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:56.704803 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0dc148-7b74d7b55-zzv9d"] Oct 30 09:56:56.710990 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:56.710955 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0dc148-7b74d7b55-zzv9d"] Oct 30 09:56:57.876965 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:56:57.876931 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa734120-4c82-426f-947c-1a36330750ba" path="/var/lib/kubelet/pods/fa734120-4c82-426f-947c-1a36330750ba/volumes" Oct 30 09:56:58.598103 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:56:58.598070 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:57:00.720152 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:00.720109 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e3d2ff/distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55"] Oct 30 09:57:05.666959 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:05.666915 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7a213d-689cb5bf76-9b72m" podUID="8db504e0-4d30-41b1-91cb-12daa3e2ac43" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:57:05.666959 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:05.666949 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7a213d-65fd87589f-rr7jz" podUID="c33c6c07-06d5-4069-9eb8-2c176a66ab2a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:57:06.578457 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:06.578418 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e3d2ff-cbb67b84b-j59gb"] Oct 30 09:57:06.611260 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:06.611217 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e3d2ff-7ffc858867-5wpzf"] Oct 30 09:57:08.547038 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:08.547009 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7a213d-689cb5bf76-9b72m" Oct 30 09:57:08.550019 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:08.549992 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7a213d-65fd87589f-rr7jz" Oct 30 09:57:08.590652 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:08.590612 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c33c6c07-06d5-4069-9eb8-2c176a66ab2a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c33c6c07-06d5-4069-9eb8-2c176a66ab2a-plugins-config-volume\") pod \"c33c6c07-06d5-4069-9eb8-2c176a66ab2a\" (UID: \"c33c6c07-06d5-4069-9eb8-2c176a66ab2a\") " Oct 30 09:57:08.590862 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:08.590688 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8db504e0-4d30-41b1-91cb-12daa3e2ac43-kube-api-access-gvzvr\" (UniqueName: \"kubernetes.io/projected/8db504e0-4d30-41b1-91cb-12daa3e2ac43-kube-api-access-gvzvr\") pod \"8db504e0-4d30-41b1-91cb-12daa3e2ac43\" (UID: \"8db504e0-4d30-41b1-91cb-12daa3e2ac43\") " Oct 30 09:57:08.590862 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:08.590830 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c33c6c07-06d5-4069-9eb8-2c176a66ab2a-kube-api-access-8gfrr\" (UniqueName: \"kubernetes.io/projected/c33c6c07-06d5-4069-9eb8-2c176a66ab2a-kube-api-access-8gfrr\") pod \"c33c6c07-06d5-4069-9eb8-2c176a66ab2a\" (UID: \"c33c6c07-06d5-4069-9eb8-2c176a66ab2a\") " Oct 30 09:57:08.591130 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:08.591102 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c33c6c07-06d5-4069-9eb8-2c176a66ab2a-plugins-config-volume" pod "c33c6c07-06d5-4069-9eb8-2c176a66ab2a" (UID: "c33c6c07-06d5-4069-9eb8-2c176a66ab2a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:57:08.591478 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:08.591452 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8db504e0-4d30-41b1-91cb-12daa3e2ac43-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8db504e0-4d30-41b1-91cb-12daa3e2ac43-plugins-config-volume\") pod \"8db504e0-4d30-41b1-91cb-12daa3e2ac43\" (UID: \"8db504e0-4d30-41b1-91cb-12daa3e2ac43\") " Oct 30 09:57:08.591893 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:08.591864 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8db504e0-4d30-41b1-91cb-12daa3e2ac43-plugins-config-volume" pod "8db504e0-4d30-41b1-91cb-12daa3e2ac43" (UID: "8db504e0-4d30-41b1-91cb-12daa3e2ac43"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:57:08.592124 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:08.592106 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8db504e0-4d30-41b1-91cb-12daa3e2ac43-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:57:08.592173 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:08.592132 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c33c6c07-06d5-4069-9eb8-2c176a66ab2a-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:57:08.593190 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:08.593167 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c33c6c07-06d5-4069-9eb8-2c176a66ab2a-kube-api-access-8gfrr" pod "c33c6c07-06d5-4069-9eb8-2c176a66ab2a" (UID: "c33c6c07-06d5-4069-9eb8-2c176a66ab2a"). InnerVolumeSpecName "kube-api-access-8gfrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:57:08.593280 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:08.593189 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8db504e0-4d30-41b1-91cb-12daa3e2ac43-kube-api-access-gvzvr" pod "8db504e0-4d30-41b1-91cb-12daa3e2ac43" (UID: "8db504e0-4d30-41b1-91cb-12daa3e2ac43"). InnerVolumeSpecName "kube-api-access-gvzvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:57:08.693944 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:08.693910 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvzvr\" (UniqueName: \"kubernetes.io/projected/8db504e0-4d30-41b1-91cb-12daa3e2ac43-kube-api-access-gvzvr\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:57:08.693944 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:08.693939 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gfrr\" (UniqueName: \"kubernetes.io/projected/c33c6c07-06d5-4069-9eb8-2c176a66ab2a-kube-api-access-8gfrr\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:57:08.795602 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:08.795516 4562 generic.go:356] "Generic (PLEG): container finished" podID="c33c6c07-06d5-4069-9eb8-2c176a66ab2a" containerID="a62b796d94f3c63ac074a903113ce6f9ab0a573ae27eea5c1888390cdc2dabf4" exitCode=0 Oct 30 09:57:08.795602 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:08.795559 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7a213d-65fd87589f-rr7jz" Oct 30 09:57:08.795857 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:08.795606 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7a213d-65fd87589f-rr7jz" event={"ID":"c33c6c07-06d5-4069-9eb8-2c176a66ab2a","Type":"ContainerDied","Data":"a62b796d94f3c63ac074a903113ce6f9ab0a573ae27eea5c1888390cdc2dabf4"} Oct 30 09:57:08.795857 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:08.795665 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7a213d-65fd87589f-rr7jz" event={"ID":"c33c6c07-06d5-4069-9eb8-2c176a66ab2a","Type":"ContainerDied","Data":"e8992c5d7548b1d7a42664749d7f2a863206fa2fc4906e27e2a0e419a428d882"} Oct 30 09:57:08.795857 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:08.795695 4562 scope.go:122] "RemoveContainer" containerID="a62b796d94f3c63ac074a903113ce6f9ab0a573ae27eea5c1888390cdc2dabf4" Oct 30 09:57:08.797133 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:08.797113 4562 generic.go:356] "Generic (PLEG): container finished" podID="8db504e0-4d30-41b1-91cb-12daa3e2ac43" containerID="edcdbff1f4f7329212ff0cd68aed0bac37e8b0a25ea633b113a37429a4fa77b3" exitCode=0 Oct 30 09:57:08.797268 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:08.797152 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7a213d-689cb5bf76-9b72m" Oct 30 09:57:08.797268 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:08.797174 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7a213d-689cb5bf76-9b72m" event={"ID":"8db504e0-4d30-41b1-91cb-12daa3e2ac43","Type":"ContainerDied","Data":"edcdbff1f4f7329212ff0cd68aed0bac37e8b0a25ea633b113a37429a4fa77b3"} Oct 30 09:57:08.797268 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:08.797209 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7a213d-689cb5bf76-9b72m" event={"ID":"8db504e0-4d30-41b1-91cb-12daa3e2ac43","Type":"ContainerDied","Data":"0d20b53da21cd8c69e65cd01b5fcf87a7f8ee7684419845f096ccdfcae856aff"} Oct 30 09:57:08.808360 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:08.808342 4562 scope.go:122] "RemoveContainer" containerID="a62b796d94f3c63ac074a903113ce6f9ab0a573ae27eea5c1888390cdc2dabf4" Oct 30 09:57:08.808721 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:57:08.808692 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a62b796d94f3c63ac074a903113ce6f9ab0a573ae27eea5c1888390cdc2dabf4\": container with ID starting with a62b796d94f3c63ac074a903113ce6f9ab0a573ae27eea5c1888390cdc2dabf4 not found: ID does not exist" containerID="a62b796d94f3c63ac074a903113ce6f9ab0a573ae27eea5c1888390cdc2dabf4" Oct 30 09:57:08.808839 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:08.808730 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a62b796d94f3c63ac074a903113ce6f9ab0a573ae27eea5c1888390cdc2dabf4"} err="failed to get container status \"a62b796d94f3c63ac074a903113ce6f9ab0a573ae27eea5c1888390cdc2dabf4\": rpc error: code = NotFound desc = could not find container \"a62b796d94f3c63ac074a903113ce6f9ab0a573ae27eea5c1888390cdc2dabf4\": container with ID starting with a62b796d94f3c63ac074a903113ce6f9ab0a573ae27eea5c1888390cdc2dabf4 not found: ID does not exist" Oct 30 09:57:08.808839 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:08.808756 4562 scope.go:122] "RemoveContainer" containerID="edcdbff1f4f7329212ff0cd68aed0bac37e8b0a25ea633b113a37429a4fa77b3" Oct 30 09:57:08.819363 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:08.819338 4562 scope.go:122] "RemoveContainer" containerID="edcdbff1f4f7329212ff0cd68aed0bac37e8b0a25ea633b113a37429a4fa77b3" Oct 30 09:57:08.819721 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:57:08.819700 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edcdbff1f4f7329212ff0cd68aed0bac37e8b0a25ea633b113a37429a4fa77b3\": container with ID starting with edcdbff1f4f7329212ff0cd68aed0bac37e8b0a25ea633b113a37429a4fa77b3 not found: ID does not exist" containerID="edcdbff1f4f7329212ff0cd68aed0bac37e8b0a25ea633b113a37429a4fa77b3" Oct 30 09:57:08.819797 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:08.819730 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edcdbff1f4f7329212ff0cd68aed0bac37e8b0a25ea633b113a37429a4fa77b3"} err="failed to get container status \"edcdbff1f4f7329212ff0cd68aed0bac37e8b0a25ea633b113a37429a4fa77b3\": rpc error: code = NotFound desc = could not find container \"edcdbff1f4f7329212ff0cd68aed0bac37e8b0a25ea633b113a37429a4fa77b3\": container with ID starting with edcdbff1f4f7329212ff0cd68aed0bac37e8b0a25ea633b113a37429a4fa77b3 not found: ID does not exist" Oct 30 09:57:08.850741 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:08.850515 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7a213d-689cb5bf76-9b72m"] Oct 30 09:57:08.855984 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:08.855953 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7a213d-689cb5bf76-9b72m"] Oct 30 09:57:08.889147 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:08.889112 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7a213d-65fd87589f-rr7jz"] Oct 30 09:57:08.892979 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:08.892953 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7a213d-65fd87589f-rr7jz"] Oct 30 09:57:09.876106 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:09.876070 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8db504e0-4d30-41b1-91cb-12daa3e2ac43" path="/var/lib/kubelet/pods/8db504e0-4d30-41b1-91cb-12daa3e2ac43/volumes" Oct 30 09:57:09.876564 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:09.876549 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c33c6c07-06d5-4069-9eb8-2c176a66ab2a" path="/var/lib/kubelet/pods/c33c6c07-06d5-4069-9eb8-2c176a66ab2a/volumes" Oct 30 09:57:11.583677 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:11.583604 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-e3d2ff-cbb67b84b-j59gb" podUID="0bab9e9b-9d5e-4dde-81e2-cd274b635f8e" containerName="payload-processing" containerID="cri-o://2bbf69aa8d522dd6e2fb9873498c00f12a7eb7ade596713c152f77db67616e38" gracePeriod=25 Oct 30 09:57:11.613536 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:11.613488 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e3d2ff-7ffc858867-5wpzf" podUID="e2f602e8-2490-4eee-944b-f50c6ae747cc" containerName="payload-pre-processing" containerID="cri-o://1b2d8fbd5b7920e8fb63db6c48c5ccf9efa1e8504a6a6da159b1a2751db5ed9b" gracePeriod=25 Oct 30 09:57:11.843468 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:11.843366 4562 generic.go:356] "Generic (PLEG): container finished" podID="e2f602e8-2490-4eee-944b-f50c6ae747cc" containerID="1b2d8fbd5b7920e8fb63db6c48c5ccf9efa1e8504a6a6da159b1a2751db5ed9b" exitCode=0 Oct 30 09:57:11.843620 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:11.843556 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e3d2ff-7ffc858867-5wpzf" event={"ID":"e2f602e8-2490-4eee-944b-f50c6ae747cc","Type":"ContainerDied","Data":"1b2d8fbd5b7920e8fb63db6c48c5ccf9efa1e8504a6a6da159b1a2751db5ed9b"} Oct 30 09:57:11.845675 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:11.845645 4562 generic.go:356] "Generic (PLEG): container finished" podID="0bab9e9b-9d5e-4dde-81e2-cd274b635f8e" containerID="2bbf69aa8d522dd6e2fb9873498c00f12a7eb7ade596713c152f77db67616e38" exitCode=0 Oct 30 09:57:11.845730 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:11.845691 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-e3d2ff-cbb67b84b-j59gb" event={"ID":"0bab9e9b-9d5e-4dde-81e2-cd274b635f8e","Type":"ContainerDied","Data":"2bbf69aa8d522dd6e2fb9873498c00f12a7eb7ade596713c152f77db67616e38"} Oct 30 09:57:11.944582 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:11.944550 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-e3d2ff-cbb67b84b-j59gb" Oct 30 09:57:12.007348 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:12.007313 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e3d2ff-7ffc858867-5wpzf" Oct 30 09:57:12.053213 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:12.052968 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2f602e8-2490-4eee-944b-f50c6ae747cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2f602e8-2490-4eee-944b-f50c6ae747cc-plugins-config-volume\") pod \"e2f602e8-2490-4eee-944b-f50c6ae747cc\" (UID: \"e2f602e8-2490-4eee-944b-f50c6ae747cc\") " Oct 30 09:57:12.053213 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:12.053079 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bab9e9b-9d5e-4dde-81e2-cd274b635f8e-kube-api-access-dglsk\" (UniqueName: \"kubernetes.io/projected/0bab9e9b-9d5e-4dde-81e2-cd274b635f8e-kube-api-access-dglsk\") pod \"0bab9e9b-9d5e-4dde-81e2-cd274b635f8e\" (UID: \"0bab9e9b-9d5e-4dde-81e2-cd274b635f8e\") " Oct 30 09:57:12.053531 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:12.053285 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2f602e8-2490-4eee-944b-f50c6ae747cc-kube-api-access-bkl8k\" (UniqueName: \"kubernetes.io/projected/e2f602e8-2490-4eee-944b-f50c6ae747cc-kube-api-access-bkl8k\") pod \"e2f602e8-2490-4eee-944b-f50c6ae747cc\" (UID: \"e2f602e8-2490-4eee-944b-f50c6ae747cc\") " Oct 30 09:57:12.053531 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:12.053374 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bab9e9b-9d5e-4dde-81e2-cd274b635f8e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bab9e9b-9d5e-4dde-81e2-cd274b635f8e-plugins-config-volume\") pod \"0bab9e9b-9d5e-4dde-81e2-cd274b635f8e\" (UID: \"0bab9e9b-9d5e-4dde-81e2-cd274b635f8e\") " Oct 30 09:57:12.059010 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:12.054810 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bab9e9b-9d5e-4dde-81e2-cd274b635f8e-plugins-config-volume" pod "0bab9e9b-9d5e-4dde-81e2-cd274b635f8e" (UID: "0bab9e9b-9d5e-4dde-81e2-cd274b635f8e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:57:12.059010 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:12.056302 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2f602e8-2490-4eee-944b-f50c6ae747cc-plugins-config-volume" pod "e2f602e8-2490-4eee-944b-f50c6ae747cc" (UID: "e2f602e8-2490-4eee-944b-f50c6ae747cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:57:12.059705 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:12.059673 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bab9e9b-9d5e-4dde-81e2-cd274b635f8e-kube-api-access-dglsk" pod "0bab9e9b-9d5e-4dde-81e2-cd274b635f8e" (UID: "0bab9e9b-9d5e-4dde-81e2-cd274b635f8e"). InnerVolumeSpecName "kube-api-access-dglsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:57:12.061620 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:12.061595 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2f602e8-2490-4eee-944b-f50c6ae747cc-kube-api-access-bkl8k" pod "e2f602e8-2490-4eee-944b-f50c6ae747cc" (UID: "e2f602e8-2490-4eee-944b-f50c6ae747cc"). InnerVolumeSpecName "kube-api-access-bkl8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:57:12.155709 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:12.155620 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bab9e9b-9d5e-4dde-81e2-cd274b635f8e-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:57:12.155709 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:12.155652 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2f602e8-2490-4eee-944b-f50c6ae747cc-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:57:12.155709 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:12.155661 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dglsk\" (UniqueName: \"kubernetes.io/projected/0bab9e9b-9d5e-4dde-81e2-cd274b635f8e-kube-api-access-dglsk\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:57:12.155709 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:12.155671 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkl8k\" (UniqueName: \"kubernetes.io/projected/e2f602e8-2490-4eee-944b-f50c6ae747cc-kube-api-access-bkl8k\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:57:12.854752 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:12.854705 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e3d2ff-7ffc858867-5wpzf" event={"ID":"e2f602e8-2490-4eee-944b-f50c6ae747cc","Type":"ContainerDied","Data":"6d8c91bf30799fd0fb2d4ebc50e80ef1c058ef22c767a6478c144b7d67ea41d5"} Oct 30 09:57:12.855245 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:12.854720 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e3d2ff-7ffc858867-5wpzf" Oct 30 09:57:12.855245 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:12.854802 4562 scope.go:122] "RemoveContainer" containerID="1b2d8fbd5b7920e8fb63db6c48c5ccf9efa1e8504a6a6da159b1a2751db5ed9b" Oct 30 09:57:12.856579 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:12.856556 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-e3d2ff-cbb67b84b-j59gb" event={"ID":"0bab9e9b-9d5e-4dde-81e2-cd274b635f8e","Type":"ContainerDied","Data":"df96ea221efe0afdb24d9e4c97b2bd1d3302c2156faf5203a4da2491ab2d7c08"} Oct 30 09:57:12.856698 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:12.856542 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-e3d2ff-cbb67b84b-j59gb" Oct 30 09:57:12.867541 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:12.867348 4562 scope.go:122] "RemoveContainer" containerID="2bbf69aa8d522dd6e2fb9873498c00f12a7eb7ade596713c152f77db67616e38" Oct 30 09:57:12.924055 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:12.924023 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e3d2ff-7ffc858867-5wpzf"] Oct 30 09:57:12.936659 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:12.936578 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e3d2ff-7ffc858867-5wpzf"] Oct 30 09:57:12.973595 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:12.973530 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e3d2ff-cbb67b84b-j59gb"] Oct 30 09:57:12.977171 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:12.977142 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e3d2ff-cbb67b84b-j59gb"] Oct 30 09:57:13.878278 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:13.878242 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bab9e9b-9d5e-4dde-81e2-cd274b635f8e" path="/var/lib/kubelet/pods/0bab9e9b-9d5e-4dde-81e2-cd274b635f8e/volumes" Oct 30 09:57:13.878785 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:13.878764 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2f602e8-2490-4eee-944b-f50c6ae747cc" path="/var/lib/kubelet/pods/e2f602e8-2490-4eee-944b-f50c6ae747cc/volumes" Oct 30 09:57:15.742557 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:15.742501 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55" podUID="88be34c3-a9e9-4489-a449-206a99ef1663" containerName="main" containerID="cri-o://2e3a51433bccf125f099e070d60bba655bbb755e38aae223d00d08ee795595ee" gracePeriod=45 Oct 30 09:57:15.849957 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:15.849363 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55" podUID="88be34c3-a9e9-4489-a449-206a99ef1663" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": dial tcp 10.133.0.74:8000: connect: connection refused" Oct 30 09:57:15.865939 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:15.865902 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-176b6c/distinct-w3-176b6c-kserve-f57b7579c-k4pc5"] Oct 30 09:57:15.894620 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:15.894585 4562 generic.go:356] "Generic (PLEG): container finished" podID="88be34c3-a9e9-4489-a449-206a99ef1663" containerID="2e3a51433bccf125f099e070d60bba655bbb755e38aae223d00d08ee795595ee" exitCode=0 Oct 30 09:57:15.895801 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:15.894877 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55" event={"ID":"88be34c3-a9e9-4489-a449-206a99ef1663","Type":"ContainerDied","Data":"2e3a51433bccf125f099e070d60bba655bbb755e38aae223d00d08ee795595ee"} Oct 30 09:57:16.038206 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:16.038172 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55" Oct 30 09:57:16.128475 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:16.124968 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88be34c3-a9e9-4489-a449-206a99ef1663-tls-certs\" (UniqueName: \"kubernetes.io/secret/88be34c3-a9e9-4489-a449-206a99ef1663-tls-certs\") pod \"88be34c3-a9e9-4489-a449-206a99ef1663\" (UID: \"88be34c3-a9e9-4489-a449-206a99ef1663\") " Oct 30 09:57:16.128475 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:16.125133 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88be34c3-a9e9-4489-a449-206a99ef1663-kube-api-access-24zx9\" (UniqueName: \"kubernetes.io/projected/88be34c3-a9e9-4489-a449-206a99ef1663-kube-api-access-24zx9\") pod \"88be34c3-a9e9-4489-a449-206a99ef1663\" (UID: \"88be34c3-a9e9-4489-a449-206a99ef1663\") " Oct 30 09:57:16.128475 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:16.125195 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88be34c3-a9e9-4489-a449-206a99ef1663-home\" (UniqueName: \"kubernetes.io/empty-dir/88be34c3-a9e9-4489-a449-206a99ef1663-home\") pod \"88be34c3-a9e9-4489-a449-206a99ef1663\" (UID: \"88be34c3-a9e9-4489-a449-206a99ef1663\") " Oct 30 09:57:16.128475 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:16.125255 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88be34c3-a9e9-4489-a449-206a99ef1663-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88be34c3-a9e9-4489-a449-206a99ef1663-tmp-dir\") pod \"88be34c3-a9e9-4489-a449-206a99ef1663\" (UID: \"88be34c3-a9e9-4489-a449-206a99ef1663\") " Oct 30 09:57:16.128475 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:16.125374 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88be34c3-a9e9-4489-a449-206a99ef1663-dshm\" (UniqueName: \"kubernetes.io/empty-dir/88be34c3-a9e9-4489-a449-206a99ef1663-dshm\") pod \"88be34c3-a9e9-4489-a449-206a99ef1663\" (UID: \"88be34c3-a9e9-4489-a449-206a99ef1663\") " Oct 30 09:57:16.128475 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:16.125461 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88be34c3-a9e9-4489-a449-206a99ef1663-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88be34c3-a9e9-4489-a449-206a99ef1663-model-cache\") pod \"88be34c3-a9e9-4489-a449-206a99ef1663\" (UID: \"88be34c3-a9e9-4489-a449-206a99ef1663\") " Oct 30 09:57:16.132751 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:16.132709 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88be34c3-a9e9-4489-a449-206a99ef1663-tmp-dir" pod "88be34c3-a9e9-4489-a449-206a99ef1663" (UID: "88be34c3-a9e9-4489-a449-206a99ef1663"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:16.132980 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:16.132958 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88be34c3-a9e9-4489-a449-206a99ef1663-home" pod "88be34c3-a9e9-4489-a449-206a99ef1663" (UID: "88be34c3-a9e9-4489-a449-206a99ef1663"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:16.139942 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:16.139513 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88be34c3-a9e9-4489-a449-206a99ef1663-model-cache" pod "88be34c3-a9e9-4489-a449-206a99ef1663" (UID: "88be34c3-a9e9-4489-a449-206a99ef1663"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:16.143461 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:16.141766 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88be34c3-a9e9-4489-a449-206a99ef1663-tls-certs" pod "88be34c3-a9e9-4489-a449-206a99ef1663" (UID: "88be34c3-a9e9-4489-a449-206a99ef1663"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:57:16.151054 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:16.151005 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88be34c3-a9e9-4489-a449-206a99ef1663-kube-api-access-24zx9" pod "88be34c3-a9e9-4489-a449-206a99ef1663" (UID: "88be34c3-a9e9-4489-a449-206a99ef1663"). InnerVolumeSpecName "kube-api-access-24zx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:57:16.151054 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:16.151040 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88be34c3-a9e9-4489-a449-206a99ef1663-dshm" pod "88be34c3-a9e9-4489-a449-206a99ef1663" (UID: "88be34c3-a9e9-4489-a449-206a99ef1663"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:16.227213 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:16.227124 4562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88be34c3-a9e9-4489-a449-206a99ef1663-dshm\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:57:16.227213 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:16.227155 4562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88be34c3-a9e9-4489-a449-206a99ef1663-model-cache\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:57:16.227213 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:16.227165 4562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88be34c3-a9e9-4489-a449-206a99ef1663-tls-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:57:16.227213 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:16.227173 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24zx9\" (UniqueName: \"kubernetes.io/projected/88be34c3-a9e9-4489-a449-206a99ef1663-kube-api-access-24zx9\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:57:16.227213 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:16.227182 4562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88be34c3-a9e9-4489-a449-206a99ef1663-home\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:57:16.227213 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:16.227190 4562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88be34c3-a9e9-4489-a449-206a99ef1663-tmp-dir\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:57:16.398406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:16.398313 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2f6ee2-844666dcbd-56ppf"] Oct 30 09:57:16.431095 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:16.431059 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2f6ee2-684d557c9-gf6cq"] Oct 30 09:57:16.903590 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:16.903563 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55" Oct 30 09:57:16.903590 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:16.903585 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e3d2ff/distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55" event={"ID":"88be34c3-a9e9-4489-a449-206a99ef1663","Type":"ContainerDied","Data":"a99dbbef07136fd285aee2b8b3eb53e3db6426957f08cedd9e672a1be889cd11"} Oct 30 09:57:16.904081 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:16.903629 4562 scope.go:122] "RemoveContainer" containerID="2e3a51433bccf125f099e070d60bba655bbb755e38aae223d00d08ee795595ee" Oct 30 09:57:16.956780 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:16.956744 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e3d2ff/distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55"] Oct 30 09:57:16.969633 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:16.969595 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-e3d2ff/distinct-2-w2-e3d2ff-kserve-6fc55b8758-hmv55"] Oct 30 09:57:17.875224 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:17.875189 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88be34c3-a9e9-4489-a449-206a99ef1663" path="/var/lib/kubelet/pods/88be34c3-a9e9-4489-a449-206a99ef1663/volumes" Oct 30 09:57:19.246738 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:19.245896 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-c5447d-5c56c8576b-zpvxb"] Oct 30 09:57:19.246738 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:19.246136 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c5447d-5c56c8576b-zpvxb" podUID="18ca1016-fd69-495d-9398-320aa16c48cc" containerName="maas-api" containerID="cri-o://1cda1433af6840decad3edacb0763b0d317a98501b7112767b1d4cc4b89a233e" gracePeriod=30 Oct 30 09:57:19.589506 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:19.589471 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c5447d-5c56c8576b-zpvxb" Oct 30 09:57:19.598580 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:19.598550 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18ca1016-fd69-495d-9398-320aa16c48cc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/18ca1016-fd69-495d-9398-320aa16c48cc-metrics-certs\") pod \"18ca1016-fd69-495d-9398-320aa16c48cc\" (UID: \"18ca1016-fd69-495d-9398-320aa16c48cc\") " Oct 30 09:57:19.598756 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:19.598635 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18ca1016-fd69-495d-9398-320aa16c48cc-kube-api-access-lg5l5\" (UniqueName: \"kubernetes.io/projected/18ca1016-fd69-495d-9398-320aa16c48cc-kube-api-access-lg5l5\") pod \"18ca1016-fd69-495d-9398-320aa16c48cc\" (UID: \"18ca1016-fd69-495d-9398-320aa16c48cc\") " Oct 30 09:57:19.598756 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:19.598741 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18ca1016-fd69-495d-9398-320aa16c48cc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18ca1016-fd69-495d-9398-320aa16c48cc-maas-api-tls\") pod \"18ca1016-fd69-495d-9398-320aa16c48cc\" (UID: \"18ca1016-fd69-495d-9398-320aa16c48cc\") " Oct 30 09:57:19.599807 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:19.599782 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18ca1016-fd69-495d-9398-320aa16c48cc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18ca1016-fd69-495d-9398-320aa16c48cc-openshift-service-ca\") pod \"18ca1016-fd69-495d-9398-320aa16c48cc\" (UID: \"18ca1016-fd69-495d-9398-320aa16c48cc\") " Oct 30 09:57:19.600222 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:19.600177 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18ca1016-fd69-495d-9398-320aa16c48cc-openshift-service-ca" pod "18ca1016-fd69-495d-9398-320aa16c48cc" (UID: "18ca1016-fd69-495d-9398-320aa16c48cc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:57:19.601073 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:19.601044 4562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18ca1016-fd69-495d-9398-320aa16c48cc-openshift-service-ca\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:57:19.640462 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:19.637775 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18ca1016-fd69-495d-9398-320aa16c48cc-metrics-certs" pod "18ca1016-fd69-495d-9398-320aa16c48cc" (UID: "18ca1016-fd69-495d-9398-320aa16c48cc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:57:19.640462 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:19.637869 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18ca1016-fd69-495d-9398-320aa16c48cc-maas-api-tls" pod "18ca1016-fd69-495d-9398-320aa16c48cc" (UID: "18ca1016-fd69-495d-9398-320aa16c48cc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:57:19.640462 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:19.637958 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18ca1016-fd69-495d-9398-320aa16c48cc-kube-api-access-lg5l5" pod "18ca1016-fd69-495d-9398-320aa16c48cc" (UID: "18ca1016-fd69-495d-9398-320aa16c48cc"). InnerVolumeSpecName "kube-api-access-lg5l5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:57:19.715406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:19.702203 4562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18ca1016-fd69-495d-9398-320aa16c48cc-maas-api-tls\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:57:19.715406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:19.702237 4562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18ca1016-fd69-495d-9398-320aa16c48cc-metrics-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:57:19.715406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:19.702251 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lg5l5\" (UniqueName: \"kubernetes.io/projected/18ca1016-fd69-495d-9398-320aa16c48cc-kube-api-access-lg5l5\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:57:19.927877 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:19.927842 4562 generic.go:356] "Generic (PLEG): container finished" podID="18ca1016-fd69-495d-9398-320aa16c48cc" containerID="1cda1433af6840decad3edacb0763b0d317a98501b7112767b1d4cc4b89a233e" exitCode=0 Oct 30 09:57:19.928077 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:19.927889 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c5447d-5c56c8576b-zpvxb" Oct 30 09:57:19.928077 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:19.927921 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c5447d-5c56c8576b-zpvxb" event={"ID":"18ca1016-fd69-495d-9398-320aa16c48cc","Type":"ContainerDied","Data":"1cda1433af6840decad3edacb0763b0d317a98501b7112767b1d4cc4b89a233e"} Oct 30 09:57:19.928077 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:19.927979 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c5447d-5c56c8576b-zpvxb" event={"ID":"18ca1016-fd69-495d-9398-320aa16c48cc","Type":"ContainerDied","Data":"3c5639d0cb09fe979099fa0d9ad4f8d78a5395a1a678c1ea545588d3aa063067"} Oct 30 09:57:19.928077 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:19.927996 4562 scope.go:122] "RemoveContainer" containerID="1cda1433af6840decad3edacb0763b0d317a98501b7112767b1d4cc4b89a233e" Oct 30 09:57:19.942316 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:19.942263 4562 scope.go:122] "RemoveContainer" containerID="1cda1433af6840decad3edacb0763b0d317a98501b7112767b1d4cc4b89a233e" Oct 30 09:57:19.942726 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:57:19.942683 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cda1433af6840decad3edacb0763b0d317a98501b7112767b1d4cc4b89a233e\": container with ID starting with 1cda1433af6840decad3edacb0763b0d317a98501b7112767b1d4cc4b89a233e not found: ID does not exist" containerID="1cda1433af6840decad3edacb0763b0d317a98501b7112767b1d4cc4b89a233e" Oct 30 09:57:19.942862 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:19.942721 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cda1433af6840decad3edacb0763b0d317a98501b7112767b1d4cc4b89a233e"} err="failed to get container status \"1cda1433af6840decad3edacb0763b0d317a98501b7112767b1d4cc4b89a233e\": rpc error: code = NotFound desc = could not find container \"1cda1433af6840decad3edacb0763b0d317a98501b7112767b1d4cc4b89a233e\": container with ID starting with 1cda1433af6840decad3edacb0763b0d317a98501b7112767b1d4cc4b89a233e not found: ID does not exist" Oct 30 09:57:19.985777 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:19.985697 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-c5447d-5c56c8576b-zpvxb"] Oct 30 09:57:19.998467 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:19.998439 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-c5447d-5c56c8576b-zpvxb"] Oct 30 09:57:21.399152 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:21.399109 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-2f6ee2-844666dcbd-56ppf" podUID="36c4eb3b-f628-48a2-82b6-aa1826bbf0c6" containerName="payload-processing" containerID="cri-o://8ca6619bf30eedb4671b3a0f00b5f91ba7721b029a3e1678ed45b93513584e02" gracePeriod=25 Oct 30 09:57:21.431351 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:21.431301 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f6ee2-684d557c9-gf6cq" podUID="78e55c08-322e-464e-a798-2d3b2d894972" containerName="payload-pre-processing" containerID="cri-o://17b2b9376656089f75041188893d68877fd15b9e9719194dfde5615eb2037c5b" gracePeriod=25 Oct 30 09:57:21.572206 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:21.572050 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-176b6c-6bf94b8cd6-sq67c"] Oct 30 09:57:21.885361 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:21.885324 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18ca1016-fd69-495d-9398-320aa16c48cc" path="/var/lib/kubelet/pods/18ca1016-fd69-495d-9398-320aa16c48cc/volumes" Oct 30 09:57:26.572794 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:26.572728 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-176b6c-6bf94b8cd6-sq67c" podUID="6a347b84-639e-4f96-9794-4463ee2b8af6" containerName="payload-processing" containerID="cri-o://1cdedacca131e43ad38a08faefe34643e9f70f57568b184304b2f18b7876424d" gracePeriod=25 Oct 30 09:57:26.971003 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:26.970969 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-176b6c-6bf94b8cd6-sq67c" Oct 30 09:57:26.976577 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:26.976549 4562 generic.go:356] "Generic (PLEG): container finished" podID="6a347b84-639e-4f96-9794-4463ee2b8af6" containerID="1cdedacca131e43ad38a08faefe34643e9f70f57568b184304b2f18b7876424d" exitCode=0 Oct 30 09:57:26.976705 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:26.976597 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-176b6c-6bf94b8cd6-sq67c" Oct 30 09:57:26.976705 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:26.976637 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-176b6c-6bf94b8cd6-sq67c" event={"ID":"6a347b84-639e-4f96-9794-4463ee2b8af6","Type":"ContainerDied","Data":"1cdedacca131e43ad38a08faefe34643e9f70f57568b184304b2f18b7876424d"} Oct 30 09:57:26.976705 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:26.976696 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-176b6c-6bf94b8cd6-sq67c" event={"ID":"6a347b84-639e-4f96-9794-4463ee2b8af6","Type":"ContainerDied","Data":"a2e4a0a002a462a907c563c5fa4aedef367d7b595a48c0a15ff10a6fd182e735"} Oct 30 09:57:26.976814 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:26.976718 4562 scope.go:122] "RemoveContainer" containerID="1cdedacca131e43ad38a08faefe34643e9f70f57568b184304b2f18b7876424d" Oct 30 09:57:26.986599 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:26.986573 4562 scope.go:122] "RemoveContainer" containerID="1cdedacca131e43ad38a08faefe34643e9f70f57568b184304b2f18b7876424d" Oct 30 09:57:26.986868 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:57:26.986850 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cdedacca131e43ad38a08faefe34643e9f70f57568b184304b2f18b7876424d\": container with ID starting with 1cdedacca131e43ad38a08faefe34643e9f70f57568b184304b2f18b7876424d not found: ID does not exist" containerID="1cdedacca131e43ad38a08faefe34643e9f70f57568b184304b2f18b7876424d" Oct 30 09:57:26.986927 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:26.986878 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cdedacca131e43ad38a08faefe34643e9f70f57568b184304b2f18b7876424d"} err="failed to get container status \"1cdedacca131e43ad38a08faefe34643e9f70f57568b184304b2f18b7876424d\": rpc error: code = NotFound desc = could not find container \"1cdedacca131e43ad38a08faefe34643e9f70f57568b184304b2f18b7876424d\": container with ID starting with 1cdedacca131e43ad38a08faefe34643e9f70f57568b184304b2f18b7876424d not found: ID does not exist" Oct 30 09:57:27.006083 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.006053 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a347b84-639e-4f96-9794-4463ee2b8af6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a347b84-639e-4f96-9794-4463ee2b8af6-plugins-config-volume\") pod \"6a347b84-639e-4f96-9794-4463ee2b8af6\" (UID: \"6a347b84-639e-4f96-9794-4463ee2b8af6\") " Oct 30 09:57:27.006221 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.006099 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a347b84-639e-4f96-9794-4463ee2b8af6-kube-api-access-k4srh\" (UniqueName: \"kubernetes.io/projected/6a347b84-639e-4f96-9794-4463ee2b8af6-kube-api-access-k4srh\") pod \"6a347b84-639e-4f96-9794-4463ee2b8af6\" (UID: \"6a347b84-639e-4f96-9794-4463ee2b8af6\") " Oct 30 09:57:27.006530 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.006503 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a347b84-639e-4f96-9794-4463ee2b8af6-plugins-config-volume" pod "6a347b84-639e-4f96-9794-4463ee2b8af6" (UID: "6a347b84-639e-4f96-9794-4463ee2b8af6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:57:27.006612 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.006593 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a347b84-639e-4f96-9794-4463ee2b8af6-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:57:27.008281 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.008262 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a347b84-639e-4f96-9794-4463ee2b8af6-kube-api-access-k4srh" pod "6a347b84-639e-4f96-9794-4463ee2b8af6" (UID: "6a347b84-639e-4f96-9794-4463ee2b8af6"). InnerVolumeSpecName "kube-api-access-k4srh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:57:27.108397 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.108349 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4srh\" (UniqueName: \"kubernetes.io/projected/6a347b84-639e-4f96-9794-4463ee2b8af6-kube-api-access-k4srh\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:57:27.325650 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.325610 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-176b6c-6bf94b8cd6-sq67c"] Oct 30 09:57:27.332793 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.332762 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-176b6c-6bf94b8cd6-sq67c"] Oct 30 09:57:27.878564 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.878523 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a347b84-639e-4f96-9794-4463ee2b8af6" path="/var/lib/kubelet/pods/6a347b84-639e-4f96-9794-4463ee2b8af6/volumes" Oct 30 09:57:27.881806 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.881752 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da7a0f7c-4rnj7"] Oct 30 09:57:27.882698 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.882670 4562 cpu_manager.go:414] "containerMap: removing container" podUID="7f4a9934-3725-4b3c-82e4-900bdcd57e18" containerName="payload-processing" Oct 30 09:57:27.882698 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.882697 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f4a9934-3725-4b3c-82e4-900bdcd57e18" containerName="payload-processing" Oct 30 09:57:27.882888 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.882716 4562 cpu_manager.go:414] "containerMap: removing container" podUID="0bab9e9b-9d5e-4dde-81e2-cd274b635f8e" containerName="payload-processing" Oct 30 09:57:27.882888 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.882724 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bab9e9b-9d5e-4dde-81e2-cd274b635f8e" containerName="payload-processing" Oct 30 09:57:27.882888 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.882735 4562 cpu_manager.go:414] "containerMap: removing container" podUID="3e125f79-421e-4ac0-8f37-8bbd5f430c9a" containerName="payload-processing" Oct 30 09:57:27.882888 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.882743 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e125f79-421e-4ac0-8f37-8bbd5f430c9a" containerName="payload-processing" Oct 30 09:57:27.882888 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.882756 4562 cpu_manager.go:414] "containerMap: removing container" podUID="c0dbb750-1c20-41a5-b33f-5784a1f683d9" containerName="payload-pre-processing" Oct 30 09:57:27.882888 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.882763 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0dbb750-1c20-41a5-b33f-5784a1f683d9" containerName="payload-pre-processing" Oct 30 09:57:27.882888 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.882773 4562 cpu_manager.go:414] "containerMap: removing container" podUID="e574e901-6a15-4cb8-9a91-0a96e5347b07" containerName="payload-processing" Oct 30 09:57:27.882888 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.882780 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e574e901-6a15-4cb8-9a91-0a96e5347b07" containerName="payload-processing" Oct 30 09:57:27.882888 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.882810 4562 cpu_manager.go:414] "containerMap: removing container" podUID="7a163089-8bc6-4fde-afbb-7970c8ae3a08" containerName="maas-api" Oct 30 09:57:27.882888 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.882818 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a163089-8bc6-4fde-afbb-7970c8ae3a08" containerName="maas-api" Oct 30 09:57:27.882888 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.882834 4562 cpu_manager.go:414] "containerMap: removing container" podUID="e2f602e8-2490-4eee-944b-f50c6ae747cc" containerName="payload-pre-processing" Oct 30 09:57:27.882888 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.882841 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2f602e8-2490-4eee-944b-f50c6ae747cc" containerName="payload-pre-processing" Oct 30 09:57:27.882888 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.882856 4562 cpu_manager.go:414] "containerMap: removing container" podUID="0ecc2c25-ad9a-438b-9b2c-ab6e98905576" containerName="maas-api" Oct 30 09:57:27.882888 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.882863 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ecc2c25-ad9a-438b-9b2c-ab6e98905576" containerName="maas-api" Oct 30 09:57:27.882888 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.882878 4562 cpu_manager.go:414] "containerMap: removing container" podUID="c9ff897f-c520-4d0f-8820-e185b1d47dac" containerName="main" Oct 30 09:57:27.882888 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.882884 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9ff897f-c520-4d0f-8820-e185b1d47dac" containerName="main" Oct 30 09:57:27.882888 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.882898 4562 cpu_manager.go:414] "containerMap: removing container" podUID="6a347b84-639e-4f96-9794-4463ee2b8af6" containerName="payload-processing" Oct 30 09:57:27.883854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.882905 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a347b84-639e-4f96-9794-4463ee2b8af6" containerName="payload-processing" Oct 30 09:57:27.883854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.882917 4562 cpu_manager.go:414] "containerMap: removing container" podUID="c15af94f-dcf8-47ab-a39a-34de624afe00" containerName="payload-pre-processing" Oct 30 09:57:27.883854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.882924 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c15af94f-dcf8-47ab-a39a-34de624afe00" containerName="payload-pre-processing" Oct 30 09:57:27.883854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.882933 4562 cpu_manager.go:414] "containerMap: removing container" podUID="c8073ce8-37bf-4ad9-9311-d17bdc00fbb0" containerName="payload-processing" Oct 30 09:57:27.883854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.882940 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8073ce8-37bf-4ad9-9311-d17bdc00fbb0" containerName="payload-processing" Oct 30 09:57:27.883854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.882950 4562 cpu_manager.go:414] "containerMap: removing container" podUID="17acd011-7a83-4ac2-9689-3a5568b7f94b" containerName="maas-api" Oct 30 09:57:27.883854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.882957 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17acd011-7a83-4ac2-9689-3a5568b7f94b" containerName="maas-api" Oct 30 09:57:27.883854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.882970 4562 cpu_manager.go:414] "containerMap: removing container" podUID="a75406b8-14aa-4fc1-975e-620b911cb9b7" containerName="payload-pre-processing" Oct 30 09:57:27.883854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.882977 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a75406b8-14aa-4fc1-975e-620b911cb9b7" containerName="payload-pre-processing" Oct 30 09:57:27.883854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.882996 4562 cpu_manager.go:414] "containerMap: removing container" podUID="37fad2d6-d1ba-4440-bf0f-0f46c12987fb" containerName="maas-api" Oct 30 09:57:27.883854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.883004 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37fad2d6-d1ba-4440-bf0f-0f46c12987fb" containerName="maas-api" Oct 30 09:57:27.883854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.883014 4562 cpu_manager.go:414] "containerMap: removing container" podUID="d696e344-8d35-4896-84dc-9a4dc225f291" containerName="payload-processing" Oct 30 09:57:27.883854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.883022 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d696e344-8d35-4896-84dc-9a4dc225f291" containerName="payload-processing" Oct 30 09:57:27.883854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.883060 4562 cpu_manager.go:414] "containerMap: removing container" podUID="821c88fc-b72b-4104-97f3-e3528428498e" containerName="istio-proxy" Oct 30 09:57:27.883854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.883067 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="821c88fc-b72b-4104-97f3-e3528428498e" containerName="istio-proxy" Oct 30 09:57:27.883854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.883076 4562 cpu_manager.go:414] "containerMap: removing container" podUID="821c88fc-b72b-4104-97f3-e3528428498e" containerName="istio-proxy" Oct 30 09:57:27.883854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.883083 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="821c88fc-b72b-4104-97f3-e3528428498e" containerName="istio-proxy" Oct 30 09:57:27.883854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.883092 4562 cpu_manager.go:414] "containerMap: removing container" podUID="d7f74510-23f8-4f0b-af8e-6c9861215203" containerName="istio-proxy" Oct 30 09:57:27.883854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.883099 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7f74510-23f8-4f0b-af8e-6c9861215203" containerName="istio-proxy" Oct 30 09:57:27.883854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.883117 4562 cpu_manager.go:414] "containerMap: removing container" podUID="4a4b5a98-b1b2-41ff-9130-1f2367ccff29" containerName="payload-pre-processing" Oct 30 09:57:27.883854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.883124 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a4b5a98-b1b2-41ff-9130-1f2367ccff29" containerName="payload-pre-processing" Oct 30 09:57:27.883854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.883135 4562 cpu_manager.go:414] "containerMap: removing container" podUID="aea403b0-3184-48ef-8e1a-b1f91b56d703" containerName="istio-proxy" Oct 30 09:57:27.883854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.883143 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aea403b0-3184-48ef-8e1a-b1f91b56d703" containerName="istio-proxy" Oct 30 09:57:27.883854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.883155 4562 cpu_manager.go:414] "containerMap: removing container" podUID="18ca1016-fd69-495d-9398-320aa16c48cc" containerName="maas-api" Oct 30 09:57:27.883854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.883162 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18ca1016-fd69-495d-9398-320aa16c48cc" containerName="maas-api" Oct 30 09:57:27.883854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.883174 4562 cpu_manager.go:414] "containerMap: removing container" podUID="fa734120-4c82-426f-947c-1a36330750ba" containerName="payload-pre-processing" Oct 30 09:57:27.883854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.883181 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa734120-4c82-426f-947c-1a36330750ba" containerName="payload-pre-processing" Oct 30 09:57:27.883854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.883191 4562 cpu_manager.go:414] "containerMap: removing container" podUID="310de27f-d65f-4c73-840a-df4b5f6aed8b" containerName="istio-proxy" Oct 30 09:57:27.883854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.883198 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="310de27f-d65f-4c73-840a-df4b5f6aed8b" containerName="istio-proxy" Oct 30 09:57:27.883854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.883216 4562 cpu_manager.go:414] "containerMap: removing container" podUID="c33c6c07-06d5-4069-9eb8-2c176a66ab2a" containerName="payload-pre-processing" Oct 30 09:57:27.883854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.883223 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c33c6c07-06d5-4069-9eb8-2c176a66ab2a" containerName="payload-pre-processing" Oct 30 09:57:27.883854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.883237 4562 cpu_manager.go:414] "containerMap: removing container" podUID="88be34c3-a9e9-4489-a449-206a99ef1663" containerName="main" Oct 30 09:57:27.883854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.883244 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88be34c3-a9e9-4489-a449-206a99ef1663" containerName="main" Oct 30 09:57:27.883854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.883259 4562 cpu_manager.go:414] "containerMap: removing container" podUID="8db504e0-4d30-41b1-91cb-12daa3e2ac43" containerName="payload-processing" Oct 30 09:57:27.883854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.883267 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8db504e0-4d30-41b1-91cb-12daa3e2ac43" containerName="payload-processing" Oct 30 09:57:27.883854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.883457 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ecc2c25-ad9a-438b-9b2c-ab6e98905576" containerName="maas-api" Oct 30 09:57:27.883854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.883478 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9ff897f-c520-4d0f-8820-e185b1d47dac" containerName="main" Oct 30 09:57:27.883854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.883491 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a347b84-639e-4f96-9794-4463ee2b8af6" containerName="payload-processing" Oct 30 09:57:27.883854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.883500 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="c15af94f-dcf8-47ab-a39a-34de624afe00" containerName="payload-pre-processing" Oct 30 09:57:27.883854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.883509 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8073ce8-37bf-4ad9-9311-d17bdc00fbb0" containerName="payload-processing" Oct 30 09:57:27.883854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.883516 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="17acd011-7a83-4ac2-9689-3a5568b7f94b" containerName="maas-api" Oct 30 09:57:27.883854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.883526 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="a75406b8-14aa-4fc1-975e-620b911cb9b7" containerName="payload-pre-processing" Oct 30 09:57:27.883854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.883537 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="37fad2d6-d1ba-4440-bf0f-0f46c12987fb" containerName="maas-api" Oct 30 09:57:27.883854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.883547 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="d696e344-8d35-4896-84dc-9a4dc225f291" containerName="payload-processing" Oct 30 09:57:27.883854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.883557 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="821c88fc-b72b-4104-97f3-e3528428498e" containerName="istio-proxy" Oct 30 09:57:27.883854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.883565 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="821c88fc-b72b-4104-97f3-e3528428498e" containerName="istio-proxy" Oct 30 09:57:27.883854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.883574 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7f74510-23f8-4f0b-af8e-6c9861215203" containerName="istio-proxy" Oct 30 09:57:27.883854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.883583 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a4b5a98-b1b2-41ff-9130-1f2367ccff29" containerName="payload-pre-processing" Oct 30 09:57:27.883854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.883593 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="18ca1016-fd69-495d-9398-320aa16c48cc" containerName="maas-api" Oct 30 09:57:27.883854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.883603 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa734120-4c82-426f-947c-1a36330750ba" containerName="payload-pre-processing" Oct 30 09:57:27.883854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.883613 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="310de27f-d65f-4c73-840a-df4b5f6aed8b" containerName="istio-proxy" Oct 30 09:57:27.883854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.883621 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="c33c6c07-06d5-4069-9eb8-2c176a66ab2a" containerName="payload-pre-processing" Oct 30 09:57:27.883854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.883631 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="88be34c3-a9e9-4489-a449-206a99ef1663" containerName="main" Oct 30 09:57:27.883854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.883645 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="8db504e0-4d30-41b1-91cb-12daa3e2ac43" containerName="payload-processing" Oct 30 09:57:27.883854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.883656 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f4a9934-3725-4b3c-82e4-900bdcd57e18" containerName="payload-processing" Oct 30 09:57:27.883854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.883667 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bab9e9b-9d5e-4dde-81e2-cd274b635f8e" containerName="payload-processing" Oct 30 09:57:27.883854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.883676 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e125f79-421e-4ac0-8f37-8bbd5f430c9a" containerName="payload-processing" Oct 30 09:57:27.883854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.883685 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0dbb750-1c20-41a5-b33f-5784a1f683d9" containerName="payload-pre-processing" Oct 30 09:57:27.883854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.883695 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="e574e901-6a15-4cb8-9a91-0a96e5347b07" containerName="payload-processing" Oct 30 09:57:27.883854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.883707 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a163089-8bc6-4fde-afbb-7970c8ae3a08" containerName="maas-api" Oct 30 09:57:27.883854 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.883717 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2f602e8-2490-4eee-944b-f50c6ae747cc" containerName="payload-pre-processing" Oct 30 09:57:27.889481 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.889456 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7a0f7c-4rnj7" Oct 30 09:57:27.914367 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.914338 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:57:27.927281 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.927247 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:57:27.932462 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:27.932433 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da7a0f7c-4rnj7"] Oct 30 09:57:28.023097 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:28.023063 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0caf9cd-3ae5-4d1f-9815-71b2310cf896-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da7a0f7c-4rnj7\" (UID: \"c0caf9cd-3ae5-4d1f-9815-71b2310cf896\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7a0f7c-4rnj7" Oct 30 09:57:28.023277 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:28.023125 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0caf9cd-3ae5-4d1f-9815-71b2310cf896-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da7a0f7c-4rnj7\" (UID: \"c0caf9cd-3ae5-4d1f-9815-71b2310cf896\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7a0f7c-4rnj7" Oct 30 09:57:28.023335 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:28.023311 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r425w\" (UniqueName: \"kubernetes.io/projected/c0caf9cd-3ae5-4d1f-9815-71b2310cf896-kube-api-access-r425w\") pod \"maas-api-revoke-sub-da7a0f7c-4rnj7\" (UID: \"c0caf9cd-3ae5-4d1f-9815-71b2310cf896\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7a0f7c-4rnj7" Oct 30 09:57:28.124650 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:28.124615 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r425w\" (UniqueName: \"kubernetes.io/projected/c0caf9cd-3ae5-4d1f-9815-71b2310cf896-kube-api-access-r425w\") pod \"maas-api-revoke-sub-da7a0f7c-4rnj7\" (UID: \"c0caf9cd-3ae5-4d1f-9815-71b2310cf896\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7a0f7c-4rnj7" Oct 30 09:57:28.124827 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:28.124767 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0caf9cd-3ae5-4d1f-9815-71b2310cf896-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da7a0f7c-4rnj7\" (UID: \"c0caf9cd-3ae5-4d1f-9815-71b2310cf896\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7a0f7c-4rnj7" Oct 30 09:57:28.124827 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:28.124812 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0caf9cd-3ae5-4d1f-9815-71b2310cf896-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da7a0f7c-4rnj7\" (UID: \"c0caf9cd-3ae5-4d1f-9815-71b2310cf896\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7a0f7c-4rnj7" Oct 30 09:57:28.125534 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:28.125514 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0caf9cd-3ae5-4d1f-9815-71b2310cf896-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da7a0f7c-4rnj7\" (UID: \"c0caf9cd-3ae5-4d1f-9815-71b2310cf896\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7a0f7c-4rnj7" Oct 30 09:57:28.127138 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:28.127114 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0caf9cd-3ae5-4d1f-9815-71b2310cf896-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da7a0f7c-4rnj7\" (UID: \"c0caf9cd-3ae5-4d1f-9815-71b2310cf896\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7a0f7c-4rnj7" Oct 30 09:57:28.144290 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:28.144229 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r425w\" (UniqueName: \"kubernetes.io/projected/c0caf9cd-3ae5-4d1f-9815-71b2310cf896-kube-api-access-r425w\") pod \"maas-api-revoke-sub-da7a0f7c-4rnj7\" (UID: \"c0caf9cd-3ae5-4d1f-9815-71b2310cf896\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7a0f7c-4rnj7" Oct 30 09:57:28.205159 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:28.205121 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7a0f7c-4rnj7" Oct 30 09:57:28.507217 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:28.507188 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da7a0f7c-4rnj7"] Oct 30 09:57:28.507358 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:57:28.507309 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0caf9cd_3ae5_4d1f_9815_71b2310cf896.slice/crio-9d75bd3f11b58fc2f2d0c699c0c9edb4fc4ee9b0f5d20e976ab32eacc125c4f0 WatchSource:0}: Error finding container 9d75bd3f11b58fc2f2d0c699c0c9edb4fc4ee9b0f5d20e976ab32eacc125c4f0: Status 404 returned error can't find the container with id 9d75bd3f11b58fc2f2d0c699c0c9edb4fc4ee9b0f5d20e976ab32eacc125c4f0 Oct 30 09:57:28.596972 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:57:28.596929 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:57:28.992457 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:28.992422 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7a0f7c-4rnj7" event={"ID":"c0caf9cd-3ae5-4d1f-9815-71b2310cf896","Type":"ContainerStarted","Data":"7aba65f79d6f3d19fe36cd00553ad7a2c3678d5a3c1d6f5ae2b2f6bdc542557e"} Oct 30 09:57:28.992851 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:28.992480 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7a0f7c-4rnj7" event={"ID":"c0caf9cd-3ae5-4d1f-9815-71b2310cf896","Type":"ContainerStarted","Data":"9d75bd3f11b58fc2f2d0c699c0c9edb4fc4ee9b0f5d20e976ab32eacc125c4f0"} Oct 30 09:57:29.032898 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:29.032845 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7a0f7c-4rnj7" podStartSLOduration=2.032832007 podStartE2EDuration="2.032832007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:57:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:57:29.031331089 +0000 UTC m=+3809.292892076" watchObservedRunningTime="2026-10-30 09:57:29.032832007 +0000 UTC m=+3809.294392995" Oct 30 09:57:30.001546 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:30.001503 4562 generic.go:356] "Generic (PLEG): container finished" podID="c0caf9cd-3ae5-4d1f-9815-71b2310cf896" containerID="7aba65f79d6f3d19fe36cd00553ad7a2c3678d5a3c1d6f5ae2b2f6bdc542557e" exitCode=0 Oct 30 09:57:30.001962 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:30.001626 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7a0f7c-4rnj7" event={"ID":"c0caf9cd-3ae5-4d1f-9815-71b2310cf896","Type":"ContainerDied","Data":"7aba65f79d6f3d19fe36cd00553ad7a2c3678d5a3c1d6f5ae2b2f6bdc542557e"} Oct 30 09:57:30.893432 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:30.893370 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-176b6c/distinct-w3-176b6c-kserve-f57b7579c-k4pc5" podUID="28fd4072-399c-4be5-a0d8-1be6966da4af" containerName="main" containerID="cri-o://b376c572007906a2386d0e29cc622df795f0467d733dcda1cf97611cd380e422" gracePeriod=45 Oct 30 09:57:31.000514 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:31.000474 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f6ee2-684d557c9-gf6cq" podUID="78e55c08-322e-464e-a798-2d3b2d894972" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:57:31.000672 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:31.000492 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-2f6ee2-844666dcbd-56ppf" podUID="36c4eb3b-f628-48a2-82b6-aa1826bbf0c6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:57:31.009957 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:31.009928 4562 generic.go:356] "Generic (PLEG): container finished" podID="28fd4072-399c-4be5-a0d8-1be6966da4af" containerID="b376c572007906a2386d0e29cc622df795f0467d733dcda1cf97611cd380e422" exitCode=0 Oct 30 09:57:31.010325 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:31.010003 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-176b6c/distinct-w3-176b6c-kserve-f57b7579c-k4pc5" event={"ID":"28fd4072-399c-4be5-a0d8-1be6966da4af","Type":"ContainerDied","Data":"b376c572007906a2386d0e29cc622df795f0467d733dcda1cf97611cd380e422"} Oct 30 09:57:31.214516 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:31.214491 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-176b6c/distinct-w3-176b6c-kserve-f57b7579c-k4pc5" Oct 30 09:57:31.218218 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:31.218189 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7a0f7c-4rnj7" Oct 30 09:57:31.266191 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:31.266158 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28fd4072-399c-4be5-a0d8-1be6966da4af-home\" (UniqueName: \"kubernetes.io/empty-dir/28fd4072-399c-4be5-a0d8-1be6966da4af-home\") pod \"28fd4072-399c-4be5-a0d8-1be6966da4af\" (UID: \"28fd4072-399c-4be5-a0d8-1be6966da4af\") " Oct 30 09:57:31.266464 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:31.266436 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28fd4072-399c-4be5-a0d8-1be6966da4af-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28fd4072-399c-4be5-a0d8-1be6966da4af-tmp-dir\") pod \"28fd4072-399c-4be5-a0d8-1be6966da4af\" (UID: \"28fd4072-399c-4be5-a0d8-1be6966da4af\") " Oct 30 09:57:31.266464 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:31.266460 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28fd4072-399c-4be5-a0d8-1be6966da4af-home" pod "28fd4072-399c-4be5-a0d8-1be6966da4af" (UID: "28fd4072-399c-4be5-a0d8-1be6966da4af"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:31.266678 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:31.266495 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28fd4072-399c-4be5-a0d8-1be6966da4af-tls-certs\" (UniqueName: \"kubernetes.io/secret/28fd4072-399c-4be5-a0d8-1be6966da4af-tls-certs\") pod \"28fd4072-399c-4be5-a0d8-1be6966da4af\" (UID: \"28fd4072-399c-4be5-a0d8-1be6966da4af\") " Oct 30 09:57:31.266678 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:31.266535 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28fd4072-399c-4be5-a0d8-1be6966da4af-dshm\" (UniqueName: \"kubernetes.io/empty-dir/28fd4072-399c-4be5-a0d8-1be6966da4af-dshm\") pod \"28fd4072-399c-4be5-a0d8-1be6966da4af\" (UID: \"28fd4072-399c-4be5-a0d8-1be6966da4af\") " Oct 30 09:57:31.266678 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:31.266659 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0caf9cd-3ae5-4d1f-9815-71b2310cf896-kube-api-access-r425w\" (UniqueName: \"kubernetes.io/projected/c0caf9cd-3ae5-4d1f-9815-71b2310cf896-kube-api-access-r425w\") pod \"c0caf9cd-3ae5-4d1f-9815-71b2310cf896\" (UID: \"c0caf9cd-3ae5-4d1f-9815-71b2310cf896\") " Oct 30 09:57:31.266855 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:31.266706 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28fd4072-399c-4be5-a0d8-1be6966da4af-tmp-dir" pod "28fd4072-399c-4be5-a0d8-1be6966da4af" (UID: "28fd4072-399c-4be5-a0d8-1be6966da4af"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:31.266855 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:31.266722 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0caf9cd-3ae5-4d1f-9815-71b2310cf896-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0caf9cd-3ae5-4d1f-9815-71b2310cf896-maas-api-xks-ca\") pod \"c0caf9cd-3ae5-4d1f-9815-71b2310cf896\" (UID: \"c0caf9cd-3ae5-4d1f-9815-71b2310cf896\") " Oct 30 09:57:31.266855 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:31.266771 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28fd4072-399c-4be5-a0d8-1be6966da4af-kube-api-access-vjfdp\" (UniqueName: \"kubernetes.io/projected/28fd4072-399c-4be5-a0d8-1be6966da4af-kube-api-access-vjfdp\") pod \"28fd4072-399c-4be5-a0d8-1be6966da4af\" (UID: \"28fd4072-399c-4be5-a0d8-1be6966da4af\") " Oct 30 09:57:31.266855 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:31.266833 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28fd4072-399c-4be5-a0d8-1be6966da4af-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28fd4072-399c-4be5-a0d8-1be6966da4af-model-cache\") pod \"28fd4072-399c-4be5-a0d8-1be6966da4af\" (UID: \"28fd4072-399c-4be5-a0d8-1be6966da4af\") " Oct 30 09:57:31.267078 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:31.266897 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0caf9cd-3ae5-4d1f-9815-71b2310cf896-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0caf9cd-3ae5-4d1f-9815-71b2310cf896-maas-api-openshift-ca\") pod \"c0caf9cd-3ae5-4d1f-9815-71b2310cf896\" (UID: \"c0caf9cd-3ae5-4d1f-9815-71b2310cf896\") " Oct 30 09:57:31.267621 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:31.267599 4562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28fd4072-399c-4be5-a0d8-1be6966da4af-home\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:57:31.267808 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:31.267791 4562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28fd4072-399c-4be5-a0d8-1be6966da4af-tmp-dir\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:57:31.268736 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:31.268089 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28fd4072-399c-4be5-a0d8-1be6966da4af-model-cache" pod "28fd4072-399c-4be5-a0d8-1be6966da4af" (UID: "28fd4072-399c-4be5-a0d8-1be6966da4af"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:31.268736 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:31.268160 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0caf9cd-3ae5-4d1f-9815-71b2310cf896-maas-api-openshift-ca" pod "c0caf9cd-3ae5-4d1f-9815-71b2310cf896" (UID: "c0caf9cd-3ae5-4d1f-9815-71b2310cf896"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:57:31.272725 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:31.272696 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28fd4072-399c-4be5-a0d8-1be6966da4af-kube-api-access-vjfdp" pod "28fd4072-399c-4be5-a0d8-1be6966da4af" (UID: "28fd4072-399c-4be5-a0d8-1be6966da4af"). InnerVolumeSpecName "kube-api-access-vjfdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:57:31.279542 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:31.279510 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28fd4072-399c-4be5-a0d8-1be6966da4af-tls-certs" pod "28fd4072-399c-4be5-a0d8-1be6966da4af" (UID: "28fd4072-399c-4be5-a0d8-1be6966da4af"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:57:31.279806 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:31.279776 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0caf9cd-3ae5-4d1f-9815-71b2310cf896-maas-api-xks-ca" pod "c0caf9cd-3ae5-4d1f-9815-71b2310cf896" (UID: "c0caf9cd-3ae5-4d1f-9815-71b2310cf896"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:57:31.279992 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:31.279964 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0caf9cd-3ae5-4d1f-9815-71b2310cf896-kube-api-access-r425w" pod "c0caf9cd-3ae5-4d1f-9815-71b2310cf896" (UID: "c0caf9cd-3ae5-4d1f-9815-71b2310cf896"). InnerVolumeSpecName "kube-api-access-r425w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:57:31.280090 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:31.280075 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28fd4072-399c-4be5-a0d8-1be6966da4af-dshm" pod "28fd4072-399c-4be5-a0d8-1be6966da4af" (UID: "28fd4072-399c-4be5-a0d8-1be6966da4af"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:31.369049 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:31.369012 4562 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0caf9cd-3ae5-4d1f-9815-71b2310cf896-maas-api-xks-ca\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:57:31.369049 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:31.369046 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjfdp\" (UniqueName: \"kubernetes.io/projected/28fd4072-399c-4be5-a0d8-1be6966da4af-kube-api-access-vjfdp\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:57:31.369049 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:31.369058 4562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28fd4072-399c-4be5-a0d8-1be6966da4af-model-cache\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:57:31.369296 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:31.369068 4562 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0caf9cd-3ae5-4d1f-9815-71b2310cf896-maas-api-openshift-ca\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:57:31.369296 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:31.369077 4562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28fd4072-399c-4be5-a0d8-1be6966da4af-tls-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:57:31.369296 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:31.369087 4562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28fd4072-399c-4be5-a0d8-1be6966da4af-dshm\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:57:31.369296 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:31.369095 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r425w\" (UniqueName: \"kubernetes.io/projected/c0caf9cd-3ae5-4d1f-9815-71b2310cf896-kube-api-access-r425w\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:57:32.025232 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:32.025188 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7a0f7c-4rnj7" event={"ID":"c0caf9cd-3ae5-4d1f-9815-71b2310cf896","Type":"ContainerDied","Data":"9d75bd3f11b58fc2f2d0c699c0c9edb4fc4ee9b0f5d20e976ab32eacc125c4f0"} Oct 30 09:57:32.025792 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:32.025320 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7a0f7c-4rnj7" Oct 30 09:57:32.025792 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:32.025330 4562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d75bd3f11b58fc2f2d0c699c0c9edb4fc4ee9b0f5d20e976ab32eacc125c4f0" Oct 30 09:57:32.029401 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:32.028620 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-176b6c/distinct-w3-176b6c-kserve-f57b7579c-k4pc5" event={"ID":"28fd4072-399c-4be5-a0d8-1be6966da4af","Type":"ContainerDied","Data":"f13dfee49c3cfd70cd8f918bf692c1d364589bd30d60b61eebf432420c19bfd5"} Oct 30 09:57:32.029401 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:32.028676 4562 scope.go:122] "RemoveContainer" containerID="b376c572007906a2386d0e29cc622df795f0467d733dcda1cf97611cd380e422" Oct 30 09:57:32.029401 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:32.028812 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-176b6c/distinct-w3-176b6c-kserve-f57b7579c-k4pc5" Oct 30 09:57:32.090635 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:32.090600 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-176b6c/distinct-w3-176b6c-kserve-f57b7579c-k4pc5"] Oct 30 09:57:32.093756 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:32.093723 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-176b6c/distinct-w3-176b6c-kserve-f57b7579c-k4pc5"] Oct 30 09:57:33.876451 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:33.876407 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28fd4072-399c-4be5-a0d8-1be6966da4af" path="/var/lib/kubelet/pods/28fd4072-399c-4be5-a0d8-1be6966da4af/volumes" Oct 30 09:57:37.745405 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:37.741742 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-bdd208-784594d77d-z6qzr"] Oct 30 09:57:37.791441 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:37.791206 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bdd208-6dff5fbd47-69qxt"] Oct 30 09:57:39.480055 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:39.480024 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-e3d2ff-78f84d76-w6jp2"] Oct 30 09:57:39.480485 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:39.480236 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e3d2ff-78f84d76-w6jp2" podUID="a70bbb8b-7d34-488e-bd32-fa463616f2e7" containerName="maas-api" containerID="cri-o://a7d2ccfe44e7c50d45db7e518245b48ccb2a192e868810bc37454627b691224f" gracePeriod=30 Oct 30 09:57:39.819600 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:39.819568 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e3d2ff-78f84d76-w6jp2" Oct 30 09:57:39.866404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:39.865932 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a70bbb8b-7d34-488e-bd32-fa463616f2e7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a70bbb8b-7d34-488e-bd32-fa463616f2e7-openshift-service-ca\") pod \"a70bbb8b-7d34-488e-bd32-fa463616f2e7\" (UID: \"a70bbb8b-7d34-488e-bd32-fa463616f2e7\") " Oct 30 09:57:39.866404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:39.865990 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a70bbb8b-7d34-488e-bd32-fa463616f2e7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a70bbb8b-7d34-488e-bd32-fa463616f2e7-metrics-certs\") pod \"a70bbb8b-7d34-488e-bd32-fa463616f2e7\" (UID: \"a70bbb8b-7d34-488e-bd32-fa463616f2e7\") " Oct 30 09:57:39.866404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:39.866099 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a70bbb8b-7d34-488e-bd32-fa463616f2e7-kube-api-access-lvh6v\" (UniqueName: \"kubernetes.io/projected/a70bbb8b-7d34-488e-bd32-fa463616f2e7-kube-api-access-lvh6v\") pod \"a70bbb8b-7d34-488e-bd32-fa463616f2e7\" (UID: \"a70bbb8b-7d34-488e-bd32-fa463616f2e7\") " Oct 30 09:57:39.876402 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:39.866848 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a70bbb8b-7d34-488e-bd32-fa463616f2e7-openshift-service-ca" pod "a70bbb8b-7d34-488e-bd32-fa463616f2e7" (UID: "a70bbb8b-7d34-488e-bd32-fa463616f2e7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:57:39.876402 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:39.867297 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a70bbb8b-7d34-488e-bd32-fa463616f2e7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a70bbb8b-7d34-488e-bd32-fa463616f2e7-maas-api-tls\") pod \"a70bbb8b-7d34-488e-bd32-fa463616f2e7\" (UID: \"a70bbb8b-7d34-488e-bd32-fa463616f2e7\") " Oct 30 09:57:39.876402 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:39.867951 4562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a70bbb8b-7d34-488e-bd32-fa463616f2e7-openshift-service-ca\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:57:39.916403 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:39.908295 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a70bbb8b-7d34-488e-bd32-fa463616f2e7-maas-api-tls" pod "a70bbb8b-7d34-488e-bd32-fa463616f2e7" (UID: "a70bbb8b-7d34-488e-bd32-fa463616f2e7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:57:39.932568 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:39.932534 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a70bbb8b-7d34-488e-bd32-fa463616f2e7-kube-api-access-lvh6v" pod "a70bbb8b-7d34-488e-bd32-fa463616f2e7" (UID: "a70bbb8b-7d34-488e-bd32-fa463616f2e7"). InnerVolumeSpecName "kube-api-access-lvh6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:57:39.937510 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:39.934493 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a70bbb8b-7d34-488e-bd32-fa463616f2e7-metrics-certs" pod "a70bbb8b-7d34-488e-bd32-fa463616f2e7" (UID: "a70bbb8b-7d34-488e-bd32-fa463616f2e7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:57:39.970720 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:39.969807 4562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a70bbb8b-7d34-488e-bd32-fa463616f2e7-maas-api-tls\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:57:39.970720 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:39.969835 4562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a70bbb8b-7d34-488e-bd32-fa463616f2e7-metrics-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:57:39.970720 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:39.969848 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvh6v\" (UniqueName: \"kubernetes.io/projected/a70bbb8b-7d34-488e-bd32-fa463616f2e7-kube-api-access-lvh6v\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:57:40.091013 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:40.090926 4562 generic.go:356] "Generic (PLEG): container finished" podID="a70bbb8b-7d34-488e-bd32-fa463616f2e7" containerID="a7d2ccfe44e7c50d45db7e518245b48ccb2a192e868810bc37454627b691224f" exitCode=0 Oct 30 09:57:40.091013 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:40.090992 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e3d2ff-78f84d76-w6jp2" Oct 30 09:57:40.091249 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:40.091006 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e3d2ff-78f84d76-w6jp2" event={"ID":"a70bbb8b-7d34-488e-bd32-fa463616f2e7","Type":"ContainerDied","Data":"a7d2ccfe44e7c50d45db7e518245b48ccb2a192e868810bc37454627b691224f"} Oct 30 09:57:40.091249 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:40.091088 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e3d2ff-78f84d76-w6jp2" event={"ID":"a70bbb8b-7d34-488e-bd32-fa463616f2e7","Type":"ContainerDied","Data":"3c1e95a0d0a354e9f853c992f3395149ab9219305545fd631a9e251a62a61706"} Oct 30 09:57:40.091249 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:40.091112 4562 scope.go:122] "RemoveContainer" containerID="a7d2ccfe44e7c50d45db7e518245b48ccb2a192e868810bc37454627b691224f" Oct 30 09:57:40.102930 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:40.102911 4562 scope.go:122] "RemoveContainer" containerID="a7d2ccfe44e7c50d45db7e518245b48ccb2a192e868810bc37454627b691224f" Oct 30 09:57:40.103213 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:57:40.103194 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7d2ccfe44e7c50d45db7e518245b48ccb2a192e868810bc37454627b691224f\": container with ID starting with a7d2ccfe44e7c50d45db7e518245b48ccb2a192e868810bc37454627b691224f not found: ID does not exist" containerID="a7d2ccfe44e7c50d45db7e518245b48ccb2a192e868810bc37454627b691224f" Oct 30 09:57:40.103294 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:40.103222 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7d2ccfe44e7c50d45db7e518245b48ccb2a192e868810bc37454627b691224f"} err="failed to get container status \"a7d2ccfe44e7c50d45db7e518245b48ccb2a192e868810bc37454627b691224f\": rpc error: code = NotFound desc = could not find container \"a7d2ccfe44e7c50d45db7e518245b48ccb2a192e868810bc37454627b691224f\": container with ID starting with a7d2ccfe44e7c50d45db7e518245b48ccb2a192e868810bc37454627b691224f not found: ID does not exist" Oct 30 09:57:40.149162 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:40.149133 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-e3d2ff-78f84d76-w6jp2"] Oct 30 09:57:40.152078 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:40.152056 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-e3d2ff-78f84d76-w6jp2"] Oct 30 09:57:40.999984 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:40.999944 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-2f6ee2-844666dcbd-56ppf" podUID="36c4eb3b-f628-48a2-82b6-aa1826bbf0c6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:57:41.000521 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:41.000123 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f6ee2-684d557c9-gf6cq" podUID="78e55c08-322e-464e-a798-2d3b2d894972" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:57:41.714638 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:41.714610 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2f6ee2-844666dcbd-56ppf" Oct 30 09:57:41.750775 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:41.750743 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f6ee2-684d557c9-gf6cq" Oct 30 09:57:41.796883 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:41.796835 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78e55c08-322e-464e-a798-2d3b2d894972-plugins-config-volume" pod "78e55c08-322e-464e-a798-2d3b2d894972" (UID: "78e55c08-322e-464e-a798-2d3b2d894972"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:57:41.797074 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:41.796328 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78e55c08-322e-464e-a798-2d3b2d894972-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78e55c08-322e-464e-a798-2d3b2d894972-plugins-config-volume\") pod \"78e55c08-322e-464e-a798-2d3b2d894972\" (UID: \"78e55c08-322e-464e-a798-2d3b2d894972\") " Oct 30 09:57:41.798081 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:41.798053 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36c4eb3b-f628-48a2-82b6-aa1826bbf0c6-kube-api-access-xr8p9\" (UniqueName: \"kubernetes.io/projected/36c4eb3b-f628-48a2-82b6-aa1826bbf0c6-kube-api-access-xr8p9\") pod \"36c4eb3b-f628-48a2-82b6-aa1826bbf0c6\" (UID: \"36c4eb3b-f628-48a2-82b6-aa1826bbf0c6\") " Oct 30 09:57:41.798852 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:41.798816 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36c4eb3b-f628-48a2-82b6-aa1826bbf0c6-plugins-config-volume" pod "36c4eb3b-f628-48a2-82b6-aa1826bbf0c6" (UID: "36c4eb3b-f628-48a2-82b6-aa1826bbf0c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:57:41.799039 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:41.799023 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36c4eb3b-f628-48a2-82b6-aa1826bbf0c6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36c4eb3b-f628-48a2-82b6-aa1826bbf0c6-plugins-config-volume\") pod \"36c4eb3b-f628-48a2-82b6-aa1826bbf0c6\" (UID: \"36c4eb3b-f628-48a2-82b6-aa1826bbf0c6\") " Oct 30 09:57:41.799290 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:41.799273 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78e55c08-322e-464e-a798-2d3b2d894972-kube-api-access-qrqbn\" (UniqueName: \"kubernetes.io/projected/78e55c08-322e-464e-a798-2d3b2d894972-kube-api-access-qrqbn\") pod \"78e55c08-322e-464e-a798-2d3b2d894972\" (UID: \"78e55c08-322e-464e-a798-2d3b2d894972\") " Oct 30 09:57:41.801675 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:41.801650 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78e55c08-322e-464e-a798-2d3b2d894972-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:57:41.801816 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:41.801802 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36c4eb3b-f628-48a2-82b6-aa1826bbf0c6-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:57:41.801976 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:41.801949 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36c4eb3b-f628-48a2-82b6-aa1826bbf0c6-kube-api-access-xr8p9" pod "36c4eb3b-f628-48a2-82b6-aa1826bbf0c6" (UID: "36c4eb3b-f628-48a2-82b6-aa1826bbf0c6"). InnerVolumeSpecName "kube-api-access-xr8p9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:57:41.807751 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:41.807689 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78e55c08-322e-464e-a798-2d3b2d894972-kube-api-access-qrqbn" pod "78e55c08-322e-464e-a798-2d3b2d894972" (UID: "78e55c08-322e-464e-a798-2d3b2d894972"). InnerVolumeSpecName "kube-api-access-qrqbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:57:41.875053 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:41.875021 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a70bbb8b-7d34-488e-bd32-fa463616f2e7" path="/var/lib/kubelet/pods/a70bbb8b-7d34-488e-bd32-fa463616f2e7/volumes" Oct 30 09:57:41.903639 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:41.903608 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr8p9\" (UniqueName: \"kubernetes.io/projected/36c4eb3b-f628-48a2-82b6-aa1826bbf0c6-kube-api-access-xr8p9\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:57:41.903639 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:41.903633 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrqbn\" (UniqueName: \"kubernetes.io/projected/78e55c08-322e-464e-a798-2d3b2d894972-kube-api-access-qrqbn\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:57:42.115100 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:42.115066 4562 generic.go:356] "Generic (PLEG): container finished" podID="78e55c08-322e-464e-a798-2d3b2d894972" containerID="17b2b9376656089f75041188893d68877fd15b9e9719194dfde5615eb2037c5b" exitCode=0 Oct 30 09:57:42.115633 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:42.115125 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f6ee2-684d557c9-gf6cq" Oct 30 09:57:42.115633 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:42.115151 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f6ee2-684d557c9-gf6cq" event={"ID":"78e55c08-322e-464e-a798-2d3b2d894972","Type":"ContainerDied","Data":"17b2b9376656089f75041188893d68877fd15b9e9719194dfde5615eb2037c5b"} Oct 30 09:57:42.115633 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:42.115210 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f6ee2-684d557c9-gf6cq" event={"ID":"78e55c08-322e-464e-a798-2d3b2d894972","Type":"ContainerDied","Data":"cea4333656c8e203ea2ba3de09577bc4575aceff77de2b803867bd798018488e"} Oct 30 09:57:42.115633 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:42.115232 4562 scope.go:122] "RemoveContainer" containerID="17b2b9376656089f75041188893d68877fd15b9e9719194dfde5615eb2037c5b" Oct 30 09:57:42.116875 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:42.116846 4562 generic.go:356] "Generic (PLEG): container finished" podID="36c4eb3b-f628-48a2-82b6-aa1826bbf0c6" containerID="8ca6619bf30eedb4671b3a0f00b5f91ba7721b029a3e1678ed45b93513584e02" exitCode=0 Oct 30 09:57:42.117008 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:42.116904 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2f6ee2-844666dcbd-56ppf" Oct 30 09:57:42.117008 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:42.116929 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2f6ee2-844666dcbd-56ppf" event={"ID":"36c4eb3b-f628-48a2-82b6-aa1826bbf0c6","Type":"ContainerDied","Data":"8ca6619bf30eedb4671b3a0f00b5f91ba7721b029a3e1678ed45b93513584e02"} Oct 30 09:57:42.117008 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:42.116969 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2f6ee2-844666dcbd-56ppf" event={"ID":"36c4eb3b-f628-48a2-82b6-aa1826bbf0c6","Type":"ContainerDied","Data":"80736d71329f2a29dd13f1804aeb633be5fdd354e6f25006f9f565a557dafb44"} Oct 30 09:57:42.126240 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:42.126220 4562 scope.go:122] "RemoveContainer" containerID="17b2b9376656089f75041188893d68877fd15b9e9719194dfde5615eb2037c5b" Oct 30 09:57:42.126589 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:57:42.126561 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17b2b9376656089f75041188893d68877fd15b9e9719194dfde5615eb2037c5b\": container with ID starting with 17b2b9376656089f75041188893d68877fd15b9e9719194dfde5615eb2037c5b not found: ID does not exist" containerID="17b2b9376656089f75041188893d68877fd15b9e9719194dfde5615eb2037c5b" Oct 30 09:57:42.126691 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:42.126600 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17b2b9376656089f75041188893d68877fd15b9e9719194dfde5615eb2037c5b"} err="failed to get container status \"17b2b9376656089f75041188893d68877fd15b9e9719194dfde5615eb2037c5b\": rpc error: code = NotFound desc = could not find container \"17b2b9376656089f75041188893d68877fd15b9e9719194dfde5615eb2037c5b\": container with ID starting with 17b2b9376656089f75041188893d68877fd15b9e9719194dfde5615eb2037c5b not found: ID does not exist" Oct 30 09:57:42.126691 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:42.126624 4562 scope.go:122] "RemoveContainer" containerID="8ca6619bf30eedb4671b3a0f00b5f91ba7721b029a3e1678ed45b93513584e02" Oct 30 09:57:42.136203 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:42.136180 4562 scope.go:122] "RemoveContainer" containerID="8ca6619bf30eedb4671b3a0f00b5f91ba7721b029a3e1678ed45b93513584e02" Oct 30 09:57:42.136600 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:57:42.136575 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ca6619bf30eedb4671b3a0f00b5f91ba7721b029a3e1678ed45b93513584e02\": container with ID starting with 8ca6619bf30eedb4671b3a0f00b5f91ba7721b029a3e1678ed45b93513584e02 not found: ID does not exist" containerID="8ca6619bf30eedb4671b3a0f00b5f91ba7721b029a3e1678ed45b93513584e02" Oct 30 09:57:42.136676 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:42.136610 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ca6619bf30eedb4671b3a0f00b5f91ba7721b029a3e1678ed45b93513584e02"} err="failed to get container status \"8ca6619bf30eedb4671b3a0f00b5f91ba7721b029a3e1678ed45b93513584e02\": rpc error: code = NotFound desc = could not find container \"8ca6619bf30eedb4671b3a0f00b5f91ba7721b029a3e1678ed45b93513584e02\": container with ID starting with 8ca6619bf30eedb4671b3a0f00b5f91ba7721b029a3e1678ed45b93513584e02 not found: ID does not exist" Oct 30 09:57:42.168873 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:42.168833 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2f6ee2-684d557c9-gf6cq"] Oct 30 09:57:42.172188 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:42.172165 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2f6ee2-684d557c9-gf6cq"] Oct 30 09:57:42.206288 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:42.206248 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2f6ee2-844666dcbd-56ppf"] Oct 30 09:57:42.209333 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:42.209307 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2f6ee2-844666dcbd-56ppf"] Oct 30 09:57:42.742095 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:42.742056 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-bdd208-784594d77d-z6qzr" podUID="dcea2042-49d4-4703-a59c-fdb378c3299c" containerName="payload-processing" containerID="cri-o://b4175ed8790dea968d2520bb5f7b0c432e370d743980fe5b2de34c5d0bf9d198" gracePeriod=25 Oct 30 09:57:42.792617 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:42.792561 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bdd208-6dff5fbd47-69qxt" podUID="e4f0d99e-ad40-4806-8cda-148b2ba50945" containerName="payload-pre-processing" containerID="cri-o://746442c8724d9cf10a1839423c6052d8110106277aee6f024a59df1e2508ca3f" gracePeriod=25 Oct 30 09:57:42.924718 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:42.924650 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bdd208-6dff5fbd47-69qxt" podUID="e4f0d99e-ad40-4806-8cda-148b2ba50945" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:57:43.015341 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:43.015312 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-bdd208-784594d77d-z6qzr" Oct 30 09:57:43.123424 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:43.123374 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcea2042-49d4-4703-a59c-fdb378c3299c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcea2042-49d4-4703-a59c-fdb378c3299c-plugins-config-volume\") pod \"dcea2042-49d4-4703-a59c-fdb378c3299c\" (UID: \"dcea2042-49d4-4703-a59c-fdb378c3299c\") " Oct 30 09:57:43.123930 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:43.123605 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcea2042-49d4-4703-a59c-fdb378c3299c-kube-api-access-c26vn\" (UniqueName: \"kubernetes.io/projected/dcea2042-49d4-4703-a59c-fdb378c3299c-kube-api-access-c26vn\") pod \"dcea2042-49d4-4703-a59c-fdb378c3299c\" (UID: \"dcea2042-49d4-4703-a59c-fdb378c3299c\") " Oct 30 09:57:43.123930 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:43.123755 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcea2042-49d4-4703-a59c-fdb378c3299c-plugins-config-volume" pod "dcea2042-49d4-4703-a59c-fdb378c3299c" (UID: "dcea2042-49d4-4703-a59c-fdb378c3299c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:57:43.124174 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:43.124143 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcea2042-49d4-4703-a59c-fdb378c3299c-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:57:43.125233 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:43.125206 4562 generic.go:356] "Generic (PLEG): container finished" podID="dcea2042-49d4-4703-a59c-fdb378c3299c" containerID="b4175ed8790dea968d2520bb5f7b0c432e370d743980fe5b2de34c5d0bf9d198" exitCode=0 Oct 30 09:57:43.125367 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:43.125249 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-bdd208-784594d77d-z6qzr" Oct 30 09:57:43.125367 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:43.125289 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-bdd208-784594d77d-z6qzr" event={"ID":"dcea2042-49d4-4703-a59c-fdb378c3299c","Type":"ContainerDied","Data":"b4175ed8790dea968d2520bb5f7b0c432e370d743980fe5b2de34c5d0bf9d198"} Oct 30 09:57:43.125367 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:43.125347 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-bdd208-784594d77d-z6qzr" event={"ID":"dcea2042-49d4-4703-a59c-fdb378c3299c","Type":"ContainerDied","Data":"44572447af7542973d8d65589213accb19ecdbbdebf5642636e4bd74a650dd70"} Oct 30 09:57:43.125629 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:43.125374 4562 scope.go:122] "RemoveContainer" containerID="b4175ed8790dea968d2520bb5f7b0c432e370d743980fe5b2de34c5d0bf9d198" Oct 30 09:57:43.126230 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:43.126204 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcea2042-49d4-4703-a59c-fdb378c3299c-kube-api-access-c26vn" pod "dcea2042-49d4-4703-a59c-fdb378c3299c" (UID: "dcea2042-49d4-4703-a59c-fdb378c3299c"). InnerVolumeSpecName "kube-api-access-c26vn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:57:43.142328 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:43.142307 4562 scope.go:122] "RemoveContainer" containerID="b4175ed8790dea968d2520bb5f7b0c432e370d743980fe5b2de34c5d0bf9d198" Oct 30 09:57:43.142670 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:57:43.142650 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4175ed8790dea968d2520bb5f7b0c432e370d743980fe5b2de34c5d0bf9d198\": container with ID starting with b4175ed8790dea968d2520bb5f7b0c432e370d743980fe5b2de34c5d0bf9d198 not found: ID does not exist" containerID="b4175ed8790dea968d2520bb5f7b0c432e370d743980fe5b2de34c5d0bf9d198" Oct 30 09:57:43.142757 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:43.142686 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4175ed8790dea968d2520bb5f7b0c432e370d743980fe5b2de34c5d0bf9d198"} err="failed to get container status \"b4175ed8790dea968d2520bb5f7b0c432e370d743980fe5b2de34c5d0bf9d198\": rpc error: code = NotFound desc = could not find container \"b4175ed8790dea968d2520bb5f7b0c432e370d743980fe5b2de34c5d0bf9d198\": container with ID starting with b4175ed8790dea968d2520bb5f7b0c432e370d743980fe5b2de34c5d0bf9d198 not found: ID does not exist" Oct 30 09:57:43.225318 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:43.225243 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c26vn\" (UniqueName: \"kubernetes.io/projected/dcea2042-49d4-4703-a59c-fdb378c3299c-kube-api-access-c26vn\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:57:43.476068 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:43.476035 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-bdd208-784594d77d-z6qzr"] Oct 30 09:57:43.489192 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:43.489156 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-bdd208-784594d77d-z6qzr"] Oct 30 09:57:43.875132 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:43.875091 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36c4eb3b-f628-48a2-82b6-aa1826bbf0c6" path="/var/lib/kubelet/pods/36c4eb3b-f628-48a2-82b6-aa1826bbf0c6/volumes" Oct 30 09:57:43.875583 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:43.875566 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78e55c08-322e-464e-a798-2d3b2d894972" path="/var/lib/kubelet/pods/78e55c08-322e-464e-a798-2d3b2d894972/volumes" Oct 30 09:57:43.875949 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:43.875934 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcea2042-49d4-4703-a59c-fdb378c3299c" path="/var/lib/kubelet/pods/dcea2042-49d4-4703-a59c-fdb378c3299c/volumes" Oct 30 09:57:43.922340 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:43.922302 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-bdd208-784594d77d-z6qzr" podUID="dcea2042-49d4-4703-a59c-fdb378c3299c" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.243:50052\" within 1s: context deadline exceeded" Oct 30 09:57:43.922499 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:57:43.922375 4562 logging.go:55] [core] [Channel #16022 SubChannel #16024]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.243:50052", ServerName: "10.133.0.243:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.243:50052: operation was canceled" Oct 30 09:57:47.440264 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:47.440228 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw"] Oct 30 09:57:47.441082 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:47.441047 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw" podUID="ecc12bf1-47c5-4d5e-b295-079f027d8019" containerName="istio-proxy" containerID="cri-o://c2b900f0f688c57b8463d6b13a51e3f17e8855adb654dd1deda65057d159acb6" gracePeriod=30 Oct 30 09:57:50.347182 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:50.347134 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-176b6c-65457966fb-55v76"] Oct 30 09:57:50.347738 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:50.347416 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-176b6c-65457966fb-55v76" podUID="bd2e3358-d08c-4e6d-b0b4-1825157b40fc" containerName="maas-api" containerID="cri-o://8fe4fb58282d6b704766ee1376c893e237d884440471d19bbeb1c8349edd998d" gracePeriod=30 Oct 30 09:57:50.674434 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:50.674408 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-176b6c-65457966fb-55v76" Oct 30 09:57:50.775406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:50.763996 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd2e3358-d08c-4e6d-b0b4-1825157b40fc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd2e3358-d08c-4e6d-b0b4-1825157b40fc-metrics-certs\") pod \"bd2e3358-d08c-4e6d-b0b4-1825157b40fc\" (UID: \"bd2e3358-d08c-4e6d-b0b4-1825157b40fc\") " Oct 30 09:57:50.775406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:50.764071 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd2e3358-d08c-4e6d-b0b4-1825157b40fc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd2e3358-d08c-4e6d-b0b4-1825157b40fc-openshift-service-ca\") pod \"bd2e3358-d08c-4e6d-b0b4-1825157b40fc\" (UID: \"bd2e3358-d08c-4e6d-b0b4-1825157b40fc\") " Oct 30 09:57:50.775406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:50.764108 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd2e3358-d08c-4e6d-b0b4-1825157b40fc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd2e3358-d08c-4e6d-b0b4-1825157b40fc-maas-api-tls\") pod \"bd2e3358-d08c-4e6d-b0b4-1825157b40fc\" (UID: \"bd2e3358-d08c-4e6d-b0b4-1825157b40fc\") " Oct 30 09:57:50.775406 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:50.764350 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd2e3358-d08c-4e6d-b0b4-1825157b40fc-kube-api-access-pvf46\" (UniqueName: \"kubernetes.io/projected/bd2e3358-d08c-4e6d-b0b4-1825157b40fc-kube-api-access-pvf46\") pod \"bd2e3358-d08c-4e6d-b0b4-1825157b40fc\" (UID: \"bd2e3358-d08c-4e6d-b0b4-1825157b40fc\") " Oct 30 09:57:50.808865 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:50.808797 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd2e3358-d08c-4e6d-b0b4-1825157b40fc-maas-api-tls" pod "bd2e3358-d08c-4e6d-b0b4-1825157b40fc" (UID: "bd2e3358-d08c-4e6d-b0b4-1825157b40fc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:57:50.809425 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:50.809362 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd2e3358-d08c-4e6d-b0b4-1825157b40fc-openshift-service-ca" pod "bd2e3358-d08c-4e6d-b0b4-1825157b40fc" (UID: "bd2e3358-d08c-4e6d-b0b4-1825157b40fc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:57:50.819132 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:50.819059 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd2e3358-d08c-4e6d-b0b4-1825157b40fc-kube-api-access-pvf46" pod "bd2e3358-d08c-4e6d-b0b4-1825157b40fc" (UID: "bd2e3358-d08c-4e6d-b0b4-1825157b40fc"). InnerVolumeSpecName "kube-api-access-pvf46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:57:50.819369 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:50.819343 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd2e3358-d08c-4e6d-b0b4-1825157b40fc-metrics-certs" pod "bd2e3358-d08c-4e6d-b0b4-1825157b40fc" (UID: "bd2e3358-d08c-4e6d-b0b4-1825157b40fc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:57:50.866039 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:50.865994 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvf46\" (UniqueName: \"kubernetes.io/projected/bd2e3358-d08c-4e6d-b0b4-1825157b40fc-kube-api-access-pvf46\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:57:50.866039 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:50.866037 4562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd2e3358-d08c-4e6d-b0b4-1825157b40fc-metrics-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:57:50.866039 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:50.866049 4562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd2e3358-d08c-4e6d-b0b4-1825157b40fc-openshift-service-ca\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:57:50.866039 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:50.866057 4562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd2e3358-d08c-4e6d-b0b4-1825157b40fc-maas-api-tls\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:57:51.184715 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:51.184666 4562 generic.go:356] "Generic (PLEG): container finished" podID="bd2e3358-d08c-4e6d-b0b4-1825157b40fc" containerID="8fe4fb58282d6b704766ee1376c893e237d884440471d19bbeb1c8349edd998d" exitCode=0 Oct 30 09:57:51.184906 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:51.184727 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-176b6c-65457966fb-55v76" event={"ID":"bd2e3358-d08c-4e6d-b0b4-1825157b40fc","Type":"ContainerDied","Data":"8fe4fb58282d6b704766ee1376c893e237d884440471d19bbeb1c8349edd998d"} Oct 30 09:57:51.184906 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:51.184781 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-176b6c-65457966fb-55v76" event={"ID":"bd2e3358-d08c-4e6d-b0b4-1825157b40fc","Type":"ContainerDied","Data":"6c1cf211080cb515c1dff27e197fc7260af1b70065d313350d15a4fd96590ecb"} Oct 30 09:57:51.184906 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:51.184811 4562 scope.go:122] "RemoveContainer" containerID="8fe4fb58282d6b704766ee1376c893e237d884440471d19bbeb1c8349edd998d" Oct 30 09:57:51.184906 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:51.184740 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-176b6c-65457966fb-55v76" Oct 30 09:57:51.196694 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:51.196672 4562 scope.go:122] "RemoveContainer" containerID="8fe4fb58282d6b704766ee1376c893e237d884440471d19bbeb1c8349edd998d" Oct 30 09:57:51.196978 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:57:51.196958 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fe4fb58282d6b704766ee1376c893e237d884440471d19bbeb1c8349edd998d\": container with ID starting with 8fe4fb58282d6b704766ee1376c893e237d884440471d19bbeb1c8349edd998d not found: ID does not exist" containerID="8fe4fb58282d6b704766ee1376c893e237d884440471d19bbeb1c8349edd998d" Oct 30 09:57:51.197024 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:51.196988 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fe4fb58282d6b704766ee1376c893e237d884440471d19bbeb1c8349edd998d"} err="failed to get container status \"8fe4fb58282d6b704766ee1376c893e237d884440471d19bbeb1c8349edd998d\": rpc error: code = NotFound desc = could not find container \"8fe4fb58282d6b704766ee1376c893e237d884440471d19bbeb1c8349edd998d\": container with ID starting with 8fe4fb58282d6b704766ee1376c893e237d884440471d19bbeb1c8349edd998d not found: ID does not exist" Oct 30 09:57:51.238484 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:51.238457 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-176b6c-65457966fb-55v76"] Oct 30 09:57:51.241667 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:51.241641 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-176b6c-65457966fb-55v76"] Oct 30 09:57:51.876001 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:51.875965 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd2e3358-d08c-4e6d-b0b4-1825157b40fc" path="/var/lib/kubelet/pods/bd2e3358-d08c-4e6d-b0b4-1825157b40fc/volumes" Oct 30 09:57:52.788995 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:52.788965 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw" Oct 30 09:57:52.906113 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:52.906066 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecc12bf1-47c5-4d5e-b295-079f027d8019-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecc12bf1-47c5-4d5e-b295-079f027d8019-workload-certs\") pod \"ecc12bf1-47c5-4d5e-b295-079f027d8019\" (UID: \"ecc12bf1-47c5-4d5e-b295-079f027d8019\") " Oct 30 09:57:52.906606 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:52.906216 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecc12bf1-47c5-4d5e-b295-079f027d8019-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecc12bf1-47c5-4d5e-b295-079f027d8019-istiod-ca-cert\") pod \"ecc12bf1-47c5-4d5e-b295-079f027d8019\" (UID: \"ecc12bf1-47c5-4d5e-b295-079f027d8019\") " Oct 30 09:57:52.906606 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:52.906251 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ecc12bf1-47c5-4d5e-b295-079f027d8019-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecc12bf1-47c5-4d5e-b295-079f027d8019-istio-podinfo\") pod \"ecc12bf1-47c5-4d5e-b295-079f027d8019\" (UID: \"ecc12bf1-47c5-4d5e-b295-079f027d8019\") " Oct 30 09:57:52.906606 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:52.906312 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecc12bf1-47c5-4d5e-b295-079f027d8019-istio-token\" (UniqueName: \"kubernetes.io/projected/ecc12bf1-47c5-4d5e-b295-079f027d8019-istio-token\") pod \"ecc12bf1-47c5-4d5e-b295-079f027d8019\" (UID: \"ecc12bf1-47c5-4d5e-b295-079f027d8019\") " Oct 30 09:57:52.906606 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:52.906406 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecc12bf1-47c5-4d5e-b295-079f027d8019-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc12bf1-47c5-4d5e-b295-079f027d8019-credential-socket\") pod \"ecc12bf1-47c5-4d5e-b295-079f027d8019\" (UID: \"ecc12bf1-47c5-4d5e-b295-079f027d8019\") " Oct 30 09:57:52.906606 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:52.906458 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecc12bf1-47c5-4d5e-b295-079f027d8019-kube-api-access-bb6rz\" (UniqueName: \"kubernetes.io/projected/ecc12bf1-47c5-4d5e-b295-079f027d8019-kube-api-access-bb6rz\") pod \"ecc12bf1-47c5-4d5e-b295-079f027d8019\" (UID: \"ecc12bf1-47c5-4d5e-b295-079f027d8019\") " Oct 30 09:57:52.906606 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:52.906540 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecc12bf1-47c5-4d5e-b295-079f027d8019-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecc12bf1-47c5-4d5e-b295-079f027d8019-istio-data\") pod \"ecc12bf1-47c5-4d5e-b295-079f027d8019\" (UID: \"ecc12bf1-47c5-4d5e-b295-079f027d8019\") " Oct 30 09:57:52.906975 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:52.906600 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecc12bf1-47c5-4d5e-b295-079f027d8019-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecc12bf1-47c5-4d5e-b295-079f027d8019-istio-envoy\") pod \"ecc12bf1-47c5-4d5e-b295-079f027d8019\" (UID: \"ecc12bf1-47c5-4d5e-b295-079f027d8019\") " Oct 30 09:57:52.906975 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:52.906654 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecc12bf1-47c5-4d5e-b295-079f027d8019-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc12bf1-47c5-4d5e-b295-079f027d8019-workload-socket\") pod \"ecc12bf1-47c5-4d5e-b295-079f027d8019\" (UID: \"ecc12bf1-47c5-4d5e-b295-079f027d8019\") " Oct 30 09:57:52.907613 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:52.907558 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecc12bf1-47c5-4d5e-b295-079f027d8019-workload-socket" pod "ecc12bf1-47c5-4d5e-b295-079f027d8019" (UID: "ecc12bf1-47c5-4d5e-b295-079f027d8019"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:52.929417 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:52.928846 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecc12bf1-47c5-4d5e-b295-079f027d8019-istiod-ca-cert" pod "ecc12bf1-47c5-4d5e-b295-079f027d8019" (UID: "ecc12bf1-47c5-4d5e-b295-079f027d8019"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:57:52.935808 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:52.935451 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ecc12bf1-47c5-4d5e-b295-079f027d8019-istio-podinfo" pod "ecc12bf1-47c5-4d5e-b295-079f027d8019" (UID: "ecc12bf1-47c5-4d5e-b295-079f027d8019"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:57:52.938369 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:52.938173 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecc12bf1-47c5-4d5e-b295-079f027d8019-istio-token" pod "ecc12bf1-47c5-4d5e-b295-079f027d8019" (UID: "ecc12bf1-47c5-4d5e-b295-079f027d8019"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:57:52.940088 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:52.939816 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecc12bf1-47c5-4d5e-b295-079f027d8019-credential-socket" pod "ecc12bf1-47c5-4d5e-b295-079f027d8019" (UID: "ecc12bf1-47c5-4d5e-b295-079f027d8019"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:52.942527 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:52.942295 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecc12bf1-47c5-4d5e-b295-079f027d8019-kube-api-access-bb6rz" pod "ecc12bf1-47c5-4d5e-b295-079f027d8019" (UID: "ecc12bf1-47c5-4d5e-b295-079f027d8019"). InnerVolumeSpecName "kube-api-access-bb6rz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:57:52.942971 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:52.942696 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecc12bf1-47c5-4d5e-b295-079f027d8019-istio-data" pod "ecc12bf1-47c5-4d5e-b295-079f027d8019" (UID: "ecc12bf1-47c5-4d5e-b295-079f027d8019"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:52.951695 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:52.907920 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecc12bf1-47c5-4d5e-b295-079f027d8019-workload-certs" pod "ecc12bf1-47c5-4d5e-b295-079f027d8019" (UID: "ecc12bf1-47c5-4d5e-b295-079f027d8019"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:52.951934 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:52.951883 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecc12bf1-47c5-4d5e-b295-079f027d8019-istio-envoy" pod "ecc12bf1-47c5-4d5e-b295-079f027d8019" (UID: "ecc12bf1-47c5-4d5e-b295-079f027d8019"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:57:52.973067 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:52.972660 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bdd208-6dff5fbd47-69qxt" podUID="e4f0d99e-ad40-4806-8cda-148b2ba50945" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:57:53.008578 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:53.008493 4562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecc12bf1-47c5-4d5e-b295-079f027d8019-istio-data\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:57:53.008578 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:53.008522 4562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecc12bf1-47c5-4d5e-b295-079f027d8019-istio-envoy\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:57:53.008578 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:53.008531 4562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc12bf1-47c5-4d5e-b295-079f027d8019-workload-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:57:53.008578 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:53.008540 4562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecc12bf1-47c5-4d5e-b295-079f027d8019-workload-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:57:53.008578 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:53.008548 4562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecc12bf1-47c5-4d5e-b295-079f027d8019-istiod-ca-cert\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:57:53.008578 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:53.008556 4562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecc12bf1-47c5-4d5e-b295-079f027d8019-istio-podinfo\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:57:53.008578 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:53.008563 4562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecc12bf1-47c5-4d5e-b295-079f027d8019-istio-token\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:57:53.008578 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:53.008573 4562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc12bf1-47c5-4d5e-b295-079f027d8019-credential-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:57:53.008944 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:53.008588 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bb6rz\" (UniqueName: \"kubernetes.io/projected/ecc12bf1-47c5-4d5e-b295-079f027d8019-kube-api-access-bb6rz\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:57:53.201540 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:53.201509 4562 generic.go:356] "Generic (PLEG): container finished" podID="ecc12bf1-47c5-4d5e-b295-079f027d8019" containerID="c2b900f0f688c57b8463d6b13a51e3f17e8855adb654dd1deda65057d159acb6" exitCode=0 Oct 30 09:57:53.201736 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:53.201605 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw" Oct 30 09:57:53.201736 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:53.201645 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw" event={"ID":"ecc12bf1-47c5-4d5e-b295-079f027d8019","Type":"ContainerDied","Data":"c2b900f0f688c57b8463d6b13a51e3f17e8855adb654dd1deda65057d159acb6"} Oct 30 09:57:53.201736 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:53.201715 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw" event={"ID":"ecc12bf1-47c5-4d5e-b295-079f027d8019","Type":"ContainerDied","Data":"4c3cfd6be84834cd152104098304003f668de00a7f760021ced3b68746176af3"} Oct 30 09:57:53.201921 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:53.201761 4562 scope.go:122] "RemoveContainer" containerID="c2b900f0f688c57b8463d6b13a51e3f17e8855adb654dd1deda65057d159acb6" Oct 30 09:57:53.220050 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:53.220026 4562 scope.go:122] "RemoveContainer" containerID="c2b900f0f688c57b8463d6b13a51e3f17e8855adb654dd1deda65057d159acb6" Oct 30 09:57:53.220657 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:57:53.220630 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2b900f0f688c57b8463d6b13a51e3f17e8855adb654dd1deda65057d159acb6\": container with ID starting with c2b900f0f688c57b8463d6b13a51e3f17e8855adb654dd1deda65057d159acb6 not found: ID does not exist" containerID="c2b900f0f688c57b8463d6b13a51e3f17e8855adb654dd1deda65057d159acb6" Oct 30 09:57:53.223877 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:53.223835 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2b900f0f688c57b8463d6b13a51e3f17e8855adb654dd1deda65057d159acb6"} err="failed to get container status \"c2b900f0f688c57b8463d6b13a51e3f17e8855adb654dd1deda65057d159acb6\": rpc error: code = NotFound desc = could not find container \"c2b900f0f688c57b8463d6b13a51e3f17e8855adb654dd1deda65057d159acb6\": container with ID starting with c2b900f0f688c57b8463d6b13a51e3f17e8855adb654dd1deda65057d159acb6 not found: ID does not exist" Oct 30 09:57:53.260583 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:53.260505 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw"] Oct 30 09:57:53.265101 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:53.265071 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e3d2ff-openshift-default-77d9c5757-j7rjw"] Oct 30 09:57:53.884345 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:53.884308 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecc12bf1-47c5-4d5e-b295-079f027d8019" path="/var/lib/kubelet/pods/ecc12bf1-47c5-4d5e-b295-079f027d8019/volumes" Oct 30 09:57:55.933594 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:55.933560 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz"] Oct 30 09:57:55.933998 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:55.933813 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz" podUID="04c56890-74da-4365-b273-13751658dd1a" containerName="istio-proxy" containerID="cri-o://b68d33df98d10d578a9b113d5168530e948f5655067a56e110ba6421856728a4" gracePeriod=30 Oct 30 09:57:56.549504 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:56.549462 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ae795b-sgwx6"] Oct 30 09:57:56.550043 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:56.550029 4562 cpu_manager.go:414] "containerMap: removing container" podUID="28fd4072-399c-4be5-a0d8-1be6966da4af" containerName="main" Oct 30 09:57:56.550098 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:56.550044 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28fd4072-399c-4be5-a0d8-1be6966da4af" containerName="main" Oct 30 09:57:56.550098 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:56.550059 4562 cpu_manager.go:414] "containerMap: removing container" podUID="bd2e3358-d08c-4e6d-b0b4-1825157b40fc" containerName="maas-api" Oct 30 09:57:56.550098 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:56.550064 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd2e3358-d08c-4e6d-b0b4-1825157b40fc" containerName="maas-api" Oct 30 09:57:56.550098 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:56.550075 4562 cpu_manager.go:414] "containerMap: removing container" podUID="78e55c08-322e-464e-a798-2d3b2d894972" containerName="payload-pre-processing" Oct 30 09:57:56.550098 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:56.550082 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78e55c08-322e-464e-a798-2d3b2d894972" containerName="payload-pre-processing" Oct 30 09:57:56.550098 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:56.550096 4562 cpu_manager.go:414] "containerMap: removing container" podUID="36c4eb3b-f628-48a2-82b6-aa1826bbf0c6" containerName="payload-processing" Oct 30 09:57:56.550098 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:56.550100 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36c4eb3b-f628-48a2-82b6-aa1826bbf0c6" containerName="payload-processing" Oct 30 09:57:56.550327 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:56.550115 4562 cpu_manager.go:414] "containerMap: removing container" podUID="c0caf9cd-3ae5-4d1f-9815-71b2310cf896" containerName="revoke-keys" Oct 30 09:57:56.550327 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:56.550120 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0caf9cd-3ae5-4d1f-9815-71b2310cf896" containerName="revoke-keys" Oct 30 09:57:56.550327 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:56.550140 4562 cpu_manager.go:414] "containerMap: removing container" podUID="dcea2042-49d4-4703-a59c-fdb378c3299c" containerName="payload-processing" Oct 30 09:57:56.550327 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:56.550145 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcea2042-49d4-4703-a59c-fdb378c3299c" containerName="payload-processing" Oct 30 09:57:56.550327 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:56.550155 4562 cpu_manager.go:414] "containerMap: removing container" podUID="a70bbb8b-7d34-488e-bd32-fa463616f2e7" containerName="maas-api" Oct 30 09:57:56.550327 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:56.550160 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a70bbb8b-7d34-488e-bd32-fa463616f2e7" containerName="maas-api" Oct 30 09:57:56.550327 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:56.550168 4562 cpu_manager.go:414] "containerMap: removing container" podUID="ecc12bf1-47c5-4d5e-b295-079f027d8019" containerName="istio-proxy" Oct 30 09:57:56.550327 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:56.550172 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecc12bf1-47c5-4d5e-b295-079f027d8019" containerName="istio-proxy" Oct 30 09:57:56.550327 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:56.550263 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0caf9cd-3ae5-4d1f-9815-71b2310cf896" containerName="revoke-keys" Oct 30 09:57:56.550327 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:56.550276 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcea2042-49d4-4703-a59c-fdb378c3299c" containerName="payload-processing" Oct 30 09:57:56.550327 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:56.550281 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="a70bbb8b-7d34-488e-bd32-fa463616f2e7" containerName="maas-api" Oct 30 09:57:56.550327 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:56.550287 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecc12bf1-47c5-4d5e-b295-079f027d8019" containerName="istio-proxy" Oct 30 09:57:56.550327 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:56.550293 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="28fd4072-399c-4be5-a0d8-1be6966da4af" containerName="main" Oct 30 09:57:56.550327 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:56.550299 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd2e3358-d08c-4e6d-b0b4-1825157b40fc" containerName="maas-api" Oct 30 09:57:56.550327 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:56.550304 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="78e55c08-322e-464e-a798-2d3b2d894972" containerName="payload-pre-processing" Oct 30 09:57:56.550327 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:56.550311 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="36c4eb3b-f628-48a2-82b6-aa1826bbf0c6" containerName="payload-processing" Oct 30 09:57:56.554496 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:56.554474 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ae795b-sgwx6" Oct 30 09:57:56.572808 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:56.572778 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ae795b-sgwx6"] Oct 30 09:57:56.590908 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:56.590871 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:57:56.591143 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:56.591055 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:57:56.668482 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:56.668446 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86823ef9-9577-47c3-b6a4-1b50e8f1f3bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-ae795b-sgwx6\" (UID: \"86823ef9-9577-47c3-b6a4-1b50e8f1f3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ae795b-sgwx6" Oct 30 09:57:56.668482 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:56.668490 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4p4z\" (UniqueName: \"kubernetes.io/projected/86823ef9-9577-47c3-b6a4-1b50e8f1f3bf-kube-api-access-w4p4z\") pod \"maas-api-revoke-keys-e2e-extmod-a-ae795b-sgwx6\" (UID: \"86823ef9-9577-47c3-b6a4-1b50e8f1f3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ae795b-sgwx6" Oct 30 09:57:56.668705 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:56.668638 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86823ef9-9577-47c3-b6a4-1b50e8f1f3bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-ae795b-sgwx6\" (UID: \"86823ef9-9577-47c3-b6a4-1b50e8f1f3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ae795b-sgwx6" Oct 30 09:57:56.769947 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:56.769905 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86823ef9-9577-47c3-b6a4-1b50e8f1f3bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-ae795b-sgwx6\" (UID: \"86823ef9-9577-47c3-b6a4-1b50e8f1f3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ae795b-sgwx6" Oct 30 09:57:56.770102 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:56.769960 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4p4z\" (UniqueName: \"kubernetes.io/projected/86823ef9-9577-47c3-b6a4-1b50e8f1f3bf-kube-api-access-w4p4z\") pod \"maas-api-revoke-keys-e2e-extmod-a-ae795b-sgwx6\" (UID: \"86823ef9-9577-47c3-b6a4-1b50e8f1f3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ae795b-sgwx6" Oct 30 09:57:56.770102 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:56.770023 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86823ef9-9577-47c3-b6a4-1b50e8f1f3bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-ae795b-sgwx6\" (UID: \"86823ef9-9577-47c3-b6a4-1b50e8f1f3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ae795b-sgwx6" Oct 30 09:57:56.770672 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:56.770641 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86823ef9-9577-47c3-b6a4-1b50e8f1f3bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-ae795b-sgwx6\" (UID: \"86823ef9-9577-47c3-b6a4-1b50e8f1f3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ae795b-sgwx6" Oct 30 09:57:56.772836 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:56.772789 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86823ef9-9577-47c3-b6a4-1b50e8f1f3bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-ae795b-sgwx6\" (UID: \"86823ef9-9577-47c3-b6a4-1b50e8f1f3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ae795b-sgwx6" Oct 30 09:57:56.789770 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:56.789745 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4p4z\" (UniqueName: \"kubernetes.io/projected/86823ef9-9577-47c3-b6a4-1b50e8f1f3bf-kube-api-access-w4p4z\") pod \"maas-api-revoke-keys-e2e-extmod-a-ae795b-sgwx6\" (UID: \"86823ef9-9577-47c3-b6a4-1b50e8f1f3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ae795b-sgwx6" Oct 30 09:57:56.868998 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:56.868926 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ae795b-sgwx6" Oct 30 09:57:57.165174 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:57:57.165140 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86823ef9_9577_47c3_b6a4_1b50e8f1f3bf.slice/crio-c2d8c2ce4dfc966c625081edac8ee41e7282f60453f066ebe875de0c44aa13b8 WatchSource:0}: Error finding container c2d8c2ce4dfc966c625081edac8ee41e7282f60453f066ebe875de0c44aa13b8: Status 404 returned error can't find the container with id c2d8c2ce4dfc966c625081edac8ee41e7282f60453f066ebe875de0c44aa13b8 Oct 30 09:57:57.166132 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:57.166108 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ae795b-sgwx6"] Oct 30 09:57:57.235884 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:57.235855 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ae795b-sgwx6" event={"ID":"86823ef9-9577-47c3-b6a4-1b50e8f1f3bf","Type":"ContainerStarted","Data":"c2d8c2ce4dfc966c625081edac8ee41e7282f60453f066ebe875de0c44aa13b8"} Oct 30 09:57:57.777670 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:57.777578 4562 patch_prober.go:29] interesting pod/e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 09:57:57.778218 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:57.778183 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz" podUID="04c56890-74da-4365-b273-13751658dd1a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 09:57:58.244267 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:58.244229 4562 generic.go:356] "Generic (PLEG): container finished" podID="86823ef9-9577-47c3-b6a4-1b50e8f1f3bf" containerID="288550846e5b6e44d6a7b825404ac71b18652bd3cd41fbef9786f725ed32bfe1" exitCode=0 Oct 30 09:57:58.244750 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:58.244308 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ae795b-sgwx6" event={"ID":"86823ef9-9577-47c3-b6a4-1b50e8f1f3bf","Type":"ContainerDied","Data":"288550846e5b6e44d6a7b825404ac71b18652bd3cd41fbef9786f725ed32bfe1"} Oct 30 09:57:58.325263 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:58.325227 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb"] Oct 30 09:57:58.325755 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:58.325718 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb" podUID="cbe6d301-322a-4e4b-880b-36d5704928e4" containerName="istio-proxy" containerID="cri-o://4e8810ec5af8d99242d5b02d5aca72833b7a7ec6e75474e2a4bae2cefb2b3f35" gracePeriod=30 Oct 30 09:57:58.601502 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:57:58.601459 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:57:59.395044 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:59.395018 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ae795b-sgwx6" Oct 30 09:57:59.509770 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:59.509681 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86823ef9-9577-47c3-b6a4-1b50e8f1f3bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86823ef9-9577-47c3-b6a4-1b50e8f1f3bf-maas-api-xks-ca\") pod \"86823ef9-9577-47c3-b6a4-1b50e8f1f3bf\" (UID: \"86823ef9-9577-47c3-b6a4-1b50e8f1f3bf\") " Oct 30 09:57:59.509770 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:59.509744 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86823ef9-9577-47c3-b6a4-1b50e8f1f3bf-kube-api-access-w4p4z\" (UniqueName: \"kubernetes.io/projected/86823ef9-9577-47c3-b6a4-1b50e8f1f3bf-kube-api-access-w4p4z\") pod \"86823ef9-9577-47c3-b6a4-1b50e8f1f3bf\" (UID: \"86823ef9-9577-47c3-b6a4-1b50e8f1f3bf\") " Oct 30 09:57:59.510041 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:59.509780 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86823ef9-9577-47c3-b6a4-1b50e8f1f3bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86823ef9-9577-47c3-b6a4-1b50e8f1f3bf-maas-api-openshift-ca\") pod \"86823ef9-9577-47c3-b6a4-1b50e8f1f3bf\" (UID: \"86823ef9-9577-47c3-b6a4-1b50e8f1f3bf\") " Oct 30 09:57:59.510250 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:59.510224 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86823ef9-9577-47c3-b6a4-1b50e8f1f3bf-maas-api-openshift-ca" pod "86823ef9-9577-47c3-b6a4-1b50e8f1f3bf" (UID: "86823ef9-9577-47c3-b6a4-1b50e8f1f3bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:57:59.511799 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:59.511775 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86823ef9-9577-47c3-b6a4-1b50e8f1f3bf-maas-api-xks-ca" pod "86823ef9-9577-47c3-b6a4-1b50e8f1f3bf" (UID: "86823ef9-9577-47c3-b6a4-1b50e8f1f3bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:57:59.511913 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:59.511887 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86823ef9-9577-47c3-b6a4-1b50e8f1f3bf-kube-api-access-w4p4z" pod "86823ef9-9577-47c3-b6a4-1b50e8f1f3bf" (UID: "86823ef9-9577-47c3-b6a4-1b50e8f1f3bf"). InnerVolumeSpecName "kube-api-access-w4p4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:57:59.611457 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:59.611419 4562 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86823ef9-9577-47c3-b6a4-1b50e8f1f3bf-maas-api-xks-ca\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:57:59.611457 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:59.611452 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4p4z\" (UniqueName: \"kubernetes.io/projected/86823ef9-9577-47c3-b6a4-1b50e8f1f3bf-kube-api-access-w4p4z\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:57:59.611457 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:57:59.611462 4562 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86823ef9-9577-47c3-b6a4-1b50e8f1f3bf-maas-api-openshift-ca\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:58:00.278562 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:00.278520 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ae795b-sgwx6" event={"ID":"86823ef9-9577-47c3-b6a4-1b50e8f1f3bf","Type":"ContainerDied","Data":"c2d8c2ce4dfc966c625081edac8ee41e7282f60453f066ebe875de0c44aa13b8"} Oct 30 09:58:00.278846 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:00.278824 4562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2d8c2ce4dfc966c625081edac8ee41e7282f60453f066ebe875de0c44aa13b8" Oct 30 09:58:00.279042 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:00.279028 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ae795b-sgwx6" Oct 30 09:58:01.234461 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:01.234433 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz" Oct 30 09:58:01.285597 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:01.285560 4562 generic.go:356] "Generic (PLEG): container finished" podID="04c56890-74da-4365-b273-13751658dd1a" containerID="b68d33df98d10d578a9b113d5168530e948f5655067a56e110ba6421856728a4" exitCode=0 Oct 30 09:58:01.285797 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:01.285614 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz" Oct 30 09:58:01.285797 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:01.285677 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz" event={"ID":"04c56890-74da-4365-b273-13751658dd1a","Type":"ContainerDied","Data":"b68d33df98d10d578a9b113d5168530e948f5655067a56e110ba6421856728a4"} Oct 30 09:58:01.285797 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:01.285753 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz" event={"ID":"04c56890-74da-4365-b273-13751658dd1a","Type":"ContainerDied","Data":"8c8b98717325156e89011259198090b25ca115f3092b3ca4d63d23659ea0ec43"} Oct 30 09:58:01.285797 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:01.285796 4562 scope.go:122] "RemoveContainer" containerID="b68d33df98d10d578a9b113d5168530e948f5655067a56e110ba6421856728a4" Oct 30 09:58:01.298162 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:01.298137 4562 scope.go:122] "RemoveContainer" containerID="b68d33df98d10d578a9b113d5168530e948f5655067a56e110ba6421856728a4" Oct 30 09:58:01.298504 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:58:01.298471 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b68d33df98d10d578a9b113d5168530e948f5655067a56e110ba6421856728a4\": container with ID starting with b68d33df98d10d578a9b113d5168530e948f5655067a56e110ba6421856728a4 not found: ID does not exist" containerID="b68d33df98d10d578a9b113d5168530e948f5655067a56e110ba6421856728a4" Oct 30 09:58:01.298619 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:01.298512 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b68d33df98d10d578a9b113d5168530e948f5655067a56e110ba6421856728a4"} err="failed to get container status \"b68d33df98d10d578a9b113d5168530e948f5655067a56e110ba6421856728a4\": rpc error: code = NotFound desc = could not find container \"b68d33df98d10d578a9b113d5168530e948f5655067a56e110ba6421856728a4\": container with ID starting with b68d33df98d10d578a9b113d5168530e948f5655067a56e110ba6421856728a4 not found: ID does not exist" Oct 30 09:58:01.345948 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:01.345916 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04c56890-74da-4365-b273-13751658dd1a-istio-token\" (UniqueName: \"kubernetes.io/projected/04c56890-74da-4365-b273-13751658dd1a-istio-token\") pod \"04c56890-74da-4365-b273-13751658dd1a\" (UID: \"04c56890-74da-4365-b273-13751658dd1a\") " Oct 30 09:58:01.346171 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:01.345984 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04c56890-74da-4365-b273-13751658dd1a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04c56890-74da-4365-b273-13751658dd1a-istio-data\") pod \"04c56890-74da-4365-b273-13751658dd1a\" (UID: \"04c56890-74da-4365-b273-13751658dd1a\") " Oct 30 09:58:01.346171 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:01.346055 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04c56890-74da-4365-b273-13751658dd1a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04c56890-74da-4365-b273-13751658dd1a-credential-socket\") pod \"04c56890-74da-4365-b273-13751658dd1a\" (UID: \"04c56890-74da-4365-b273-13751658dd1a\") " Oct 30 09:58:01.346171 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:01.346122 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04c56890-74da-4365-b273-13751658dd1a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04c56890-74da-4365-b273-13751658dd1a-workload-socket\") pod \"04c56890-74da-4365-b273-13751658dd1a\" (UID: \"04c56890-74da-4365-b273-13751658dd1a\") " Oct 30 09:58:01.346366 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:01.346174 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04c56890-74da-4365-b273-13751658dd1a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04c56890-74da-4365-b273-13751658dd1a-workload-certs\") pod \"04c56890-74da-4365-b273-13751658dd1a\" (UID: \"04c56890-74da-4365-b273-13751658dd1a\") " Oct 30 09:58:01.346366 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:01.346257 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04c56890-74da-4365-b273-13751658dd1a-kube-api-access-vjmc8\" (UniqueName: \"kubernetes.io/projected/04c56890-74da-4365-b273-13751658dd1a-kube-api-access-vjmc8\") pod \"04c56890-74da-4365-b273-13751658dd1a\" (UID: \"04c56890-74da-4365-b273-13751658dd1a\") " Oct 30 09:58:01.346546 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:01.346360 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04c56890-74da-4365-b273-13751658dd1a-istio-data" pod "04c56890-74da-4365-b273-13751658dd1a" (UID: "04c56890-74da-4365-b273-13751658dd1a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:58:01.346546 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:01.346376 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04c56890-74da-4365-b273-13751658dd1a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04c56890-74da-4365-b273-13751658dd1a-istiod-ca-cert\") pod \"04c56890-74da-4365-b273-13751658dd1a\" (UID: \"04c56890-74da-4365-b273-13751658dd1a\") " Oct 30 09:58:01.346546 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:01.346482 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04c56890-74da-4365-b273-13751658dd1a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04c56890-74da-4365-b273-13751658dd1a-istio-envoy\") pod \"04c56890-74da-4365-b273-13751658dd1a\" (UID: \"04c56890-74da-4365-b273-13751658dd1a\") " Oct 30 09:58:01.346717 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:01.346595 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04c56890-74da-4365-b273-13751658dd1a-credential-socket" pod "04c56890-74da-4365-b273-13751658dd1a" (UID: "04c56890-74da-4365-b273-13751658dd1a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:58:01.346717 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:01.346678 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/04c56890-74da-4365-b273-13751658dd1a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04c56890-74da-4365-b273-13751658dd1a-istio-podinfo\") pod \"04c56890-74da-4365-b273-13751658dd1a\" (UID: \"04c56890-74da-4365-b273-13751658dd1a\") " Oct 30 09:58:01.346852 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:01.346721 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04c56890-74da-4365-b273-13751658dd1a-istiod-ca-cert" pod "04c56890-74da-4365-b273-13751658dd1a" (UID: "04c56890-74da-4365-b273-13751658dd1a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:58:01.347440 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:01.347168 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04c56890-74da-4365-b273-13751658dd1a-workload-socket" pod "04c56890-74da-4365-b273-13751658dd1a" (UID: "04c56890-74da-4365-b273-13751658dd1a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:58:01.347440 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:01.347279 4562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04c56890-74da-4365-b273-13751658dd1a-istio-data\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:58:01.347440 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:01.347298 4562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04c56890-74da-4365-b273-13751658dd1a-credential-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:58:01.347440 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:01.347313 4562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04c56890-74da-4365-b273-13751658dd1a-workload-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:58:01.347440 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:01.347326 4562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04c56890-74da-4365-b273-13751658dd1a-istiod-ca-cert\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:58:01.347440 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:01.347369 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04c56890-74da-4365-b273-13751658dd1a-workload-certs" pod "04c56890-74da-4365-b273-13751658dd1a" (UID: "04c56890-74da-4365-b273-13751658dd1a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:58:01.348852 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:01.348824 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04c56890-74da-4365-b273-13751658dd1a-istio-token" pod "04c56890-74da-4365-b273-13751658dd1a" (UID: "04c56890-74da-4365-b273-13751658dd1a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:58:01.348971 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:01.348875 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04c56890-74da-4365-b273-13751658dd1a-istio-envoy" pod "04c56890-74da-4365-b273-13751658dd1a" (UID: "04c56890-74da-4365-b273-13751658dd1a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:58:01.349246 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:01.349216 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04c56890-74da-4365-b273-13751658dd1a-istio-podinfo" pod "04c56890-74da-4365-b273-13751658dd1a" (UID: "04c56890-74da-4365-b273-13751658dd1a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:58:01.349345 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:01.349322 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04c56890-74da-4365-b273-13751658dd1a-kube-api-access-vjmc8" pod "04c56890-74da-4365-b273-13751658dd1a" (UID: "04c56890-74da-4365-b273-13751658dd1a"). InnerVolumeSpecName "kube-api-access-vjmc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:58:01.448341 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:01.448310 4562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04c56890-74da-4365-b273-13751658dd1a-istio-podinfo\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:58:01.448341 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:01.448338 4562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04c56890-74da-4365-b273-13751658dd1a-istio-token\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:58:01.448341 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:01.448348 4562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04c56890-74da-4365-b273-13751658dd1a-workload-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:58:01.448648 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:01.448357 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjmc8\" (UniqueName: \"kubernetes.io/projected/04c56890-74da-4365-b273-13751658dd1a-kube-api-access-vjmc8\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:58:01.448648 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:01.448366 4562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04c56890-74da-4365-b273-13751658dd1a-istio-envoy\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:58:01.635353 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:01.635318 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz"] Oct 30 09:58:01.647520 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:01.647489 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-bdd208-openshift-default-8f948c485-wx6pz"] Oct 30 09:58:01.879581 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:01.879495 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04c56890-74da-4365-b273-13751658dd1a" path="/var/lib/kubelet/pods/04c56890-74da-4365-b273-13751658dd1a/volumes" Oct 30 09:58:02.922030 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:58:02.922009 4562 logging.go:55] [core] [Channel #16108 SubChannel #16109]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.244:50052", ServerName: "10.133.0.244:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.244:50052: connect: connection refused" Oct 30 09:58:03.078259 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:03.078227 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bdd208-6dff5fbd47-69qxt" Oct 30 09:58:03.269325 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:03.269290 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4f0d99e-ad40-4806-8cda-148b2ba50945-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4f0d99e-ad40-4806-8cda-148b2ba50945-plugins-config-volume\") pod \"e4f0d99e-ad40-4806-8cda-148b2ba50945\" (UID: \"e4f0d99e-ad40-4806-8cda-148b2ba50945\") " Oct 30 09:58:03.269497 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:03.269420 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4f0d99e-ad40-4806-8cda-148b2ba50945-kube-api-access-rnh6p\" (UniqueName: \"kubernetes.io/projected/e4f0d99e-ad40-4806-8cda-148b2ba50945-kube-api-access-rnh6p\") pod \"e4f0d99e-ad40-4806-8cda-148b2ba50945\" (UID: \"e4f0d99e-ad40-4806-8cda-148b2ba50945\") " Oct 30 09:58:03.269700 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:03.269674 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4f0d99e-ad40-4806-8cda-148b2ba50945-plugins-config-volume" pod "e4f0d99e-ad40-4806-8cda-148b2ba50945" (UID: "e4f0d99e-ad40-4806-8cda-148b2ba50945"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:58:03.269814 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:03.269793 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4f0d99e-ad40-4806-8cda-148b2ba50945-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:58:03.271531 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:03.271512 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4f0d99e-ad40-4806-8cda-148b2ba50945-kube-api-access-rnh6p" pod "e4f0d99e-ad40-4806-8cda-148b2ba50945" (UID: "e4f0d99e-ad40-4806-8cda-148b2ba50945"). InnerVolumeSpecName "kube-api-access-rnh6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:58:03.302237 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:03.302213 4562 generic.go:356] "Generic (PLEG): container finished" podID="e4f0d99e-ad40-4806-8cda-148b2ba50945" containerID="746442c8724d9cf10a1839423c6052d8110106277aee6f024a59df1e2508ca3f" exitCode=0 Oct 30 09:58:03.302346 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:03.302261 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bdd208-6dff5fbd47-69qxt" Oct 30 09:58:03.302346 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:03.302295 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bdd208-6dff5fbd47-69qxt" event={"ID":"e4f0d99e-ad40-4806-8cda-148b2ba50945","Type":"ContainerDied","Data":"746442c8724d9cf10a1839423c6052d8110106277aee6f024a59df1e2508ca3f"} Oct 30 09:58:03.302460 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:03.302350 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bdd208-6dff5fbd47-69qxt" event={"ID":"e4f0d99e-ad40-4806-8cda-148b2ba50945","Type":"ContainerDied","Data":"441c683a6fa8efa2c7fb637a5477a41f125743c850f671796fbdefabc5a4e350"} Oct 30 09:58:03.302460 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:03.302411 4562 scope.go:122] "RemoveContainer" containerID="746442c8724d9cf10a1839423c6052d8110106277aee6f024a59df1e2508ca3f" Oct 30 09:58:03.312686 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:03.312665 4562 scope.go:122] "RemoveContainer" containerID="746442c8724d9cf10a1839423c6052d8110106277aee6f024a59df1e2508ca3f" Oct 30 09:58:03.312930 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:58:03.312911 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"746442c8724d9cf10a1839423c6052d8110106277aee6f024a59df1e2508ca3f\": container with ID starting with 746442c8724d9cf10a1839423c6052d8110106277aee6f024a59df1e2508ca3f not found: ID does not exist" containerID="746442c8724d9cf10a1839423c6052d8110106277aee6f024a59df1e2508ca3f" Oct 30 09:58:03.312977 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:03.312937 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"746442c8724d9cf10a1839423c6052d8110106277aee6f024a59df1e2508ca3f"} err="failed to get container status \"746442c8724d9cf10a1839423c6052d8110106277aee6f024a59df1e2508ca3f\": rpc error: code = NotFound desc = could not find container \"746442c8724d9cf10a1839423c6052d8110106277aee6f024a59df1e2508ca3f\": container with ID starting with 746442c8724d9cf10a1839423c6052d8110106277aee6f024a59df1e2508ca3f not found: ID does not exist" Oct 30 09:58:03.356518 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:03.356494 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bdd208-6dff5fbd47-69qxt"] Oct 30 09:58:03.359041 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:03.359017 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bdd208-6dff5fbd47-69qxt"] Oct 30 09:58:03.371447 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:03.371424 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnh6p\" (UniqueName: \"kubernetes.io/projected/e4f0d99e-ad40-4806-8cda-148b2ba50945-kube-api-access-rnh6p\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:58:03.569459 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:03.569435 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb" Oct 30 09:58:03.675315 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:03.675272 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbe6d301-322a-4e4b-880b-36d5704928e4-istio-token\" (UniqueName: \"kubernetes.io/projected/cbe6d301-322a-4e4b-880b-36d5704928e4-istio-token\") pod \"cbe6d301-322a-4e4b-880b-36d5704928e4\" (UID: \"cbe6d301-322a-4e4b-880b-36d5704928e4\") " Oct 30 09:58:03.675502 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:03.675338 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbe6d301-322a-4e4b-880b-36d5704928e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe6d301-322a-4e4b-880b-36d5704928e4-credential-socket\") pod \"cbe6d301-322a-4e4b-880b-36d5704928e4\" (UID: \"cbe6d301-322a-4e4b-880b-36d5704928e4\") " Oct 30 09:58:03.675502 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:03.675366 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbe6d301-322a-4e4b-880b-36d5704928e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbe6d301-322a-4e4b-880b-36d5704928e4-istio-data\") pod \"cbe6d301-322a-4e4b-880b-36d5704928e4\" (UID: \"cbe6d301-322a-4e4b-880b-36d5704928e4\") " Oct 30 09:58:03.675502 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:03.675407 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbe6d301-322a-4e4b-880b-36d5704928e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbe6d301-322a-4e4b-880b-36d5704928e4-istio-envoy\") pod \"cbe6d301-322a-4e4b-880b-36d5704928e4\" (UID: \"cbe6d301-322a-4e4b-880b-36d5704928e4\") " Oct 30 09:58:03.675502 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:03.675447 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbe6d301-322a-4e4b-880b-36d5704928e4-kube-api-access-dr9c4\" (UniqueName: \"kubernetes.io/projected/cbe6d301-322a-4e4b-880b-36d5704928e4-kube-api-access-dr9c4\") pod \"cbe6d301-322a-4e4b-880b-36d5704928e4\" (UID: \"cbe6d301-322a-4e4b-880b-36d5704928e4\") " Oct 30 09:58:03.675502 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:03.675474 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cbe6d301-322a-4e4b-880b-36d5704928e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbe6d301-322a-4e4b-880b-36d5704928e4-istio-podinfo\") pod \"cbe6d301-322a-4e4b-880b-36d5704928e4\" (UID: \"cbe6d301-322a-4e4b-880b-36d5704928e4\") " Oct 30 09:58:03.675790 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:03.675537 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbe6d301-322a-4e4b-880b-36d5704928e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbe6d301-322a-4e4b-880b-36d5704928e4-istiod-ca-cert\") pod \"cbe6d301-322a-4e4b-880b-36d5704928e4\" (UID: \"cbe6d301-322a-4e4b-880b-36d5704928e4\") " Oct 30 09:58:03.675790 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:03.675603 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbe6d301-322a-4e4b-880b-36d5704928e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbe6d301-322a-4e4b-880b-36d5704928e4-workload-certs\") pod \"cbe6d301-322a-4e4b-880b-36d5704928e4\" (UID: \"cbe6d301-322a-4e4b-880b-36d5704928e4\") " Oct 30 09:58:03.675790 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:03.675604 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbe6d301-322a-4e4b-880b-36d5704928e4-credential-socket" pod "cbe6d301-322a-4e4b-880b-36d5704928e4" (UID: "cbe6d301-322a-4e4b-880b-36d5704928e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:58:03.675790 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:03.675654 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbe6d301-322a-4e4b-880b-36d5704928e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe6d301-322a-4e4b-880b-36d5704928e4-workload-socket\") pod \"cbe6d301-322a-4e4b-880b-36d5704928e4\" (UID: \"cbe6d301-322a-4e4b-880b-36d5704928e4\") " Oct 30 09:58:03.675994 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:03.675845 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbe6d301-322a-4e4b-880b-36d5704928e4-istio-data" pod "cbe6d301-322a-4e4b-880b-36d5704928e4" (UID: "cbe6d301-322a-4e4b-880b-36d5704928e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:58:03.676642 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:03.676252 4562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe6d301-322a-4e4b-880b-36d5704928e4-credential-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:58:03.676642 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:03.676276 4562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbe6d301-322a-4e4b-880b-36d5704928e4-istio-data\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:58:03.676642 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:03.676305 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbe6d301-322a-4e4b-880b-36d5704928e4-istiod-ca-cert" pod "cbe6d301-322a-4e4b-880b-36d5704928e4" (UID: "cbe6d301-322a-4e4b-880b-36d5704928e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:58:03.676642 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:03.676475 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbe6d301-322a-4e4b-880b-36d5704928e4-workload-socket" pod "cbe6d301-322a-4e4b-880b-36d5704928e4" (UID: "cbe6d301-322a-4e4b-880b-36d5704928e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:58:03.676642 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:03.676605 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbe6d301-322a-4e4b-880b-36d5704928e4-workload-certs" pod "cbe6d301-322a-4e4b-880b-36d5704928e4" (UID: "cbe6d301-322a-4e4b-880b-36d5704928e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:58:03.678158 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:03.678129 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbe6d301-322a-4e4b-880b-36d5704928e4-kube-api-access-dr9c4" pod "cbe6d301-322a-4e4b-880b-36d5704928e4" (UID: "cbe6d301-322a-4e4b-880b-36d5704928e4"). InnerVolumeSpecName "kube-api-access-dr9c4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:58:03.678158 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:03.678129 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cbe6d301-322a-4e4b-880b-36d5704928e4-istio-podinfo" pod "cbe6d301-322a-4e4b-880b-36d5704928e4" (UID: "cbe6d301-322a-4e4b-880b-36d5704928e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:58:03.678414 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:03.678174 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbe6d301-322a-4e4b-880b-36d5704928e4-istio-token" pod "cbe6d301-322a-4e4b-880b-36d5704928e4" (UID: "cbe6d301-322a-4e4b-880b-36d5704928e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:58:03.678668 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:03.678647 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbe6d301-322a-4e4b-880b-36d5704928e4-istio-envoy" pod "cbe6d301-322a-4e4b-880b-36d5704928e4" (UID: "cbe6d301-322a-4e4b-880b-36d5704928e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:58:03.777515 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:03.777482 4562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbe6d301-322a-4e4b-880b-36d5704928e4-istio-token\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:58:03.777515 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:03.777514 4562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbe6d301-322a-4e4b-880b-36d5704928e4-istio-envoy\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:58:03.777515 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:03.777523 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dr9c4\" (UniqueName: \"kubernetes.io/projected/cbe6d301-322a-4e4b-880b-36d5704928e4-kube-api-access-dr9c4\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:58:03.777748 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:03.777532 4562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbe6d301-322a-4e4b-880b-36d5704928e4-istio-podinfo\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:58:03.777748 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:03.777540 4562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbe6d301-322a-4e4b-880b-36d5704928e4-istiod-ca-cert\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:58:03.777748 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:03.777548 4562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbe6d301-322a-4e4b-880b-36d5704928e4-workload-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:58:03.777748 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:03.777556 4562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe6d301-322a-4e4b-880b-36d5704928e4-workload-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:58:03.781671 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:03.781642 4562 scope.go:122] "RemoveContainer" containerID="4e8810ec5af8d99242d5b02d5aca72833b7a7ec6e75474e2a4bae2cefb2b3f35" Oct 30 09:58:03.874207 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:03.874177 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4f0d99e-ad40-4806-8cda-148b2ba50945" path="/var/lib/kubelet/pods/e4f0d99e-ad40-4806-8cda-148b2ba50945/volumes" Oct 30 09:58:03.922147 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:03.922107 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bdd208-6dff5fbd47-69qxt" podUID="e4f0d99e-ad40-4806-8cda-148b2ba50945" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.244:50052\" within 1s: context deadline exceeded" Oct 30 09:58:04.308000 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:04.307965 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb" event={"ID":"cbe6d301-322a-4e4b-880b-36d5704928e4","Type":"ContainerDied","Data":"4e8810ec5af8d99242d5b02d5aca72833b7a7ec6e75474e2a4bae2cefb2b3f35"} Oct 30 09:58:04.308164 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:04.308021 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb" event={"ID":"cbe6d301-322a-4e4b-880b-36d5704928e4","Type":"ContainerDied","Data":"e7b6f538ba33a36c058042e740b31e170626c3ca030c9c89577c40352ed09000"} Oct 30 09:58:04.308164 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:04.308059 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb" Oct 30 09:58:04.389699 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:04.389660 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb"] Oct 30 09:58:04.393795 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:04.393771 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-176b6c-openshift-default-6dc4f8f6bf-fbvmb"] Oct 30 09:58:05.874765 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:05.874728 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbe6d301-322a-4e4b-880b-36d5704928e4" path="/var/lib/kubelet/pods/cbe6d301-322a-4e4b-880b-36d5704928e4/volumes" Oct 30 09:58:10.558714 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:10.558683 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ae795b-sgwx6"] Oct 30 09:58:10.571147 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:10.571113 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ae795b-sgwx6"] Oct 30 09:58:11.875187 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:11.875151 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86823ef9-9577-47c3-b6a4-1b50e8f1f3bf" path="/var/lib/kubelet/pods/86823ef9-9577-47c3-b6a4-1b50e8f1f3bf/volumes" Oct 30 09:58:17.661227 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:17.661182 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0f6f96-qm84k"] Oct 30 09:58:17.661763 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:17.661740 4562 cpu_manager.go:414] "containerMap: removing container" podUID="e4f0d99e-ad40-4806-8cda-148b2ba50945" containerName="payload-pre-processing" Oct 30 09:58:17.661763 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:17.661754 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4f0d99e-ad40-4806-8cda-148b2ba50945" containerName="payload-pre-processing" Oct 30 09:58:17.661876 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:17.661770 4562 cpu_manager.go:414] "containerMap: removing container" podUID="04c56890-74da-4365-b273-13751658dd1a" containerName="istio-proxy" Oct 30 09:58:17.661876 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:17.661775 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04c56890-74da-4365-b273-13751658dd1a" containerName="istio-proxy" Oct 30 09:58:17.661876 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:17.661794 4562 cpu_manager.go:414] "containerMap: removing container" podUID="cbe6d301-322a-4e4b-880b-36d5704928e4" containerName="istio-proxy" Oct 30 09:58:17.661876 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:17.661798 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbe6d301-322a-4e4b-880b-36d5704928e4" containerName="istio-proxy" Oct 30 09:58:17.661876 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:17.661815 4562 cpu_manager.go:414] "containerMap: removing container" podUID="86823ef9-9577-47c3-b6a4-1b50e8f1f3bf" containerName="revoke-keys" Oct 30 09:58:17.661876 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:17.661819 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86823ef9-9577-47c3-b6a4-1b50e8f1f3bf" containerName="revoke-keys" Oct 30 09:58:17.662185 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:17.661902 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbe6d301-322a-4e4b-880b-36d5704928e4" containerName="istio-proxy" Oct 30 09:58:17.662185 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:17.661913 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="86823ef9-9577-47c3-b6a4-1b50e8f1f3bf" containerName="revoke-keys" Oct 30 09:58:17.662185 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:17.661923 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4f0d99e-ad40-4806-8cda-148b2ba50945" containerName="payload-pre-processing" Oct 30 09:58:17.662185 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:17.661929 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="04c56890-74da-4365-b273-13751658dd1a" containerName="istio-proxy" Oct 30 09:58:17.665583 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:17.665561 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0f6f96-qm84k" Oct 30 09:58:17.687088 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:17.687060 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0f6f96-qm84k"] Oct 30 09:58:17.694894 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:17.694872 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:58:17.695137 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:17.695102 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:58:17.754901 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:17.754872 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2rlw\" (UniqueName: \"kubernetes.io/projected/91685b11-aa13-4726-b43e-95af40dce683-kube-api-access-g2rlw\") pod \"maas-api-revoke-keys-e2e-extmod-b-0f6f96-qm84k\" (UID: \"91685b11-aa13-4726-b43e-95af40dce683\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0f6f96-qm84k" Oct 30 09:58:17.755042 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:17.754987 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91685b11-aa13-4726-b43e-95af40dce683-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-0f6f96-qm84k\" (UID: \"91685b11-aa13-4726-b43e-95af40dce683\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0f6f96-qm84k" Oct 30 09:58:17.755087 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:17.755058 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91685b11-aa13-4726-b43e-95af40dce683-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-0f6f96-qm84k\" (UID: \"91685b11-aa13-4726-b43e-95af40dce683\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0f6f96-qm84k" Oct 30 09:58:17.855715 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:17.855678 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91685b11-aa13-4726-b43e-95af40dce683-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-0f6f96-qm84k\" (UID: \"91685b11-aa13-4726-b43e-95af40dce683\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0f6f96-qm84k" Oct 30 09:58:17.855873 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:17.855819 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91685b11-aa13-4726-b43e-95af40dce683-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-0f6f96-qm84k\" (UID: \"91685b11-aa13-4726-b43e-95af40dce683\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0f6f96-qm84k" Oct 30 09:58:17.855919 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:17.855892 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2rlw\" (UniqueName: \"kubernetes.io/projected/91685b11-aa13-4726-b43e-95af40dce683-kube-api-access-g2rlw\") pod \"maas-api-revoke-keys-e2e-extmod-b-0f6f96-qm84k\" (UID: \"91685b11-aa13-4726-b43e-95af40dce683\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0f6f96-qm84k" Oct 30 09:58:17.856462 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:17.856436 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91685b11-aa13-4726-b43e-95af40dce683-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-0f6f96-qm84k\" (UID: \"91685b11-aa13-4726-b43e-95af40dce683\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0f6f96-qm84k" Oct 30 09:58:17.858200 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:17.858181 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91685b11-aa13-4726-b43e-95af40dce683-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-0f6f96-qm84k\" (UID: \"91685b11-aa13-4726-b43e-95af40dce683\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0f6f96-qm84k" Oct 30 09:58:17.876207 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:17.876176 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2rlw\" (UniqueName: \"kubernetes.io/projected/91685b11-aa13-4726-b43e-95af40dce683-kube-api-access-g2rlw\") pod \"maas-api-revoke-keys-e2e-extmod-b-0f6f96-qm84k\" (UID: \"91685b11-aa13-4726-b43e-95af40dce683\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0f6f96-qm84k" Oct 30 09:58:17.992237 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:17.992209 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0f6f96-qm84k" Oct 30 09:58:18.290961 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:58:18.290916 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91685b11_aa13_4726_b43e_95af40dce683.slice/crio-622bdce3cc70a9e794b96d93ed94ccdcb11f80b58817edd8f83ffccb81b49d3d WatchSource:0}: Error finding container 622bdce3cc70a9e794b96d93ed94ccdcb11f80b58817edd8f83ffccb81b49d3d: Status 404 returned error can't find the container with id 622bdce3cc70a9e794b96d93ed94ccdcb11f80b58817edd8f83ffccb81b49d3d Oct 30 09:58:18.291900 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:18.291874 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0f6f96-qm84k"] Oct 30 09:58:18.404947 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:18.404909 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0f6f96-qm84k" event={"ID":"91685b11-aa13-4726-b43e-95af40dce683","Type":"ContainerStarted","Data":"622bdce3cc70a9e794b96d93ed94ccdcb11f80b58817edd8f83ffccb81b49d3d"} Oct 30 09:58:19.412234 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:19.412198 4562 generic.go:356] "Generic (PLEG): container finished" podID="91685b11-aa13-4726-b43e-95af40dce683" containerID="6e7acfb4989fee15f8a41ebfccf16defd3761ce258f55d0680ede693946a0da6" exitCode=0 Oct 30 09:58:19.412681 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:19.412280 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0f6f96-qm84k" event={"ID":"91685b11-aa13-4726-b43e-95af40dce683","Type":"ContainerDied","Data":"6e7acfb4989fee15f8a41ebfccf16defd3761ce258f55d0680ede693946a0da6"} Oct 30 09:58:20.560887 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:20.560860 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0f6f96-qm84k" Oct 30 09:58:20.686806 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:20.686714 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91685b11-aa13-4726-b43e-95af40dce683-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91685b11-aa13-4726-b43e-95af40dce683-maas-api-openshift-ca\") pod \"91685b11-aa13-4726-b43e-95af40dce683\" (UID: \"91685b11-aa13-4726-b43e-95af40dce683\") " Oct 30 09:58:20.686806 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:20.686752 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91685b11-aa13-4726-b43e-95af40dce683-kube-api-access-g2rlw\" (UniqueName: \"kubernetes.io/projected/91685b11-aa13-4726-b43e-95af40dce683-kube-api-access-g2rlw\") pod \"91685b11-aa13-4726-b43e-95af40dce683\" (UID: \"91685b11-aa13-4726-b43e-95af40dce683\") " Oct 30 09:58:20.686806 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:20.686772 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91685b11-aa13-4726-b43e-95af40dce683-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91685b11-aa13-4726-b43e-95af40dce683-maas-api-xks-ca\") pod \"91685b11-aa13-4726-b43e-95af40dce683\" (UID: \"91685b11-aa13-4726-b43e-95af40dce683\") " Oct 30 09:58:20.687211 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:20.687188 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91685b11-aa13-4726-b43e-95af40dce683-maas-api-openshift-ca" pod "91685b11-aa13-4726-b43e-95af40dce683" (UID: "91685b11-aa13-4726-b43e-95af40dce683"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:58:20.688891 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:20.688867 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91685b11-aa13-4726-b43e-95af40dce683-kube-api-access-g2rlw" pod "91685b11-aa13-4726-b43e-95af40dce683" (UID: "91685b11-aa13-4726-b43e-95af40dce683"). InnerVolumeSpecName "kube-api-access-g2rlw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:58:20.689072 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:20.689047 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91685b11-aa13-4726-b43e-95af40dce683-maas-api-xks-ca" pod "91685b11-aa13-4726-b43e-95af40dce683" (UID: "91685b11-aa13-4726-b43e-95af40dce683"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:58:20.788501 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:20.788459 4562 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91685b11-aa13-4726-b43e-95af40dce683-maas-api-openshift-ca\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:58:20.788501 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:20.788492 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2rlw\" (UniqueName: \"kubernetes.io/projected/91685b11-aa13-4726-b43e-95af40dce683-kube-api-access-g2rlw\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:58:20.788501 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:20.788508 4562 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91685b11-aa13-4726-b43e-95af40dce683-maas-api-xks-ca\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:58:21.424765 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:21.424734 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0f6f96-qm84k" Oct 30 09:58:21.424942 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:21.424762 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0f6f96-qm84k" event={"ID":"91685b11-aa13-4726-b43e-95af40dce683","Type":"ContainerDied","Data":"622bdce3cc70a9e794b96d93ed94ccdcb11f80b58817edd8f83ffccb81b49d3d"} Oct 30 09:58:21.424942 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:21.424804 4562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="622bdce3cc70a9e794b96d93ed94ccdcb11f80b58817edd8f83ffccb81b49d3d" Oct 30 09:58:25.751704 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:25.751663 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-555bd640-9nfpl"] Oct 30 09:58:25.752235 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:25.752219 4562 cpu_manager.go:414] "containerMap: removing container" podUID="91685b11-aa13-4726-b43e-95af40dce683" containerName="revoke-keys" Oct 30 09:58:25.752284 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:25.752235 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91685b11-aa13-4726-b43e-95af40dce683" containerName="revoke-keys" Oct 30 09:58:25.752339 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:25.752329 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="91685b11-aa13-4726-b43e-95af40dce683" containerName="revoke-keys" Oct 30 09:58:25.755777 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:25.755758 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-555bd640-9nfpl" Oct 30 09:58:25.773581 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:25.773556 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-555bd640-9nfpl"] Oct 30 09:58:25.782781 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:25.782754 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:58:25.782910 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:25.782770 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:58:25.850814 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:25.850773 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/007dfc94-fbc7-456d-8760-2235a48f4277-maas-api-xks-ca\") pod \"maas-api-revoke-sub-555bd640-9nfpl\" (UID: \"007dfc94-fbc7-456d-8760-2235a48f4277\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-555bd640-9nfpl" Oct 30 09:58:25.850988 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:25.850830 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/007dfc94-fbc7-456d-8760-2235a48f4277-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-555bd640-9nfpl\" (UID: \"007dfc94-fbc7-456d-8760-2235a48f4277\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-555bd640-9nfpl" Oct 30 09:58:25.850988 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:25.850963 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm2jt\" (UniqueName: \"kubernetes.io/projected/007dfc94-fbc7-456d-8760-2235a48f4277-kube-api-access-vm2jt\") pod \"maas-api-revoke-sub-555bd640-9nfpl\" (UID: \"007dfc94-fbc7-456d-8760-2235a48f4277\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-555bd640-9nfpl" Oct 30 09:58:25.952725 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:25.952693 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/007dfc94-fbc7-456d-8760-2235a48f4277-maas-api-xks-ca\") pod \"maas-api-revoke-sub-555bd640-9nfpl\" (UID: \"007dfc94-fbc7-456d-8760-2235a48f4277\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-555bd640-9nfpl" Oct 30 09:58:25.952908 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:25.952743 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/007dfc94-fbc7-456d-8760-2235a48f4277-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-555bd640-9nfpl\" (UID: \"007dfc94-fbc7-456d-8760-2235a48f4277\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-555bd640-9nfpl" Oct 30 09:58:25.952908 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:25.952816 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm2jt\" (UniqueName: \"kubernetes.io/projected/007dfc94-fbc7-456d-8760-2235a48f4277-kube-api-access-vm2jt\") pod \"maas-api-revoke-sub-555bd640-9nfpl\" (UID: \"007dfc94-fbc7-456d-8760-2235a48f4277\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-555bd640-9nfpl" Oct 30 09:58:25.953473 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:25.953447 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/007dfc94-fbc7-456d-8760-2235a48f4277-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-555bd640-9nfpl\" (UID: \"007dfc94-fbc7-456d-8760-2235a48f4277\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-555bd640-9nfpl" Oct 30 09:58:25.955068 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:25.955045 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/007dfc94-fbc7-456d-8760-2235a48f4277-maas-api-xks-ca\") pod \"maas-api-revoke-sub-555bd640-9nfpl\" (UID: \"007dfc94-fbc7-456d-8760-2235a48f4277\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-555bd640-9nfpl" Oct 30 09:58:25.972797 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:25.972775 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm2jt\" (UniqueName: \"kubernetes.io/projected/007dfc94-fbc7-456d-8760-2235a48f4277-kube-api-access-vm2jt\") pod \"maas-api-revoke-sub-555bd640-9nfpl\" (UID: \"007dfc94-fbc7-456d-8760-2235a48f4277\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-555bd640-9nfpl" Oct 30 09:58:26.068945 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:26.068872 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-555bd640-9nfpl" Oct 30 09:58:26.367547 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:58:26.367507 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod007dfc94_fbc7_456d_8760_2235a48f4277.slice/crio-ae8deae5fd7e682f3d67f27c7110a8fb1c5dfa87c67acf2dfb249fb52aaa5f69 WatchSource:0}: Error finding container ae8deae5fd7e682f3d67f27c7110a8fb1c5dfa87c67acf2dfb249fb52aaa5f69: Status 404 returned error can't find the container with id ae8deae5fd7e682f3d67f27c7110a8fb1c5dfa87c67acf2dfb249fb52aaa5f69 Oct 30 09:58:26.367729 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:26.367706 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-555bd640-9nfpl"] Oct 30 09:58:26.457375 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:26.457340 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-555bd640-9nfpl" event={"ID":"007dfc94-fbc7-456d-8760-2235a48f4277","Type":"ContainerStarted","Data":"ae8deae5fd7e682f3d67f27c7110a8fb1c5dfa87c67acf2dfb249fb52aaa5f69"} Oct 30 09:58:27.465824 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:27.465789 4562 generic.go:356] "Generic (PLEG): container finished" podID="007dfc94-fbc7-456d-8760-2235a48f4277" containerID="968d4db8a39b5bcce0d14bccbcf6b960befa721d0c41be41463f62d259f0ce2a" exitCode=0 Oct 30 09:58:27.466306 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:27.465864 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-555bd640-9nfpl" event={"ID":"007dfc94-fbc7-456d-8760-2235a48f4277","Type":"ContainerDied","Data":"968d4db8a39b5bcce0d14bccbcf6b960befa721d0c41be41463f62d259f0ce2a"} Oct 30 09:58:28.160177 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:28.160146 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0f6f96-5d59889f9f-c8xpd"] Oct 30 09:58:28.603814 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:58:28.603777 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:58:28.635260 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:28.635239 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-555bd640-9nfpl" Oct 30 09:58:28.790454 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:28.785923 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/007dfc94-fbc7-456d-8760-2235a48f4277-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/007dfc94-fbc7-456d-8760-2235a48f4277-maas-api-xks-ca\") pod \"007dfc94-fbc7-456d-8760-2235a48f4277\" (UID: \"007dfc94-fbc7-456d-8760-2235a48f4277\") " Oct 30 09:58:28.790454 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:28.786093 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/007dfc94-fbc7-456d-8760-2235a48f4277-kube-api-access-vm2jt\" (UniqueName: \"kubernetes.io/projected/007dfc94-fbc7-456d-8760-2235a48f4277-kube-api-access-vm2jt\") pod \"007dfc94-fbc7-456d-8760-2235a48f4277\" (UID: \"007dfc94-fbc7-456d-8760-2235a48f4277\") " Oct 30 09:58:28.790454 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:28.786133 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/007dfc94-fbc7-456d-8760-2235a48f4277-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/007dfc94-fbc7-456d-8760-2235a48f4277-maas-api-openshift-ca\") pod \"007dfc94-fbc7-456d-8760-2235a48f4277\" (UID: \"007dfc94-fbc7-456d-8760-2235a48f4277\") " Oct 30 09:58:28.793051 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:28.792770 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/007dfc94-fbc7-456d-8760-2235a48f4277-maas-api-openshift-ca" pod "007dfc94-fbc7-456d-8760-2235a48f4277" (UID: "007dfc94-fbc7-456d-8760-2235a48f4277"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:58:28.809261 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:28.809199 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/007dfc94-fbc7-456d-8760-2235a48f4277-maas-api-xks-ca" pod "007dfc94-fbc7-456d-8760-2235a48f4277" (UID: "007dfc94-fbc7-456d-8760-2235a48f4277"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:58:28.835604 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:28.835562 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/007dfc94-fbc7-456d-8760-2235a48f4277-kube-api-access-vm2jt" pod "007dfc94-fbc7-456d-8760-2235a48f4277" (UID: "007dfc94-fbc7-456d-8760-2235a48f4277"). InnerVolumeSpecName "kube-api-access-vm2jt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:58:28.887427 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:28.887396 4562 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/007dfc94-fbc7-456d-8760-2235a48f4277-maas-api-xks-ca\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:58:28.887427 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:28.887424 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vm2jt\" (UniqueName: \"kubernetes.io/projected/007dfc94-fbc7-456d-8760-2235a48f4277-kube-api-access-vm2jt\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:58:28.887633 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:28.887435 4562 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/007dfc94-fbc7-456d-8760-2235a48f4277-maas-api-openshift-ca\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:58:29.478909 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:29.478869 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-555bd640-9nfpl" event={"ID":"007dfc94-fbc7-456d-8760-2235a48f4277","Type":"ContainerDied","Data":"ae8deae5fd7e682f3d67f27c7110a8fb1c5dfa87c67acf2dfb249fb52aaa5f69"} Oct 30 09:58:29.478909 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:29.478910 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-555bd640-9nfpl" Oct 30 09:58:29.479164 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:29.478934 4562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae8deae5fd7e682f3d67f27c7110a8fb1c5dfa87c67acf2dfb249fb52aaa5f69" Oct 30 09:58:30.380764 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:30.380730 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf"] Oct 30 09:58:30.381210 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:30.380923 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf" podUID="cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4" containerName="istio-proxy" containerID="cri-o://5a1aca49e4c943a5bbfff5b7d725110f9c3f989317e27bcfd21290f5f3ef7fc5" gracePeriod=30 Oct 30 09:58:32.598306 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:32.598269 4562 patch_prober.go:29] interesting pod/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 09:58:32.598827 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:32.598344 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf" podUID="cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 09:58:32.725475 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:32.725432 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0f6f96-qm84k"] Oct 30 09:58:32.731549 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:32.731527 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0f6f96-qm84k"] Oct 30 09:58:33.160972 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:33.160930 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0f6f96-5d59889f9f-c8xpd" podUID="35bb8805-b6e0-4225-9345-094c08d33935" containerName="payload-pre-processing" containerID="cri-o://bc70985b974bf7ffc1367ee73adf15dcbaec421ca44f004b3b93da529c5f29be" gracePeriod=25 Oct 30 09:58:33.874965 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:33.874932 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91685b11-aa13-4726-b43e-95af40dce683" path="/var/lib/kubelet/pods/91685b11-aa13-4726-b43e-95af40dce683/volumes" Oct 30 09:58:35.523102 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:35.523068 4562 generic.go:356] "Generic (PLEG): container finished" podID="cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4" containerID="5a1aca49e4c943a5bbfff5b7d725110f9c3f989317e27bcfd21290f5f3ef7fc5" exitCode=0 Oct 30 09:58:35.523519 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:35.523147 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf" event={"ID":"cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4","Type":"ContainerDied","Data":"5a1aca49e4c943a5bbfff5b7d725110f9c3f989317e27bcfd21290f5f3ef7fc5"} Oct 30 09:58:35.648526 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:35.648465 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf" Oct 30 09:58:35.677134 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:35.677107 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4-istio-token\" (UniqueName: \"kubernetes.io/projected/cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4-istio-token\") pod \"cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4\" (UID: \"cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4\") " Oct 30 09:58:35.677258 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:35.677166 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4-istiod-ca-cert\") pod \"cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4\" (UID: \"cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4\") " Oct 30 09:58:35.677258 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:35.677193 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4-credential-socket\") pod \"cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4\" (UID: \"cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4\") " Oct 30 09:58:35.677366 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:35.677311 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4-istio-data\") pod \"cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4\" (UID: \"cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4\") " Oct 30 09:58:35.677460 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:35.677441 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4-workload-socket\") pod \"cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4\" (UID: \"cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4\") " Oct 30 09:58:35.677578 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:35.677559 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4-kube-api-access-tr6s5\" (UniqueName: \"kubernetes.io/projected/cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4-kube-api-access-tr6s5\") pod \"cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4\" (UID: \"cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4\") " Oct 30 09:58:35.677668 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:35.677578 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4-credential-socket" pod "cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4" (UID: "cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:58:35.677668 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:35.677622 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4-workload-certs\") pod \"cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4\" (UID: \"cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4\") " Oct 30 09:58:35.677668 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:35.677653 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4-istio-podinfo\") pod \"cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4\" (UID: \"cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4\") " Oct 30 09:58:35.677827 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:35.677685 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4-istio-envoy\") pod \"cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4\" (UID: \"cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4\") " Oct 30 09:58:35.677827 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:35.677663 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4-istio-data" pod "cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4" (UID: "cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:58:35.677827 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:35.677692 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4-istiod-ca-cert" pod "cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4" (UID: "cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:58:35.677827 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:35.677734 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4-workload-socket" pod "cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4" (UID: "cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:58:35.677987 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:35.677832 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4-workload-certs" pod "cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4" (UID: "cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:58:35.678195 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:35.678174 4562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4-workload-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:58:35.678276 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:35.678200 4562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4-istiod-ca-cert\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:58:35.678276 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:35.678215 4562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4-credential-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:58:35.678276 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:35.678229 4562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4-istio-data\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:58:35.678276 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:35.678242 4562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4-workload-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:58:35.679855 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:35.679817 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4-istio-envoy" pod "cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4" (UID: "cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:58:35.679963 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:35.679902 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4-istio-podinfo" pod "cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4" (UID: "cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:58:35.680047 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:35.680027 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4-istio-token" pod "cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4" (UID: "cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:58:35.680097 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:35.680061 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4-kube-api-access-tr6s5" pod "cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4" (UID: "cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4"). InnerVolumeSpecName "kube-api-access-tr6s5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:58:35.779700 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:35.779658 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tr6s5\" (UniqueName: \"kubernetes.io/projected/cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4-kube-api-access-tr6s5\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:58:35.779700 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:35.779693 4562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4-istio-podinfo\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:58:35.779901 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:35.779707 4562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4-istio-envoy\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:58:35.779901 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:35.779720 4562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4-istio-token\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:58:36.531355 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:36.531316 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf" event={"ID":"cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4","Type":"ContainerDied","Data":"b1470b2d2cf35b59b3096dacae068966b3e530fb0d59eafe0daedca9c3f19c6a"} Oct 30 09:58:36.531876 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:36.531333 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf" Oct 30 09:58:36.531876 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:36.531412 4562 scope.go:122] "RemoveContainer" containerID="5a1aca49e4c943a5bbfff5b7d725110f9c3f989317e27bcfd21290f5f3ef7fc5" Oct 30 09:58:36.597659 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:36.597631 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf"] Oct 30 09:58:36.610425 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:36.610367 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-ks2zf"] Oct 30 09:58:36.669112 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:36.669078 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-82ccfd/test-model-82ccfd-kserve-77684cd675-n8rmr"] Oct 30 09:58:37.874911 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:37.874863 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4" path="/var/lib/kubelet/pods/cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4/volumes" Oct 30 09:58:38.100000 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:38.099965 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0f6f96-5d59889f9f-c8xpd" podUID="35bb8805-b6e0-4225-9345-094c08d33935" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:58:40.222113 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:40.222078 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk"] Oct 30 09:58:40.222623 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:40.222361 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk" podUID="54cc6629-e1ca-4af7-9455-80e3b4d097f6" containerName="istio-proxy" containerID="cri-o://fa62c69fd900306ab5356014658f747fc8cdcdca034f0dd8bb719977a5af299c" gracePeriod=30 Oct 30 09:58:45.498857 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:45.498789 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk" Oct 30 09:58:45.587487 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:45.587446 4562 generic.go:356] "Generic (PLEG): container finished" podID="54cc6629-e1ca-4af7-9455-80e3b4d097f6" containerID="fa62c69fd900306ab5356014658f747fc8cdcdca034f0dd8bb719977a5af299c" exitCode=0 Oct 30 09:58:45.587686 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:45.587512 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk" Oct 30 09:58:45.587686 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:45.587518 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk" event={"ID":"54cc6629-e1ca-4af7-9455-80e3b4d097f6","Type":"ContainerDied","Data":"fa62c69fd900306ab5356014658f747fc8cdcdca034f0dd8bb719977a5af299c"} Oct 30 09:58:45.587686 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:45.587573 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk" event={"ID":"54cc6629-e1ca-4af7-9455-80e3b4d097f6","Type":"ContainerDied","Data":"928787ac3efa4bfe58572ea2eb2d41a8ce49601a8358a6bad030a3f7f0290339"} Oct 30 09:58:45.587686 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:45.587604 4562 scope.go:122] "RemoveContainer" containerID="fa62c69fd900306ab5356014658f747fc8cdcdca034f0dd8bb719977a5af299c" Oct 30 09:58:45.600255 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:45.600233 4562 scope.go:122] "RemoveContainer" containerID="21fc9cf91ea050cde727c5f8eb885da8d61174f5d4c3cd0b9bf497ac5f65a86a" Oct 30 09:58:45.609451 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:45.609433 4562 scope.go:122] "RemoveContainer" containerID="fa62c69fd900306ab5356014658f747fc8cdcdca034f0dd8bb719977a5af299c" Oct 30 09:58:45.609720 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:58:45.609702 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa62c69fd900306ab5356014658f747fc8cdcdca034f0dd8bb719977a5af299c\": container with ID starting with fa62c69fd900306ab5356014658f747fc8cdcdca034f0dd8bb719977a5af299c not found: ID does not exist" containerID="fa62c69fd900306ab5356014658f747fc8cdcdca034f0dd8bb719977a5af299c" Oct 30 09:58:45.609797 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:45.609726 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa62c69fd900306ab5356014658f747fc8cdcdca034f0dd8bb719977a5af299c"} err="failed to get container status \"fa62c69fd900306ab5356014658f747fc8cdcdca034f0dd8bb719977a5af299c\": rpc error: code = NotFound desc = could not find container \"fa62c69fd900306ab5356014658f747fc8cdcdca034f0dd8bb719977a5af299c\": container with ID starting with fa62c69fd900306ab5356014658f747fc8cdcdca034f0dd8bb719977a5af299c not found: ID does not exist" Oct 30 09:58:45.609797 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:45.609742 4562 scope.go:122] "RemoveContainer" containerID="21fc9cf91ea050cde727c5f8eb885da8d61174f5d4c3cd0b9bf497ac5f65a86a" Oct 30 09:58:45.609980 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:58:45.609963 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21fc9cf91ea050cde727c5f8eb885da8d61174f5d4c3cd0b9bf497ac5f65a86a\": container with ID starting with 21fc9cf91ea050cde727c5f8eb885da8d61174f5d4c3cd0b9bf497ac5f65a86a not found: ID does not exist" containerID="21fc9cf91ea050cde727c5f8eb885da8d61174f5d4c3cd0b9bf497ac5f65a86a" Oct 30 09:58:45.610023 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:45.609985 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21fc9cf91ea050cde727c5f8eb885da8d61174f5d4c3cd0b9bf497ac5f65a86a"} err="failed to get container status \"21fc9cf91ea050cde727c5f8eb885da8d61174f5d4c3cd0b9bf497ac5f65a86a\": rpc error: code = NotFound desc = could not find container \"21fc9cf91ea050cde727c5f8eb885da8d61174f5d4c3cd0b9bf497ac5f65a86a\": container with ID starting with 21fc9cf91ea050cde727c5f8eb885da8d61174f5d4c3cd0b9bf497ac5f65a86a not found: ID does not exist" Oct 30 09:58:45.626254 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:45.626218 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54cc6629-e1ca-4af7-9455-80e3b4d097f6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54cc6629-e1ca-4af7-9455-80e3b4d097f6-istio-data\") pod \"54cc6629-e1ca-4af7-9455-80e3b4d097f6\" (UID: \"54cc6629-e1ca-4af7-9455-80e3b4d097f6\") " Oct 30 09:58:45.627164 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:45.627140 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54cc6629-e1ca-4af7-9455-80e3b4d097f6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54cc6629-e1ca-4af7-9455-80e3b4d097f6-credential-socket\") pod \"54cc6629-e1ca-4af7-9455-80e3b4d097f6\" (UID: \"54cc6629-e1ca-4af7-9455-80e3b4d097f6\") " Oct 30 09:58:45.627666 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:45.627648 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54cc6629-e1ca-4af7-9455-80e3b4d097f6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54cc6629-e1ca-4af7-9455-80e3b4d097f6-workload-certs\") pod \"54cc6629-e1ca-4af7-9455-80e3b4d097f6\" (UID: \"54cc6629-e1ca-4af7-9455-80e3b4d097f6\") " Oct 30 09:58:45.628155 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:45.626930 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54cc6629-e1ca-4af7-9455-80e3b4d097f6-istio-data" pod "54cc6629-e1ca-4af7-9455-80e3b4d097f6" (UID: "54cc6629-e1ca-4af7-9455-80e3b4d097f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:58:45.628241 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:45.627563 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54cc6629-e1ca-4af7-9455-80e3b4d097f6-credential-socket" pod "54cc6629-e1ca-4af7-9455-80e3b4d097f6" (UID: "54cc6629-e1ca-4af7-9455-80e3b4d097f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:58:45.628241 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:45.628078 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54cc6629-e1ca-4af7-9455-80e3b4d097f6-workload-certs" pod "54cc6629-e1ca-4af7-9455-80e3b4d097f6" (UID: "54cc6629-e1ca-4af7-9455-80e3b4d097f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:58:45.628658 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:45.628624 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54cc6629-e1ca-4af7-9455-80e3b4d097f6-workload-socket" pod "54cc6629-e1ca-4af7-9455-80e3b4d097f6" (UID: "54cc6629-e1ca-4af7-9455-80e3b4d097f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:58:45.628798 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:45.628780 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54cc6629-e1ca-4af7-9455-80e3b4d097f6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54cc6629-e1ca-4af7-9455-80e3b4d097f6-workload-socket\") pod \"54cc6629-e1ca-4af7-9455-80e3b4d097f6\" (UID: \"54cc6629-e1ca-4af7-9455-80e3b4d097f6\") " Oct 30 09:58:45.628971 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:45.628944 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/54cc6629-e1ca-4af7-9455-80e3b4d097f6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54cc6629-e1ca-4af7-9455-80e3b4d097f6-istio-podinfo\") pod \"54cc6629-e1ca-4af7-9455-80e3b4d097f6\" (UID: \"54cc6629-e1ca-4af7-9455-80e3b4d097f6\") " Oct 30 09:58:45.629467 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:45.629451 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54cc6629-e1ca-4af7-9455-80e3b4d097f6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54cc6629-e1ca-4af7-9455-80e3b4d097f6-istiod-ca-cert\") pod \"54cc6629-e1ca-4af7-9455-80e3b4d097f6\" (UID: \"54cc6629-e1ca-4af7-9455-80e3b4d097f6\") " Oct 30 09:58:45.630032 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:45.630014 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54cc6629-e1ca-4af7-9455-80e3b4d097f6-kube-api-access-kg5sd\" (UniqueName: \"kubernetes.io/projected/54cc6629-e1ca-4af7-9455-80e3b4d097f6-kube-api-access-kg5sd\") pod \"54cc6629-e1ca-4af7-9455-80e3b4d097f6\" (UID: \"54cc6629-e1ca-4af7-9455-80e3b4d097f6\") " Oct 30 09:58:45.630536 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:45.629945 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54cc6629-e1ca-4af7-9455-80e3b4d097f6-istiod-ca-cert" pod "54cc6629-e1ca-4af7-9455-80e3b4d097f6" (UID: "54cc6629-e1ca-4af7-9455-80e3b4d097f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:58:45.631838 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:45.631374 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54cc6629-e1ca-4af7-9455-80e3b4d097f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54cc6629-e1ca-4af7-9455-80e3b4d097f6-istio-envoy\") pod \"54cc6629-e1ca-4af7-9455-80e3b4d097f6\" (UID: \"54cc6629-e1ca-4af7-9455-80e3b4d097f6\") " Oct 30 09:58:45.632026 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:45.632007 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54cc6629-e1ca-4af7-9455-80e3b4d097f6-istio-token\" (UniqueName: \"kubernetes.io/projected/54cc6629-e1ca-4af7-9455-80e3b4d097f6-istio-token\") pod \"54cc6629-e1ca-4af7-9455-80e3b4d097f6\" (UID: \"54cc6629-e1ca-4af7-9455-80e3b4d097f6\") " Oct 30 09:58:45.633538 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:45.633206 4562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54cc6629-e1ca-4af7-9455-80e3b4d097f6-istio-data\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:58:45.633538 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:45.633232 4562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54cc6629-e1ca-4af7-9455-80e3b4d097f6-credential-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:58:45.633538 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:45.633248 4562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54cc6629-e1ca-4af7-9455-80e3b4d097f6-workload-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:58:45.633538 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:45.633264 4562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54cc6629-e1ca-4af7-9455-80e3b4d097f6-workload-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:58:45.633538 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:45.633277 4562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54cc6629-e1ca-4af7-9455-80e3b4d097f6-istiod-ca-cert\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:58:45.635612 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:45.635579 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54cc6629-e1ca-4af7-9455-80e3b4d097f6-istio-envoy" pod "54cc6629-e1ca-4af7-9455-80e3b4d097f6" (UID: "54cc6629-e1ca-4af7-9455-80e3b4d097f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:58:45.635612 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:45.635594 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54cc6629-e1ca-4af7-9455-80e3b4d097f6-kube-api-access-kg5sd" pod "54cc6629-e1ca-4af7-9455-80e3b4d097f6" (UID: "54cc6629-e1ca-4af7-9455-80e3b4d097f6"). InnerVolumeSpecName "kube-api-access-kg5sd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:58:45.636167 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:45.635978 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54cc6629-e1ca-4af7-9455-80e3b4d097f6-istio-token" pod "54cc6629-e1ca-4af7-9455-80e3b4d097f6" (UID: "54cc6629-e1ca-4af7-9455-80e3b4d097f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:58:45.636275 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:45.636202 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/54cc6629-e1ca-4af7-9455-80e3b4d097f6-istio-podinfo" pod "54cc6629-e1ca-4af7-9455-80e3b4d097f6" (UID: "54cc6629-e1ca-4af7-9455-80e3b4d097f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:58:45.734029 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:45.733992 4562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54cc6629-e1ca-4af7-9455-80e3b4d097f6-istio-podinfo\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:58:45.734029 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:45.734025 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kg5sd\" (UniqueName: \"kubernetes.io/projected/54cc6629-e1ca-4af7-9455-80e3b4d097f6-kube-api-access-kg5sd\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:58:45.734029 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:45.734037 4562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54cc6629-e1ca-4af7-9455-80e3b4d097f6-istio-envoy\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:58:45.734269 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:45.734046 4562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54cc6629-e1ca-4af7-9455-80e3b4d097f6-istio-token\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:58:45.927664 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:45.927629 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk"] Oct 30 09:58:45.940661 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:45.940631 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-0f6f96-openshift-default-79d9885d49-h8fxk"] Oct 30 09:58:47.875468 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:47.875415 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54cc6629-e1ca-4af7-9455-80e3b4d097f6" path="/var/lib/kubelet/pods/54cc6629-e1ca-4af7-9455-80e3b4d097f6/volumes" Oct 30 09:58:48.103133 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:48.102691 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0f6f96-5d59889f9f-c8xpd" podUID="35bb8805-b6e0-4225-9345-094c08d33935" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:58:51.295703 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:51.295666 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-19c94a-69db9c4498-s4qsp"] Oct 30 09:58:51.330998 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:51.330959 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-19c94a-87d4f8f55-kt6zq"] Oct 30 09:58:51.690828 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:51.690778 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-82ccfd/test-model-82ccfd-kserve-77684cd675-n8rmr" podUID="47f3324c-84c0-4bee-bcfd-88bff68a5e10" containerName="main" containerID="cri-o://e83148d5c124e7e33be60c708dcb74d5d35a6604e22b23261827c2e9b03af8a7" gracePeriod=45 Oct 30 09:58:51.983313 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:51.983288 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-82ccfd/test-model-82ccfd-kserve-77684cd675-n8rmr" Oct 30 09:58:52.110068 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:52.108879 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47f3324c-84c0-4bee-bcfd-88bff68a5e10-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47f3324c-84c0-4bee-bcfd-88bff68a5e10-tmp-dir\") pod \"47f3324c-84c0-4bee-bcfd-88bff68a5e10\" (UID: \"47f3324c-84c0-4bee-bcfd-88bff68a5e10\") " Oct 30 09:58:52.110068 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:52.108995 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47f3324c-84c0-4bee-bcfd-88bff68a5e10-home\" (UniqueName: \"kubernetes.io/empty-dir/47f3324c-84c0-4bee-bcfd-88bff68a5e10-home\") pod \"47f3324c-84c0-4bee-bcfd-88bff68a5e10\" (UID: \"47f3324c-84c0-4bee-bcfd-88bff68a5e10\") " Oct 30 09:58:52.110068 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:52.109027 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47f3324c-84c0-4bee-bcfd-88bff68a5e10-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47f3324c-84c0-4bee-bcfd-88bff68a5e10-model-cache\") pod \"47f3324c-84c0-4bee-bcfd-88bff68a5e10\" (UID: \"47f3324c-84c0-4bee-bcfd-88bff68a5e10\") " Oct 30 09:58:52.110068 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:52.109058 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47f3324c-84c0-4bee-bcfd-88bff68a5e10-kube-api-access-sj9jp\" (UniqueName: \"kubernetes.io/projected/47f3324c-84c0-4bee-bcfd-88bff68a5e10-kube-api-access-sj9jp\") pod \"47f3324c-84c0-4bee-bcfd-88bff68a5e10\" (UID: \"47f3324c-84c0-4bee-bcfd-88bff68a5e10\") " Oct 30 09:58:52.110068 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:52.109177 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47f3324c-84c0-4bee-bcfd-88bff68a5e10-tls-certs\" (UniqueName: \"kubernetes.io/secret/47f3324c-84c0-4bee-bcfd-88bff68a5e10-tls-certs\") pod \"47f3324c-84c0-4bee-bcfd-88bff68a5e10\" (UID: \"47f3324c-84c0-4bee-bcfd-88bff68a5e10\") " Oct 30 09:58:52.110068 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:52.109225 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47f3324c-84c0-4bee-bcfd-88bff68a5e10-dshm\" (UniqueName: \"kubernetes.io/empty-dir/47f3324c-84c0-4bee-bcfd-88bff68a5e10-dshm\") pod \"47f3324c-84c0-4bee-bcfd-88bff68a5e10\" (UID: \"47f3324c-84c0-4bee-bcfd-88bff68a5e10\") " Oct 30 09:58:52.127261 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:52.127206 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47f3324c-84c0-4bee-bcfd-88bff68a5e10-tmp-dir" pod "47f3324c-84c0-4bee-bcfd-88bff68a5e10" (UID: "47f3324c-84c0-4bee-bcfd-88bff68a5e10"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:58:52.129800 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:52.129423 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47f3324c-84c0-4bee-bcfd-88bff68a5e10-model-cache" pod "47f3324c-84c0-4bee-bcfd-88bff68a5e10" (UID: "47f3324c-84c0-4bee-bcfd-88bff68a5e10"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:58:52.136518 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:52.136408 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47f3324c-84c0-4bee-bcfd-88bff68a5e10-kube-api-access-sj9jp" pod "47f3324c-84c0-4bee-bcfd-88bff68a5e10" (UID: "47f3324c-84c0-4bee-bcfd-88bff68a5e10"). InnerVolumeSpecName "kube-api-access-sj9jp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:58:52.146824 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:52.138456 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47f3324c-84c0-4bee-bcfd-88bff68a5e10-home" pod "47f3324c-84c0-4bee-bcfd-88bff68a5e10" (UID: "47f3324c-84c0-4bee-bcfd-88bff68a5e10"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:58:52.146824 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:52.143820 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47f3324c-84c0-4bee-bcfd-88bff68a5e10-tls-certs" pod "47f3324c-84c0-4bee-bcfd-88bff68a5e10" (UID: "47f3324c-84c0-4bee-bcfd-88bff68a5e10"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:58:52.154739 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:52.154712 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47f3324c-84c0-4bee-bcfd-88bff68a5e10-dshm" pod "47f3324c-84c0-4bee-bcfd-88bff68a5e10" (UID: "47f3324c-84c0-4bee-bcfd-88bff68a5e10"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:58:52.211229 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:52.211109 4562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47f3324c-84c0-4bee-bcfd-88bff68a5e10-home\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:58:52.211229 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:52.211139 4562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47f3324c-84c0-4bee-bcfd-88bff68a5e10-model-cache\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:58:52.211229 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:52.211149 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sj9jp\" (UniqueName: \"kubernetes.io/projected/47f3324c-84c0-4bee-bcfd-88bff68a5e10-kube-api-access-sj9jp\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:58:52.211229 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:52.211157 4562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47f3324c-84c0-4bee-bcfd-88bff68a5e10-tls-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:58:52.211229 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:52.211169 4562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47f3324c-84c0-4bee-bcfd-88bff68a5e10-dshm\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:58:52.211229 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:52.211180 4562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47f3324c-84c0-4bee-bcfd-88bff68a5e10-tmp-dir\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:58:52.629442 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:52.629338 4562 generic.go:356] "Generic (PLEG): container finished" podID="47f3324c-84c0-4bee-bcfd-88bff68a5e10" containerID="e83148d5c124e7e33be60c708dcb74d5d35a6604e22b23261827c2e9b03af8a7" exitCode=0 Oct 30 09:58:52.629442 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:52.629403 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-82ccfd/test-model-82ccfd-kserve-77684cd675-n8rmr" Oct 30 09:58:52.629442 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:52.629428 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-82ccfd/test-model-82ccfd-kserve-77684cd675-n8rmr" event={"ID":"47f3324c-84c0-4bee-bcfd-88bff68a5e10","Type":"ContainerDied","Data":"e83148d5c124e7e33be60c708dcb74d5d35a6604e22b23261827c2e9b03af8a7"} Oct 30 09:58:52.629930 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:52.629475 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-82ccfd/test-model-82ccfd-kserve-77684cd675-n8rmr" event={"ID":"47f3324c-84c0-4bee-bcfd-88bff68a5e10","Type":"ContainerDied","Data":"4a8a51471582553534f6f296614e07ee22be708bae1fc6757855d3ada6fdf5d1"} Oct 30 09:58:52.629930 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:52.629498 4562 scope.go:122] "RemoveContainer" containerID="e83148d5c124e7e33be60c708dcb74d5d35a6604e22b23261827c2e9b03af8a7" Oct 30 09:58:52.645555 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:52.645531 4562 scope.go:122] "RemoveContainer" containerID="e83148d5c124e7e33be60c708dcb74d5d35a6604e22b23261827c2e9b03af8a7" Oct 30 09:58:52.645974 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:58:52.645943 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e83148d5c124e7e33be60c708dcb74d5d35a6604e22b23261827c2e9b03af8a7\": container with ID starting with e83148d5c124e7e33be60c708dcb74d5d35a6604e22b23261827c2e9b03af8a7 not found: ID does not exist" containerID="e83148d5c124e7e33be60c708dcb74d5d35a6604e22b23261827c2e9b03af8a7" Oct 30 09:58:52.646072 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:52.645985 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e83148d5c124e7e33be60c708dcb74d5d35a6604e22b23261827c2e9b03af8a7"} err="failed to get container status \"e83148d5c124e7e33be60c708dcb74d5d35a6604e22b23261827c2e9b03af8a7\": rpc error: code = NotFound desc = could not find container \"e83148d5c124e7e33be60c708dcb74d5d35a6604e22b23261827c2e9b03af8a7\": container with ID starting with e83148d5c124e7e33be60c708dcb74d5d35a6604e22b23261827c2e9b03af8a7 not found: ID does not exist" Oct 30 09:58:52.686089 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:52.686064 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-82ccfd/test-model-82ccfd-kserve-77684cd675-n8rmr"] Oct 30 09:58:52.698459 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:52.698434 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-82ccfd/test-model-82ccfd-kserve-77684cd675-n8rmr"] Oct 30 09:58:53.433325 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:53.433263 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0f6f96-5d59889f9f-c8xpd" Oct 30 09:58:53.528037 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:53.527971 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35bb8805-b6e0-4225-9345-094c08d33935-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35bb8805-b6e0-4225-9345-094c08d33935-plugins-config-volume\") pod \"35bb8805-b6e0-4225-9345-094c08d33935\" (UID: \"35bb8805-b6e0-4225-9345-094c08d33935\") " Oct 30 09:58:53.530808 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:53.528721 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35bb8805-b6e0-4225-9345-094c08d33935-plugins-config-volume" pod "35bb8805-b6e0-4225-9345-094c08d33935" (UID: "35bb8805-b6e0-4225-9345-094c08d33935"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:58:53.530953 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:53.530931 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35bb8805-b6e0-4225-9345-094c08d33935-kube-api-access-zkqxl\" (UniqueName: \"kubernetes.io/projected/35bb8805-b6e0-4225-9345-094c08d33935-kube-api-access-zkqxl\") pod \"35bb8805-b6e0-4225-9345-094c08d33935\" (UID: \"35bb8805-b6e0-4225-9345-094c08d33935\") " Oct 30 09:58:53.532078 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:53.532049 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35bb8805-b6e0-4225-9345-094c08d33935-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:58:53.536744 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:53.536570 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35bb8805-b6e0-4225-9345-094c08d33935-kube-api-access-zkqxl" pod "35bb8805-b6e0-4225-9345-094c08d33935" (UID: "35bb8805-b6e0-4225-9345-094c08d33935"). InnerVolumeSpecName "kube-api-access-zkqxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:58:53.637491 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:53.637456 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkqxl\" (UniqueName: \"kubernetes.io/projected/35bb8805-b6e0-4225-9345-094c08d33935-kube-api-access-zkqxl\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:58:53.637891 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:53.637642 4562 generic.go:356] "Generic (PLEG): container finished" podID="35bb8805-b6e0-4225-9345-094c08d33935" containerID="bc70985b974bf7ffc1367ee73adf15dcbaec421ca44f004b3b93da529c5f29be" exitCode=0 Oct 30 09:58:53.637891 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:53.637688 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0f6f96-5d59889f9f-c8xpd" Oct 30 09:58:53.637891 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:53.637708 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0f6f96-5d59889f9f-c8xpd" event={"ID":"35bb8805-b6e0-4225-9345-094c08d33935","Type":"ContainerDied","Data":"bc70985b974bf7ffc1367ee73adf15dcbaec421ca44f004b3b93da529c5f29be"} Oct 30 09:58:53.637891 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:53.637762 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0f6f96-5d59889f9f-c8xpd" event={"ID":"35bb8805-b6e0-4225-9345-094c08d33935","Type":"ContainerDied","Data":"f582b735645e31f2315065d58485b9b649424934bc97f1fbdca717b56bd23511"} Oct 30 09:58:53.637891 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:53.637783 4562 scope.go:122] "RemoveContainer" containerID="bc70985b974bf7ffc1367ee73adf15dcbaec421ca44f004b3b93da529c5f29be" Oct 30 09:58:53.648619 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:53.648599 4562 scope.go:122] "RemoveContainer" containerID="bc70985b974bf7ffc1367ee73adf15dcbaec421ca44f004b3b93da529c5f29be" Oct 30 09:58:53.648911 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:58:53.648893 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc70985b974bf7ffc1367ee73adf15dcbaec421ca44f004b3b93da529c5f29be\": container with ID starting with bc70985b974bf7ffc1367ee73adf15dcbaec421ca44f004b3b93da529c5f29be not found: ID does not exist" containerID="bc70985b974bf7ffc1367ee73adf15dcbaec421ca44f004b3b93da529c5f29be" Oct 30 09:58:53.648963 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:53.648918 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc70985b974bf7ffc1367ee73adf15dcbaec421ca44f004b3b93da529c5f29be"} err="failed to get container status \"bc70985b974bf7ffc1367ee73adf15dcbaec421ca44f004b3b93da529c5f29be\": rpc error: code = NotFound desc = could not find container \"bc70985b974bf7ffc1367ee73adf15dcbaec421ca44f004b3b93da529c5f29be\": container with ID starting with bc70985b974bf7ffc1367ee73adf15dcbaec421ca44f004b3b93da529c5f29be not found: ID does not exist" Oct 30 09:58:53.694290 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:53.693530 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0f6f96-5d59889f9f-c8xpd"] Oct 30 09:58:53.699289 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:53.699257 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0f6f96-5d59889f9f-c8xpd"] Oct 30 09:58:53.875140 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:53.875095 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35bb8805-b6e0-4225-9345-094c08d33935" path="/var/lib/kubelet/pods/35bb8805-b6e0-4225-9345-094c08d33935/volumes" Oct 30 09:58:53.875669 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:53.875647 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47f3324c-84c0-4bee-bcfd-88bff68a5e10" path="/var/lib/kubelet/pods/47f3324c-84c0-4bee-bcfd-88bff68a5e10/volumes" Oct 30 09:58:56.296280 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:56.296232 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-19c94a-69db9c4498-s4qsp" podUID="18022e3d-385e-44ce-8c05-e8bae4b030d9" containerName="payload-processing" containerID="cri-o://e3756d3fb7b6f94cb8d13f43a8f527e842d89de77cdd6702de1bad035671b4b1" gracePeriod=25 Oct 30 09:58:56.331481 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:56.331369 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-19c94a-87d4f8f55-kt6zq" podUID="a8c76d69-147f-4a23-84e2-c5aedeaa359d" containerName="payload-pre-processing" containerID="cri-o://3bc4fb1e908cafea88773a7ec913ce01f8d92d8a35cf5b852e9d3e81ca2beb13" gracePeriod=25 Oct 30 09:58:58.596318 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:58:58.596279 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:58:58.764437 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:58:58.764408 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-82ccfd-6b4c8757ff-rc85k"] Oct 30 09:59:00.565451 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:00.565422 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6q7n_1f51db5a-5720-4dea-ad02-f336b35d43b1/ovn-acl-logging/0.log" Oct 30 09:59:00.580013 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:00.579985 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6q7n_1f51db5a-5720-4dea-ad02-f336b35d43b1/ovn-acl-logging/0.log" Oct 30 09:59:00.585151 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:00.585124 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-19c94a-bc94dcbd4-6k9zl"] Oct 30 09:59:00.585417 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:00.585367 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-19c94a-bc94dcbd4-6k9zl" podUID="b5a67e64-3b82-4bb2-b28e-67903deedd63" containerName="maas-api" containerID="cri-o://5cff1b5033977b6ec0a654891935c6cf9d1569c76df0c48c548750cb95f646c6" gracePeriod=30 Oct 30 09:59:00.880534 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:00.880505 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-19c94a-bc94dcbd4-6k9zl" Oct 30 09:59:00.927404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:00.927077 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5a67e64-3b82-4bb2-b28e-67903deedd63-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5a67e64-3b82-4bb2-b28e-67903deedd63-maas-api-tls\") pod \"b5a67e64-3b82-4bb2-b28e-67903deedd63\" (UID: \"b5a67e64-3b82-4bb2-b28e-67903deedd63\") " Oct 30 09:59:00.927404 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:00.927283 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5a67e64-3b82-4bb2-b28e-67903deedd63-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5a67e64-3b82-4bb2-b28e-67903deedd63-openshift-service-ca\") pod \"b5a67e64-3b82-4bb2-b28e-67903deedd63\" (UID: \"b5a67e64-3b82-4bb2-b28e-67903deedd63\") " Oct 30 09:59:00.928926 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:00.927751 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5a67e64-3b82-4bb2-b28e-67903deedd63-kube-api-access-mx6bk\" (UniqueName: \"kubernetes.io/projected/b5a67e64-3b82-4bb2-b28e-67903deedd63-kube-api-access-mx6bk\") pod \"b5a67e64-3b82-4bb2-b28e-67903deedd63\" (UID: \"b5a67e64-3b82-4bb2-b28e-67903deedd63\") " Oct 30 09:59:00.928926 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:00.927879 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5a67e64-3b82-4bb2-b28e-67903deedd63-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5a67e64-3b82-4bb2-b28e-67903deedd63-metrics-certs\") pod \"b5a67e64-3b82-4bb2-b28e-67903deedd63\" (UID: \"b5a67e64-3b82-4bb2-b28e-67903deedd63\") " Oct 30 09:59:00.930487 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:00.930455 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5a67e64-3b82-4bb2-b28e-67903deedd63-openshift-service-ca" pod "b5a67e64-3b82-4bb2-b28e-67903deedd63" (UID: "b5a67e64-3b82-4bb2-b28e-67903deedd63"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:59:00.963411 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:00.960839 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5a67e64-3b82-4bb2-b28e-67903deedd63-metrics-certs" pod "b5a67e64-3b82-4bb2-b28e-67903deedd63" (UID: "b5a67e64-3b82-4bb2-b28e-67903deedd63"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:59:00.963411 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:00.960864 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5a67e64-3b82-4bb2-b28e-67903deedd63-maas-api-tls" pod "b5a67e64-3b82-4bb2-b28e-67903deedd63" (UID: "b5a67e64-3b82-4bb2-b28e-67903deedd63"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:59:00.984681 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:00.984637 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5a67e64-3b82-4bb2-b28e-67903deedd63-kube-api-access-mx6bk" pod "b5a67e64-3b82-4bb2-b28e-67903deedd63" (UID: "b5a67e64-3b82-4bb2-b28e-67903deedd63"). InnerVolumeSpecName "kube-api-access-mx6bk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:59:01.029958 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:01.029924 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx6bk\" (UniqueName: \"kubernetes.io/projected/b5a67e64-3b82-4bb2-b28e-67903deedd63-kube-api-access-mx6bk\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:59:01.029958 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:01.029956 4562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5a67e64-3b82-4bb2-b28e-67903deedd63-metrics-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:59:01.029958 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:01.029965 4562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5a67e64-3b82-4bb2-b28e-67903deedd63-maas-api-tls\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:59:01.029958 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:01.029973 4562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5a67e64-3b82-4bb2-b28e-67903deedd63-openshift-service-ca\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:59:01.689706 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:01.689667 4562 generic.go:356] "Generic (PLEG): container finished" podID="b5a67e64-3b82-4bb2-b28e-67903deedd63" containerID="5cff1b5033977b6ec0a654891935c6cf9d1569c76df0c48c548750cb95f646c6" exitCode=0 Oct 30 09:59:01.690099 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:01.689714 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-19c94a-bc94dcbd4-6k9zl" Oct 30 09:59:01.690099 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:01.689753 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-19c94a-bc94dcbd4-6k9zl" event={"ID":"b5a67e64-3b82-4bb2-b28e-67903deedd63","Type":"ContainerDied","Data":"5cff1b5033977b6ec0a654891935c6cf9d1569c76df0c48c548750cb95f646c6"} Oct 30 09:59:01.690099 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:01.689810 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-19c94a-bc94dcbd4-6k9zl" event={"ID":"b5a67e64-3b82-4bb2-b28e-67903deedd63","Type":"ContainerDied","Data":"4055948bf9bbca6f28a0a486847799624931789a33097c6cea6895e05583f119"} Oct 30 09:59:01.690099 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:01.689833 4562 scope.go:122] "RemoveContainer" containerID="5cff1b5033977b6ec0a654891935c6cf9d1569c76df0c48c548750cb95f646c6" Oct 30 09:59:01.700135 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:01.700119 4562 scope.go:122] "RemoveContainer" containerID="5cff1b5033977b6ec0a654891935c6cf9d1569c76df0c48c548750cb95f646c6" Oct 30 09:59:01.700450 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:59:01.700425 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cff1b5033977b6ec0a654891935c6cf9d1569c76df0c48c548750cb95f646c6\": container with ID starting with 5cff1b5033977b6ec0a654891935c6cf9d1569c76df0c48c548750cb95f646c6 not found: ID does not exist" containerID="5cff1b5033977b6ec0a654891935c6cf9d1569c76df0c48c548750cb95f646c6" Oct 30 09:59:01.700500 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:01.700461 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cff1b5033977b6ec0a654891935c6cf9d1569c76df0c48c548750cb95f646c6"} err="failed to get container status \"5cff1b5033977b6ec0a654891935c6cf9d1569c76df0c48c548750cb95f646c6\": rpc error: code = NotFound desc = could not find container \"5cff1b5033977b6ec0a654891935c6cf9d1569c76df0c48c548750cb95f646c6\": container with ID starting with 5cff1b5033977b6ec0a654891935c6cf9d1569c76df0c48c548750cb95f646c6 not found: ID does not exist" Oct 30 09:59:01.745470 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:01.745443 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-19c94a-bc94dcbd4-6k9zl"] Oct 30 09:59:01.763623 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:01.759913 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-19c94a-bc94dcbd4-6k9zl"] Oct 30 09:59:01.874594 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:01.874512 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5a67e64-3b82-4bb2-b28e-67903deedd63" path="/var/lib/kubelet/pods/b5a67e64-3b82-4bb2-b28e-67903deedd63/volumes" Oct 30 09:59:03.765793 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:03.765470 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82ccfd-6b4c8757ff-rc85k" podUID="2e7713d8-4bdf-4134-9361-e4c0bca6b525" containerName="payload-pre-processing" containerID="cri-o://a8aa32d2203aa520c798dd93955b1022bf6d685380f68ad131ac93d6a131cc31" gracePeriod=25 Oct 30 09:59:03.970029 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:03.969986 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-19c94a-69db9c4498-s4qsp" podUID="18022e3d-385e-44ce-8c05-e8bae4b030d9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:59:03.970203 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:03.970044 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-19c94a-87d4f8f55-kt6zq" podUID="a8c76d69-147f-4a23-84e2-c5aedeaa359d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:59:08.107349 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:08.107317 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt"] Oct 30 09:59:08.107934 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:08.107573 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt" podUID="5f848cb2-5cdc-4f31-b063-8745991865de" containerName="istio-proxy" containerID="cri-o://a5fa224cbc089dbbe3f234b9afb2a1efa705b8512aba57382d3769ee79135d20" gracePeriod=30 Oct 30 09:59:08.736553 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:08.736520 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7c88ae-f6s6b"] Oct 30 09:59:08.737047 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:08.737035 4562 cpu_manager.go:414] "containerMap: removing container" podUID="54cc6629-e1ca-4af7-9455-80e3b4d097f6" containerName="istio-proxy" Oct 30 09:59:08.737100 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:08.737048 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54cc6629-e1ca-4af7-9455-80e3b4d097f6" containerName="istio-proxy" Oct 30 09:59:08.737100 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:08.737070 4562 cpu_manager.go:414] "containerMap: removing container" podUID="b5a67e64-3b82-4bb2-b28e-67903deedd63" containerName="maas-api" Oct 30 09:59:08.737100 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:08.737075 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5a67e64-3b82-4bb2-b28e-67903deedd63" containerName="maas-api" Oct 30 09:59:08.737100 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:08.737085 4562 cpu_manager.go:414] "containerMap: removing container" podUID="007dfc94-fbc7-456d-8760-2235a48f4277" containerName="revoke-keys" Oct 30 09:59:08.737100 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:08.737090 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="007dfc94-fbc7-456d-8760-2235a48f4277" containerName="revoke-keys" Oct 30 09:59:08.737100 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:08.737097 4562 cpu_manager.go:414] "containerMap: removing container" podUID="cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4" containerName="istio-proxy" Oct 30 09:59:08.737100 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:08.737101 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4" containerName="istio-proxy" Oct 30 09:59:08.737348 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:08.737109 4562 cpu_manager.go:414] "containerMap: removing container" podUID="47f3324c-84c0-4bee-bcfd-88bff68a5e10" containerName="main" Oct 30 09:59:08.737348 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:08.737113 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47f3324c-84c0-4bee-bcfd-88bff68a5e10" containerName="main" Oct 30 09:59:08.737348 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:08.737129 4562 cpu_manager.go:414] "containerMap: removing container" podUID="54cc6629-e1ca-4af7-9455-80e3b4d097f6" containerName="istio-proxy" Oct 30 09:59:08.737348 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:08.737133 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54cc6629-e1ca-4af7-9455-80e3b4d097f6" containerName="istio-proxy" Oct 30 09:59:08.737348 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:08.737140 4562 cpu_manager.go:414] "containerMap: removing container" podUID="35bb8805-b6e0-4225-9345-094c08d33935" containerName="payload-pre-processing" Oct 30 09:59:08.737348 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:08.737145 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35bb8805-b6e0-4225-9345-094c08d33935" containerName="payload-pre-processing" Oct 30 09:59:08.737348 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:08.737221 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="54cc6629-e1ca-4af7-9455-80e3b4d097f6" containerName="istio-proxy" Oct 30 09:59:08.737348 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:08.737229 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="35bb8805-b6e0-4225-9345-094c08d33935" containerName="payload-pre-processing" Oct 30 09:59:08.737348 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:08.737241 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5a67e64-3b82-4bb2-b28e-67903deedd63" containerName="maas-api" Oct 30 09:59:08.737348 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:08.737248 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="007dfc94-fbc7-456d-8760-2235a48f4277" containerName="revoke-keys" Oct 30 09:59:08.737348 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:08.737254 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb592c8a-aca2-4ba5-9de6-51d9cfc3bfb4" containerName="istio-proxy" Oct 30 09:59:08.737348 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:08.737259 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="47f3324c-84c0-4bee-bcfd-88bff68a5e10" containerName="main" Oct 30 09:59:08.740183 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:08.740164 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7c88ae-f6s6b" Oct 30 09:59:08.756374 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:08.756339 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:59:08.756821 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:08.756572 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:59:08.758266 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:08.758243 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7c88ae-f6s6b"] Oct 30 09:59:08.830453 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:08.830415 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq2bd\" (UniqueName: \"kubernetes.io/projected/fac89546-03b2-4263-aa81-9c59d404d0ad-kube-api-access-gq2bd\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-7c88ae-f6s6b\" (UID: \"fac89546-03b2-4263-aa81-9c59d404d0ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7c88ae-f6s6b" Oct 30 09:59:08.830633 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:08.830479 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fac89546-03b2-4263-aa81-9c59d404d0ad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-7c88ae-f6s6b\" (UID: \"fac89546-03b2-4263-aa81-9c59d404d0ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7c88ae-f6s6b" Oct 30 09:59:08.830633 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:08.830577 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fac89546-03b2-4263-aa81-9c59d404d0ad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-7c88ae-f6s6b\" (UID: \"fac89546-03b2-4263-aa81-9c59d404d0ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7c88ae-f6s6b" Oct 30 09:59:08.932202 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:08.932165 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fac89546-03b2-4263-aa81-9c59d404d0ad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-7c88ae-f6s6b\" (UID: \"fac89546-03b2-4263-aa81-9c59d404d0ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7c88ae-f6s6b" Oct 30 09:59:08.932425 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:08.932221 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fac89546-03b2-4263-aa81-9c59d404d0ad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-7c88ae-f6s6b\" (UID: \"fac89546-03b2-4263-aa81-9c59d404d0ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7c88ae-f6s6b" Oct 30 09:59:08.932425 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:08.932356 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq2bd\" (UniqueName: \"kubernetes.io/projected/fac89546-03b2-4263-aa81-9c59d404d0ad-kube-api-access-gq2bd\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-7c88ae-f6s6b\" (UID: \"fac89546-03b2-4263-aa81-9c59d404d0ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7c88ae-f6s6b" Oct 30 09:59:08.932927 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:08.932902 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fac89546-03b2-4263-aa81-9c59d404d0ad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-7c88ae-f6s6b\" (UID: \"fac89546-03b2-4263-aa81-9c59d404d0ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7c88ae-f6s6b" Oct 30 09:59:08.934996 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:08.934963 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fac89546-03b2-4263-aa81-9c59d404d0ad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-7c88ae-f6s6b\" (UID: \"fac89546-03b2-4263-aa81-9c59d404d0ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7c88ae-f6s6b" Oct 30 09:59:08.952113 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:08.952086 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq2bd\" (UniqueName: \"kubernetes.io/projected/fac89546-03b2-4263-aa81-9c59d404d0ad-kube-api-access-gq2bd\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-7c88ae-f6s6b\" (UID: \"fac89546-03b2-4263-aa81-9c59d404d0ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7c88ae-f6s6b" Oct 30 09:59:09.055135 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:09.055039 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7c88ae-f6s6b" Oct 30 09:59:09.363859 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:09.363822 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7c88ae-f6s6b"] Oct 30 09:59:09.739774 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:09.739738 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7c88ae-f6s6b" event={"ID":"fac89546-03b2-4263-aa81-9c59d404d0ad","Type":"ContainerStarted","Data":"67d56d8ace0c0c533449dbb8ad7b58742c4f8be4a4b5665516ebcb55562e5387"} Oct 30 09:59:09.739968 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:09.739805 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7c88ae-f6s6b" event={"ID":"fac89546-03b2-4263-aa81-9c59d404d0ad","Type":"ContainerStarted","Data":"a8416561a3778fc8a73e2c42b80c7a1533e719381efbbd514432b0fca5548c35"} Oct 30 09:59:09.778708 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:09.778649 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7c88ae-f6s6b" podStartSLOduration=1.77863405 podStartE2EDuration="1.77863405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:59:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:59:09.777941049 +0000 UTC m=+3910.039502039" watchObservedRunningTime="2026-10-30 09:59:09.77863405 +0000 UTC m=+3910.040195074" Oct 30 09:59:10.185782 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:10.185744 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82ccfd-6b4c8757ff-rc85k" podUID="2e7713d8-4bdf-4134-9361-e4c0bca6b525" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:59:10.476474 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:10.476441 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-82ccfd-6df95cf9bc-9nhdz"] Oct 30 09:59:10.476962 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:10.476693 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82ccfd-6df95cf9bc-9nhdz" podUID="723c7f11-8c4d-4095-a4c4-fd76af32865c" containerName="maas-api" containerID="cri-o://aa6556277b4ae257e0548d8d44cec0154ca11ad9a1a84519889047e19ea12471" gracePeriod=30 Oct 30 09:59:10.756150 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:10.756054 4562 generic.go:356] "Generic (PLEG): container finished" podID="fac89546-03b2-4263-aa81-9c59d404d0ad" containerID="67d56d8ace0c0c533449dbb8ad7b58742c4f8be4a4b5665516ebcb55562e5387" exitCode=0 Oct 30 09:59:10.756150 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:10.756119 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7c88ae-f6s6b" event={"ID":"fac89546-03b2-4263-aa81-9c59d404d0ad","Type":"ContainerDied","Data":"67d56d8ace0c0c533449dbb8ad7b58742c4f8be4a4b5665516ebcb55562e5387"} Oct 30 09:59:11.749762 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:11.749736 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82ccfd-6df95cf9bc-9nhdz" Oct 30 09:59:11.762277 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:11.762249 4562 generic.go:356] "Generic (PLEG): container finished" podID="723c7f11-8c4d-4095-a4c4-fd76af32865c" containerID="aa6556277b4ae257e0548d8d44cec0154ca11ad9a1a84519889047e19ea12471" exitCode=0 Oct 30 09:59:11.762421 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:11.762290 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82ccfd-6df95cf9bc-9nhdz" Oct 30 09:59:11.762421 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:11.762333 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82ccfd-6df95cf9bc-9nhdz" event={"ID":"723c7f11-8c4d-4095-a4c4-fd76af32865c","Type":"ContainerDied","Data":"aa6556277b4ae257e0548d8d44cec0154ca11ad9a1a84519889047e19ea12471"} Oct 30 09:59:11.762505 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:11.762422 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82ccfd-6df95cf9bc-9nhdz" event={"ID":"723c7f11-8c4d-4095-a4c4-fd76af32865c","Type":"ContainerDied","Data":"01e28ca412f2671c79668df406293db4c6113e9d2c88b6e1055313228c58cbc5"} Oct 30 09:59:11.762505 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:11.762449 4562 scope.go:122] "RemoveContainer" containerID="aa6556277b4ae257e0548d8d44cec0154ca11ad9a1a84519889047e19ea12471" Oct 30 09:59:11.771971 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:11.771945 4562 scope.go:122] "RemoveContainer" containerID="aa6556277b4ae257e0548d8d44cec0154ca11ad9a1a84519889047e19ea12471" Oct 30 09:59:11.772298 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:59:11.772272 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa6556277b4ae257e0548d8d44cec0154ca11ad9a1a84519889047e19ea12471\": container with ID starting with aa6556277b4ae257e0548d8d44cec0154ca11ad9a1a84519889047e19ea12471 not found: ID does not exist" containerID="aa6556277b4ae257e0548d8d44cec0154ca11ad9a1a84519889047e19ea12471" Oct 30 09:59:11.772470 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:11.772319 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa6556277b4ae257e0548d8d44cec0154ca11ad9a1a84519889047e19ea12471"} err="failed to get container status \"aa6556277b4ae257e0548d8d44cec0154ca11ad9a1a84519889047e19ea12471\": rpc error: code = NotFound desc = could not find container \"aa6556277b4ae257e0548d8d44cec0154ca11ad9a1a84519889047e19ea12471\": container with ID starting with aa6556277b4ae257e0548d8d44cec0154ca11ad9a1a84519889047e19ea12471 not found: ID does not exist" Oct 30 09:59:11.873153 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:11.873126 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/723c7f11-8c4d-4095-a4c4-fd76af32865c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/723c7f11-8c4d-4095-a4c4-fd76af32865c-metrics-certs\") pod \"723c7f11-8c4d-4095-a4c4-fd76af32865c\" (UID: \"723c7f11-8c4d-4095-a4c4-fd76af32865c\") " Oct 30 09:59:11.873327 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:11.873250 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/723c7f11-8c4d-4095-a4c4-fd76af32865c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/723c7f11-8c4d-4095-a4c4-fd76af32865c-maas-api-tls\") pod \"723c7f11-8c4d-4095-a4c4-fd76af32865c\" (UID: \"723c7f11-8c4d-4095-a4c4-fd76af32865c\") " Oct 30 09:59:11.873327 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:11.873299 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/723c7f11-8c4d-4095-a4c4-fd76af32865c-kube-api-access-lvdgw\" (UniqueName: \"kubernetes.io/projected/723c7f11-8c4d-4095-a4c4-fd76af32865c-kube-api-access-lvdgw\") pod \"723c7f11-8c4d-4095-a4c4-fd76af32865c\" (UID: \"723c7f11-8c4d-4095-a4c4-fd76af32865c\") " Oct 30 09:59:11.873565 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:11.873397 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/723c7f11-8c4d-4095-a4c4-fd76af32865c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/723c7f11-8c4d-4095-a4c4-fd76af32865c-openshift-service-ca\") pod \"723c7f11-8c4d-4095-a4c4-fd76af32865c\" (UID: \"723c7f11-8c4d-4095-a4c4-fd76af32865c\") " Oct 30 09:59:11.874138 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:11.874108 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/723c7f11-8c4d-4095-a4c4-fd76af32865c-openshift-service-ca" pod "723c7f11-8c4d-4095-a4c4-fd76af32865c" (UID: "723c7f11-8c4d-4095-a4c4-fd76af32865c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:59:11.879421 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:11.876339 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/723c7f11-8c4d-4095-a4c4-fd76af32865c-metrics-certs" pod "723c7f11-8c4d-4095-a4c4-fd76af32865c" (UID: "723c7f11-8c4d-4095-a4c4-fd76af32865c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:59:11.879421 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:11.876362 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/723c7f11-8c4d-4095-a4c4-fd76af32865c-maas-api-tls" pod "723c7f11-8c4d-4095-a4c4-fd76af32865c" (UID: "723c7f11-8c4d-4095-a4c4-fd76af32865c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:59:11.879421 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:11.876341 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/723c7f11-8c4d-4095-a4c4-fd76af32865c-kube-api-access-lvdgw" pod "723c7f11-8c4d-4095-a4c4-fd76af32865c" (UID: "723c7f11-8c4d-4095-a4c4-fd76af32865c"). InnerVolumeSpecName "kube-api-access-lvdgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:59:11.921652 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:11.921629 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7c88ae-f6s6b" Oct 30 09:59:11.974571 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:11.974543 4562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/723c7f11-8c4d-4095-a4c4-fd76af32865c-maas-api-tls\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:59:11.974571 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:11.974573 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvdgw\" (UniqueName: \"kubernetes.io/projected/723c7f11-8c4d-4095-a4c4-fd76af32865c-kube-api-access-lvdgw\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:59:11.974769 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:11.974588 4562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/723c7f11-8c4d-4095-a4c4-fd76af32865c-openshift-service-ca\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:59:11.974769 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:11.974603 4562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/723c7f11-8c4d-4095-a4c4-fd76af32865c-metrics-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:59:12.075535 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:12.075454 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fac89546-03b2-4263-aa81-9c59d404d0ad-kube-api-access-gq2bd\" (UniqueName: \"kubernetes.io/projected/fac89546-03b2-4263-aa81-9c59d404d0ad-kube-api-access-gq2bd\") pod \"fac89546-03b2-4263-aa81-9c59d404d0ad\" (UID: \"fac89546-03b2-4263-aa81-9c59d404d0ad\") " Oct 30 09:59:12.075535 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:12.075504 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fac89546-03b2-4263-aa81-9c59d404d0ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fac89546-03b2-4263-aa81-9c59d404d0ad-maas-api-xks-ca\") pod \"fac89546-03b2-4263-aa81-9c59d404d0ad\" (UID: \"fac89546-03b2-4263-aa81-9c59d404d0ad\") " Oct 30 09:59:12.075789 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:12.075615 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fac89546-03b2-4263-aa81-9c59d404d0ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fac89546-03b2-4263-aa81-9c59d404d0ad-maas-api-openshift-ca\") pod \"fac89546-03b2-4263-aa81-9c59d404d0ad\" (UID: \"fac89546-03b2-4263-aa81-9c59d404d0ad\") " Oct 30 09:59:12.076178 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:12.076146 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fac89546-03b2-4263-aa81-9c59d404d0ad-maas-api-openshift-ca" pod "fac89546-03b2-4263-aa81-9c59d404d0ad" (UID: "fac89546-03b2-4263-aa81-9c59d404d0ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:59:12.077881 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:12.077855 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fac89546-03b2-4263-aa81-9c59d404d0ad-maas-api-xks-ca" pod "fac89546-03b2-4263-aa81-9c59d404d0ad" (UID: "fac89546-03b2-4263-aa81-9c59d404d0ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:59:12.079308 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:12.079281 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fac89546-03b2-4263-aa81-9c59d404d0ad-kube-api-access-gq2bd" pod "fac89546-03b2-4263-aa81-9c59d404d0ad" (UID: "fac89546-03b2-4263-aa81-9c59d404d0ad"). InnerVolumeSpecName "kube-api-access-gq2bd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:59:12.103258 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:12.103233 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-82ccfd-6df95cf9bc-9nhdz"] Oct 30 09:59:12.107017 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:12.106992 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-82ccfd-6df95cf9bc-9nhdz"] Oct 30 09:59:12.177633 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:12.177603 4562 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fac89546-03b2-4263-aa81-9c59d404d0ad-maas-api-openshift-ca\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:59:12.177633 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:12.177633 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq2bd\" (UniqueName: \"kubernetes.io/projected/fac89546-03b2-4263-aa81-9c59d404d0ad-kube-api-access-gq2bd\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:59:12.177835 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:12.177648 4562 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fac89546-03b2-4263-aa81-9c59d404d0ad-maas-api-xks-ca\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:59:12.770602 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:12.770562 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7c88ae-f6s6b" event={"ID":"fac89546-03b2-4263-aa81-9c59d404d0ad","Type":"ContainerDied","Data":"a8416561a3778fc8a73e2c42b80c7a1533e719381efbbd514432b0fca5548c35"} Oct 30 09:59:12.770985 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:12.770615 4562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8416561a3778fc8a73e2c42b80c7a1533e719381efbbd514432b0fca5548c35" Oct 30 09:59:12.770985 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:12.770577 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7c88ae-f6s6b" Oct 30 09:59:13.386784 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:13.386727 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt" Oct 30 09:59:13.493283 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:13.493248 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f848cb2-5cdc-4f31-b063-8745991865de-kube-api-access-rcnlw\" (UniqueName: \"kubernetes.io/projected/5f848cb2-5cdc-4f31-b063-8745991865de-kube-api-access-rcnlw\") pod \"5f848cb2-5cdc-4f31-b063-8745991865de\" (UID: \"5f848cb2-5cdc-4f31-b063-8745991865de\") " Oct 30 09:59:13.493506 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:13.493291 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5f848cb2-5cdc-4f31-b063-8745991865de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f848cb2-5cdc-4f31-b063-8745991865de-istio-podinfo\") pod \"5f848cb2-5cdc-4f31-b063-8745991865de\" (UID: \"5f848cb2-5cdc-4f31-b063-8745991865de\") " Oct 30 09:59:13.493506 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:13.493409 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f848cb2-5cdc-4f31-b063-8745991865de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f848cb2-5cdc-4f31-b063-8745991865de-istio-data\") pod \"5f848cb2-5cdc-4f31-b063-8745991865de\" (UID: \"5f848cb2-5cdc-4f31-b063-8745991865de\") " Oct 30 09:59:13.493506 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:13.493479 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f848cb2-5cdc-4f31-b063-8745991865de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f848cb2-5cdc-4f31-b063-8745991865de-istiod-ca-cert\") pod \"5f848cb2-5cdc-4f31-b063-8745991865de\" (UID: \"5f848cb2-5cdc-4f31-b063-8745991865de\") " Oct 30 09:59:13.493769 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:13.493586 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f848cb2-5cdc-4f31-b063-8745991865de-istio-token\" (UniqueName: \"kubernetes.io/projected/5f848cb2-5cdc-4f31-b063-8745991865de-istio-token\") pod \"5f848cb2-5cdc-4f31-b063-8745991865de\" (UID: \"5f848cb2-5cdc-4f31-b063-8745991865de\") " Oct 30 09:59:13.493769 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:13.493655 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f848cb2-5cdc-4f31-b063-8745991865de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f848cb2-5cdc-4f31-b063-8745991865de-workload-socket\") pod \"5f848cb2-5cdc-4f31-b063-8745991865de\" (UID: \"5f848cb2-5cdc-4f31-b063-8745991865de\") " Oct 30 09:59:13.493769 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:13.493704 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f848cb2-5cdc-4f31-b063-8745991865de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f848cb2-5cdc-4f31-b063-8745991865de-istio-envoy\") pod \"5f848cb2-5cdc-4f31-b063-8745991865de\" (UID: \"5f848cb2-5cdc-4f31-b063-8745991865de\") " Oct 30 09:59:13.493769 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:13.493746 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f848cb2-5cdc-4f31-b063-8745991865de-istio-data" pod "5f848cb2-5cdc-4f31-b063-8745991865de" (UID: "5f848cb2-5cdc-4f31-b063-8745991865de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:13.493992 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:13.493773 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f848cb2-5cdc-4f31-b063-8745991865de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f848cb2-5cdc-4f31-b063-8745991865de-workload-certs\") pod \"5f848cb2-5cdc-4f31-b063-8745991865de\" (UID: \"5f848cb2-5cdc-4f31-b063-8745991865de\") " Oct 30 09:59:13.493992 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:13.493802 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f848cb2-5cdc-4f31-b063-8745991865de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f848cb2-5cdc-4f31-b063-8745991865de-credential-socket\") pod \"5f848cb2-5cdc-4f31-b063-8745991865de\" (UID: \"5f848cb2-5cdc-4f31-b063-8745991865de\") " Oct 30 09:59:13.493992 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:13.493949 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f848cb2-5cdc-4f31-b063-8745991865de-istiod-ca-cert" pod "5f848cb2-5cdc-4f31-b063-8745991865de" (UID: "5f848cb2-5cdc-4f31-b063-8745991865de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:59:13.494156 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:13.494091 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f848cb2-5cdc-4f31-b063-8745991865de-workload-certs" pod "5f848cb2-5cdc-4f31-b063-8745991865de" (UID: "5f848cb2-5cdc-4f31-b063-8745991865de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:13.494243 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:13.494161 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f848cb2-5cdc-4f31-b063-8745991865de-workload-socket" pod "5f848cb2-5cdc-4f31-b063-8745991865de" (UID: "5f848cb2-5cdc-4f31-b063-8745991865de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:13.494243 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:13.494186 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f848cb2-5cdc-4f31-b063-8745991865de-credential-socket" pod "5f848cb2-5cdc-4f31-b063-8745991865de" (UID: "5f848cb2-5cdc-4f31-b063-8745991865de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:13.494418 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:13.494377 4562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f848cb2-5cdc-4f31-b063-8745991865de-istio-data\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:59:13.494504 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:13.494424 4562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f848cb2-5cdc-4f31-b063-8745991865de-istiod-ca-cert\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:59:13.494504 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:13.494439 4562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f848cb2-5cdc-4f31-b063-8745991865de-workload-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:59:13.494504 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:13.494453 4562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f848cb2-5cdc-4f31-b063-8745991865de-workload-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:59:13.494504 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:13.494469 4562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f848cb2-5cdc-4f31-b063-8745991865de-credential-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:59:13.495643 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:13.495616 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f848cb2-5cdc-4f31-b063-8745991865de-kube-api-access-rcnlw" pod "5f848cb2-5cdc-4f31-b063-8745991865de" (UID: "5f848cb2-5cdc-4f31-b063-8745991865de"). InnerVolumeSpecName "kube-api-access-rcnlw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:59:13.495765 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:13.495736 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f848cb2-5cdc-4f31-b063-8745991865de-istio-token" pod "5f848cb2-5cdc-4f31-b063-8745991865de" (UID: "5f848cb2-5cdc-4f31-b063-8745991865de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:59:13.495765 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:13.495756 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f848cb2-5cdc-4f31-b063-8745991865de-istio-podinfo" pod "5f848cb2-5cdc-4f31-b063-8745991865de" (UID: "5f848cb2-5cdc-4f31-b063-8745991865de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:59:13.495893 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:13.495874 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f848cb2-5cdc-4f31-b063-8745991865de-istio-envoy" pod "5f848cb2-5cdc-4f31-b063-8745991865de" (UID: "5f848cb2-5cdc-4f31-b063-8745991865de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:13.595867 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:13.595827 4562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f848cb2-5cdc-4f31-b063-8745991865de-istio-token\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:59:13.595867 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:13.595861 4562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f848cb2-5cdc-4f31-b063-8745991865de-istio-envoy\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:59:13.595867 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:13.595870 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcnlw\" (UniqueName: \"kubernetes.io/projected/5f848cb2-5cdc-4f31-b063-8745991865de-kube-api-access-rcnlw\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:59:13.596119 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:13.595880 4562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f848cb2-5cdc-4f31-b063-8745991865de-istio-podinfo\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:59:13.779542 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:13.779506 4562 generic.go:356] "Generic (PLEG): container finished" podID="5f848cb2-5cdc-4f31-b063-8745991865de" containerID="a5fa224cbc089dbbe3f234b9afb2a1efa705b8512aba57382d3769ee79135d20" exitCode=0 Oct 30 09:59:13.779947 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:13.779554 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt" Oct 30 09:59:13.779947 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:13.779574 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt" event={"ID":"5f848cb2-5cdc-4f31-b063-8745991865de","Type":"ContainerDied","Data":"a5fa224cbc089dbbe3f234b9afb2a1efa705b8512aba57382d3769ee79135d20"} Oct 30 09:59:13.779947 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:13.779615 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt" event={"ID":"5f848cb2-5cdc-4f31-b063-8745991865de","Type":"ContainerDied","Data":"db113b981633732aff3eab15ebe56fc3c92a95c982c7420ef87f49f6bf517af8"} Oct 30 09:59:13.779947 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:13.779635 4562 scope.go:122] "RemoveContainer" containerID="a5fa224cbc089dbbe3f234b9afb2a1efa705b8512aba57382d3769ee79135d20" Oct 30 09:59:13.789630 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:13.789608 4562 scope.go:122] "RemoveContainer" containerID="eadf17790c84ef362a51c2eaa13c23aa51452a48c29fc5918c8c6f02ba3eddf2" Oct 30 09:59:13.798353 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:13.798330 4562 scope.go:122] "RemoveContainer" containerID="a5fa224cbc089dbbe3f234b9afb2a1efa705b8512aba57382d3769ee79135d20" Oct 30 09:59:13.798615 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:59:13.798594 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5fa224cbc089dbbe3f234b9afb2a1efa705b8512aba57382d3769ee79135d20\": container with ID starting with a5fa224cbc089dbbe3f234b9afb2a1efa705b8512aba57382d3769ee79135d20 not found: ID does not exist" containerID="a5fa224cbc089dbbe3f234b9afb2a1efa705b8512aba57382d3769ee79135d20" Oct 30 09:59:13.798675 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:13.798623 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5fa224cbc089dbbe3f234b9afb2a1efa705b8512aba57382d3769ee79135d20"} err="failed to get container status \"a5fa224cbc089dbbe3f234b9afb2a1efa705b8512aba57382d3769ee79135d20\": rpc error: code = NotFound desc = could not find container \"a5fa224cbc089dbbe3f234b9afb2a1efa705b8512aba57382d3769ee79135d20\": container with ID starting with a5fa224cbc089dbbe3f234b9afb2a1efa705b8512aba57382d3769ee79135d20 not found: ID does not exist" Oct 30 09:59:13.798675 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:13.798640 4562 scope.go:122] "RemoveContainer" containerID="eadf17790c84ef362a51c2eaa13c23aa51452a48c29fc5918c8c6f02ba3eddf2" Oct 30 09:59:13.798865 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:59:13.798847 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eadf17790c84ef362a51c2eaa13c23aa51452a48c29fc5918c8c6f02ba3eddf2\": container with ID starting with eadf17790c84ef362a51c2eaa13c23aa51452a48c29fc5918c8c6f02ba3eddf2 not found: ID does not exist" containerID="eadf17790c84ef362a51c2eaa13c23aa51452a48c29fc5918c8c6f02ba3eddf2" Oct 30 09:59:13.798926 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:13.798877 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eadf17790c84ef362a51c2eaa13c23aa51452a48c29fc5918c8c6f02ba3eddf2"} err="failed to get container status \"eadf17790c84ef362a51c2eaa13c23aa51452a48c29fc5918c8c6f02ba3eddf2\": rpc error: code = NotFound desc = could not find container \"eadf17790c84ef362a51c2eaa13c23aa51452a48c29fc5918c8c6f02ba3eddf2\": container with ID starting with eadf17790c84ef362a51c2eaa13c23aa51452a48c29fc5918c8c6f02ba3eddf2 not found: ID does not exist" Oct 30 09:59:13.835649 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:13.835621 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt"] Oct 30 09:59:13.840121 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:13.840095 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-19c94a-openshift-default-7fcfb9f54f-qkqrt"] Oct 30 09:59:13.874355 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:13.874322 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f848cb2-5cdc-4f31-b063-8745991865de" path="/var/lib/kubelet/pods/5f848cb2-5cdc-4f31-b063-8745991865de/volumes" Oct 30 09:59:13.874846 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:13.874830 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="723c7f11-8c4d-4095-a4c4-fd76af32865c" path="/var/lib/kubelet/pods/723c7f11-8c4d-4095-a4c4-fd76af32865c/volumes" Oct 30 09:59:13.970409 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:13.970342 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-19c94a-87d4f8f55-kt6zq" podUID="a8c76d69-147f-4a23-84e2-c5aedeaa359d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:59:13.970409 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:13.970346 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-19c94a-69db9c4498-s4qsp" podUID="18022e3d-385e-44ce-8c05-e8bae4b030d9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:59:16.596714 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:16.596688 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-19c94a-69db9c4498-s4qsp" Oct 30 09:59:16.628000 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:16.627975 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-19c94a-87d4f8f55-kt6zq" Oct 30 09:59:16.734173 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:16.734137 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8c76d69-147f-4a23-84e2-c5aedeaa359d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8c76d69-147f-4a23-84e2-c5aedeaa359d-plugins-config-volume\") pod \"a8c76d69-147f-4a23-84e2-c5aedeaa359d\" (UID: \"a8c76d69-147f-4a23-84e2-c5aedeaa359d\") " Oct 30 09:59:16.734344 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:16.734182 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18022e3d-385e-44ce-8c05-e8bae4b030d9-kube-api-access-6m9m8\" (UniqueName: \"kubernetes.io/projected/18022e3d-385e-44ce-8c05-e8bae4b030d9-kube-api-access-6m9m8\") pod \"18022e3d-385e-44ce-8c05-e8bae4b030d9\" (UID: \"18022e3d-385e-44ce-8c05-e8bae4b030d9\") " Oct 30 09:59:16.734344 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:16.734225 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18022e3d-385e-44ce-8c05-e8bae4b030d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18022e3d-385e-44ce-8c05-e8bae4b030d9-plugins-config-volume\") pod \"18022e3d-385e-44ce-8c05-e8bae4b030d9\" (UID: \"18022e3d-385e-44ce-8c05-e8bae4b030d9\") " Oct 30 09:59:16.734344 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:16.734293 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8c76d69-147f-4a23-84e2-c5aedeaa359d-kube-api-access-brmpd\" (UniqueName: \"kubernetes.io/projected/a8c76d69-147f-4a23-84e2-c5aedeaa359d-kube-api-access-brmpd\") pod \"a8c76d69-147f-4a23-84e2-c5aedeaa359d\" (UID: \"a8c76d69-147f-4a23-84e2-c5aedeaa359d\") " Oct 30 09:59:16.734591 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:16.734566 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8c76d69-147f-4a23-84e2-c5aedeaa359d-plugins-config-volume" pod "a8c76d69-147f-4a23-84e2-c5aedeaa359d" (UID: "a8c76d69-147f-4a23-84e2-c5aedeaa359d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:59:16.734736 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:16.734713 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18022e3d-385e-44ce-8c05-e8bae4b030d9-plugins-config-volume" pod "18022e3d-385e-44ce-8c05-e8bae4b030d9" (UID: "18022e3d-385e-44ce-8c05-e8bae4b030d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:59:16.734856 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:16.734838 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8c76d69-147f-4a23-84e2-c5aedeaa359d-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:59:16.734917 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:16.734887 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18022e3d-385e-44ce-8c05-e8bae4b030d9-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:59:16.736369 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:16.736350 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18022e3d-385e-44ce-8c05-e8bae4b030d9-kube-api-access-6m9m8" pod "18022e3d-385e-44ce-8c05-e8bae4b030d9" (UID: "18022e3d-385e-44ce-8c05-e8bae4b030d9"). InnerVolumeSpecName "kube-api-access-6m9m8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:59:16.736369 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:16.736363 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8c76d69-147f-4a23-84e2-c5aedeaa359d-kube-api-access-brmpd" pod "a8c76d69-147f-4a23-84e2-c5aedeaa359d" (UID: "a8c76d69-147f-4a23-84e2-c5aedeaa359d"). InnerVolumeSpecName "kube-api-access-brmpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:59:16.798659 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:16.798631 4562 generic.go:356] "Generic (PLEG): container finished" podID="a8c76d69-147f-4a23-84e2-c5aedeaa359d" containerID="3bc4fb1e908cafea88773a7ec913ce01f8d92d8a35cf5b852e9d3e81ca2beb13" exitCode=0 Oct 30 09:59:16.798824 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:16.798674 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-19c94a-87d4f8f55-kt6zq" Oct 30 09:59:16.798824 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:16.798722 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-19c94a-87d4f8f55-kt6zq" event={"ID":"a8c76d69-147f-4a23-84e2-c5aedeaa359d","Type":"ContainerDied","Data":"3bc4fb1e908cafea88773a7ec913ce01f8d92d8a35cf5b852e9d3e81ca2beb13"} Oct 30 09:59:16.798824 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:16.798783 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-19c94a-87d4f8f55-kt6zq" event={"ID":"a8c76d69-147f-4a23-84e2-c5aedeaa359d","Type":"ContainerDied","Data":"1aeb6a14d3e0105bb4b1154c2814b72cdf2d2c6fe6db91920dd55ac901c54086"} Oct 30 09:59:16.798824 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:16.798818 4562 scope.go:122] "RemoveContainer" containerID="3bc4fb1e908cafea88773a7ec913ce01f8d92d8a35cf5b852e9d3e81ca2beb13" Oct 30 09:59:16.800141 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:16.800114 4562 generic.go:356] "Generic (PLEG): container finished" podID="18022e3d-385e-44ce-8c05-e8bae4b030d9" containerID="e3756d3fb7b6f94cb8d13f43a8f527e842d89de77cdd6702de1bad035671b4b1" exitCode=0 Oct 30 09:59:16.800262 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:16.800162 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-19c94a-69db9c4498-s4qsp" Oct 30 09:59:16.800262 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:16.800206 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-19c94a-69db9c4498-s4qsp" event={"ID":"18022e3d-385e-44ce-8c05-e8bae4b030d9","Type":"ContainerDied","Data":"e3756d3fb7b6f94cb8d13f43a8f527e842d89de77cdd6702de1bad035671b4b1"} Oct 30 09:59:16.800262 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:16.800252 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-19c94a-69db9c4498-s4qsp" event={"ID":"18022e3d-385e-44ce-8c05-e8bae4b030d9","Type":"ContainerDied","Data":"bf2a482f7b499cac076136ae063fd8fd1522785ece943ba3c00e5ee250f84c34"} Oct 30 09:59:16.808536 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:16.808520 4562 scope.go:122] "RemoveContainer" containerID="3bc4fb1e908cafea88773a7ec913ce01f8d92d8a35cf5b852e9d3e81ca2beb13" Oct 30 09:59:16.808825 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:59:16.808801 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bc4fb1e908cafea88773a7ec913ce01f8d92d8a35cf5b852e9d3e81ca2beb13\": container with ID starting with 3bc4fb1e908cafea88773a7ec913ce01f8d92d8a35cf5b852e9d3e81ca2beb13 not found: ID does not exist" containerID="3bc4fb1e908cafea88773a7ec913ce01f8d92d8a35cf5b852e9d3e81ca2beb13" Oct 30 09:59:16.808904 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:16.808833 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bc4fb1e908cafea88773a7ec913ce01f8d92d8a35cf5b852e9d3e81ca2beb13"} err="failed to get container status \"3bc4fb1e908cafea88773a7ec913ce01f8d92d8a35cf5b852e9d3e81ca2beb13\": rpc error: code = NotFound desc = could not find container \"3bc4fb1e908cafea88773a7ec913ce01f8d92d8a35cf5b852e9d3e81ca2beb13\": container with ID starting with 3bc4fb1e908cafea88773a7ec913ce01f8d92d8a35cf5b852e9d3e81ca2beb13 not found: ID does not exist" Oct 30 09:59:16.808904 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:16.808856 4562 scope.go:122] "RemoveContainer" containerID="e3756d3fb7b6f94cb8d13f43a8f527e842d89de77cdd6702de1bad035671b4b1" Oct 30 09:59:16.817334 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:16.817316 4562 scope.go:122] "RemoveContainer" containerID="e3756d3fb7b6f94cb8d13f43a8f527e842d89de77cdd6702de1bad035671b4b1" Oct 30 09:59:16.817631 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:59:16.817612 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3756d3fb7b6f94cb8d13f43a8f527e842d89de77cdd6702de1bad035671b4b1\": container with ID starting with e3756d3fb7b6f94cb8d13f43a8f527e842d89de77cdd6702de1bad035671b4b1 not found: ID does not exist" containerID="e3756d3fb7b6f94cb8d13f43a8f527e842d89de77cdd6702de1bad035671b4b1" Oct 30 09:59:16.817703 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:16.817636 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3756d3fb7b6f94cb8d13f43a8f527e842d89de77cdd6702de1bad035671b4b1"} err="failed to get container status \"e3756d3fb7b6f94cb8d13f43a8f527e842d89de77cdd6702de1bad035671b4b1\": rpc error: code = NotFound desc = could not find container \"e3756d3fb7b6f94cb8d13f43a8f527e842d89de77cdd6702de1bad035671b4b1\": container with ID starting with e3756d3fb7b6f94cb8d13f43a8f527e842d89de77cdd6702de1bad035671b4b1 not found: ID does not exist" Oct 30 09:59:16.836482 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:16.836462 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brmpd\" (UniqueName: \"kubernetes.io/projected/a8c76d69-147f-4a23-84e2-c5aedeaa359d-kube-api-access-brmpd\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:59:16.836579 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:16.836485 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m9m8\" (UniqueName: \"kubernetes.io/projected/18022e3d-385e-44ce-8c05-e8bae4b030d9-kube-api-access-6m9m8\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:59:16.853448 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:16.853417 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-19c94a-69db9c4498-s4qsp"] Oct 30 09:59:16.859117 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:16.859094 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-19c94a-69db9c4498-s4qsp"] Oct 30 09:59:16.892817 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:16.892785 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-19c94a-87d4f8f55-kt6zq"] Oct 30 09:59:16.895145 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:16.895124 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-19c94a-87d4f8f55-kt6zq"] Oct 30 09:59:17.873168 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:17.873133 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18022e3d-385e-44ce-8c05-e8bae4b030d9" path="/var/lib/kubelet/pods/18022e3d-385e-44ce-8c05-e8bae4b030d9/volumes" Oct 30 09:59:17.873689 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:17.873664 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8c76d69-147f-4a23-84e2-c5aedeaa359d" path="/var/lib/kubelet/pods/a8c76d69-147f-4a23-84e2-c5aedeaa359d/volumes" Oct 30 09:59:18.329321 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:18.329283 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq"] Oct 30 09:59:18.329606 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:18.329572 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq" podUID="37d8a054-775f-4a08-baff-90631bbec3be" containerName="istio-proxy" containerID="cri-o://299d3835c6120870e50dde60384201ac3fe0a834ee5a57d76e0d2dd58e810a0b" gracePeriod=30 Oct 30 09:59:18.341820 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:18.341793 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg"] Oct 30 09:59:18.342068 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:18.342041 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg" podUID="72746dcd-3b5e-46fe-8ec7-eb177b70be60" containerName="istio-proxy" containerID="cri-o://a27353c0140cd23dd26d1ca41ca64d69838bda49128753571f1898e9e671319c" gracePeriod=30 Oct 30 09:59:18.980244 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:18.980211 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a479df-bf6sg"] Oct 30 09:59:18.980715 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:18.980701 4562 cpu_manager.go:414] "containerMap: removing container" podUID="a8c76d69-147f-4a23-84e2-c5aedeaa359d" containerName="payload-pre-processing" Oct 30 09:59:18.980759 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:18.980715 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8c76d69-147f-4a23-84e2-c5aedeaa359d" containerName="payload-pre-processing" Oct 30 09:59:18.980759 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:18.980727 4562 cpu_manager.go:414] "containerMap: removing container" podUID="723c7f11-8c4d-4095-a4c4-fd76af32865c" containerName="maas-api" Oct 30 09:59:18.980759 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:18.980733 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="723c7f11-8c4d-4095-a4c4-fd76af32865c" containerName="maas-api" Oct 30 09:59:18.980759 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:18.980748 4562 cpu_manager.go:414] "containerMap: removing container" podUID="5f848cb2-5cdc-4f31-b063-8745991865de" containerName="istio-proxy" Oct 30 09:59:18.980759 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:18.980753 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f848cb2-5cdc-4f31-b063-8745991865de" containerName="istio-proxy" Oct 30 09:59:18.980928 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:18.980762 4562 cpu_manager.go:414] "containerMap: removing container" podUID="5f848cb2-5cdc-4f31-b063-8745991865de" containerName="istio-proxy" Oct 30 09:59:18.980928 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:18.980767 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f848cb2-5cdc-4f31-b063-8745991865de" containerName="istio-proxy" Oct 30 09:59:18.980928 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:18.980776 4562 cpu_manager.go:414] "containerMap: removing container" podUID="fac89546-03b2-4263-aa81-9c59d404d0ad" containerName="revoke-keys" Oct 30 09:59:18.980928 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:18.980782 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fac89546-03b2-4263-aa81-9c59d404d0ad" containerName="revoke-keys" Oct 30 09:59:18.980928 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:18.980809 4562 cpu_manager.go:414] "containerMap: removing container" podUID="18022e3d-385e-44ce-8c05-e8bae4b030d9" containerName="payload-processing" Oct 30 09:59:18.980928 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:18.980813 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18022e3d-385e-44ce-8c05-e8bae4b030d9" containerName="payload-processing" Oct 30 09:59:18.980928 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:18.980887 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="54cc6629-e1ca-4af7-9455-80e3b4d097f6" containerName="istio-proxy" Oct 30 09:59:18.980928 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:18.980898 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="18022e3d-385e-44ce-8c05-e8bae4b030d9" containerName="payload-processing" Oct 30 09:59:18.980928 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:18.980904 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8c76d69-147f-4a23-84e2-c5aedeaa359d" containerName="payload-pre-processing" Oct 30 09:59:18.980928 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:18.980909 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="723c7f11-8c4d-4095-a4c4-fd76af32865c" containerName="maas-api" Oct 30 09:59:18.980928 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:18.980916 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f848cb2-5cdc-4f31-b063-8745991865de" containerName="istio-proxy" Oct 30 09:59:18.980928 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:18.980922 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f848cb2-5cdc-4f31-b063-8745991865de" containerName="istio-proxy" Oct 30 09:59:18.980928 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:18.980927 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="fac89546-03b2-4263-aa81-9c59d404d0ad" containerName="revoke-keys" Oct 30 09:59:18.985589 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:18.985570 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a479df-bf6sg" Oct 30 09:59:19.001783 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:19.001752 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a479df-bf6sg"] Oct 30 09:59:19.010779 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:19.010750 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:59:19.011062 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:19.011038 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:59:19.056265 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:19.056220 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07af07d3-1594-4d42-9b63-af5097dc78b7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a479df-bf6sg\" (UID: \"07af07d3-1594-4d42-9b63-af5097dc78b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a479df-bf6sg" Oct 30 09:59:19.056462 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:19.056360 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn9k8\" (UniqueName: \"kubernetes.io/projected/07af07d3-1594-4d42-9b63-af5097dc78b7-kube-api-access-qn9k8\") pod \"maas-api-revoke-keys-e2e-inf-b-a479df-bf6sg\" (UID: \"07af07d3-1594-4d42-9b63-af5097dc78b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a479df-bf6sg" Oct 30 09:59:19.056539 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:19.056500 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07af07d3-1594-4d42-9b63-af5097dc78b7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a479df-bf6sg\" (UID: \"07af07d3-1594-4d42-9b63-af5097dc78b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a479df-bf6sg" Oct 30 09:59:19.157895 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:19.157857 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn9k8\" (UniqueName: \"kubernetes.io/projected/07af07d3-1594-4d42-9b63-af5097dc78b7-kube-api-access-qn9k8\") pod \"maas-api-revoke-keys-e2e-inf-b-a479df-bf6sg\" (UID: \"07af07d3-1594-4d42-9b63-af5097dc78b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a479df-bf6sg" Oct 30 09:59:19.158105 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:19.157959 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07af07d3-1594-4d42-9b63-af5097dc78b7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a479df-bf6sg\" (UID: \"07af07d3-1594-4d42-9b63-af5097dc78b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a479df-bf6sg" Oct 30 09:59:19.158105 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:19.158022 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07af07d3-1594-4d42-9b63-af5097dc78b7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a479df-bf6sg\" (UID: \"07af07d3-1594-4d42-9b63-af5097dc78b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a479df-bf6sg" Oct 30 09:59:19.158816 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:19.158782 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07af07d3-1594-4d42-9b63-af5097dc78b7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a479df-bf6sg\" (UID: \"07af07d3-1594-4d42-9b63-af5097dc78b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a479df-bf6sg" Oct 30 09:59:19.160502 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:19.160480 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07af07d3-1594-4d42-9b63-af5097dc78b7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a479df-bf6sg\" (UID: \"07af07d3-1594-4d42-9b63-af5097dc78b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a479df-bf6sg" Oct 30 09:59:19.177484 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:19.177460 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn9k8\" (UniqueName: \"kubernetes.io/projected/07af07d3-1594-4d42-9b63-af5097dc78b7-kube-api-access-qn9k8\") pod \"maas-api-revoke-keys-e2e-inf-b-a479df-bf6sg\" (UID: \"07af07d3-1594-4d42-9b63-af5097dc78b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a479df-bf6sg" Oct 30 09:59:19.299327 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:19.299246 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a479df-bf6sg" Oct 30 09:59:19.597546 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:59:19.597516 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07af07d3_1594_4d42_9b63_af5097dc78b7.slice/crio-4622823683e184f2e2015244733b7f6e1e2e5db0b23e22e879fa508426aaebb3 WatchSource:0}: Error finding container 4622823683e184f2e2015244733b7f6e1e2e5db0b23e22e879fa508426aaebb3: Status 404 returned error can't find the container with id 4622823683e184f2e2015244733b7f6e1e2e5db0b23e22e879fa508426aaebb3 Oct 30 09:59:19.598708 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:19.598681 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a479df-bf6sg"] Oct 30 09:59:19.820725 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:19.820693 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a479df-bf6sg" event={"ID":"07af07d3-1594-4d42-9b63-af5097dc78b7","Type":"ContainerStarted","Data":"4622823683e184f2e2015244733b7f6e1e2e5db0b23e22e879fa508426aaebb3"} Oct 30 09:59:20.185630 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:20.185554 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82ccfd-6b4c8757ff-rc85k" podUID="2e7713d8-4bdf-4134-9361-e4c0bca6b525" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:59:20.828353 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:20.828321 4562 generic.go:356] "Generic (PLEG): container finished" podID="07af07d3-1594-4d42-9b63-af5097dc78b7" containerID="70b67ad06014a71f8cd8b77b55b949cfa667c81ff6357f66bb061d8b20cb04ab" exitCode=0 Oct 30 09:59:20.828539 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:20.828366 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a479df-bf6sg" event={"ID":"07af07d3-1594-4d42-9b63-af5097dc78b7","Type":"ContainerDied","Data":"70b67ad06014a71f8cd8b77b55b949cfa667c81ff6357f66bb061d8b20cb04ab"} Oct 30 09:59:21.339832 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:21.339798 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7c88ae-f6s6b"] Oct 30 09:59:21.342319 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:21.342280 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7c88ae-f6s6b"] Oct 30 09:59:21.876050 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:21.876012 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fac89546-03b2-4263-aa81-9c59d404d0ad" path="/var/lib/kubelet/pods/fac89546-03b2-4263-aa81-9c59d404d0ad/volumes" Oct 30 09:59:21.984802 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:21.984778 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a479df-bf6sg" Oct 30 09:59:22.087472 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:22.087392 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07af07d3-1594-4d42-9b63-af5097dc78b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07af07d3-1594-4d42-9b63-af5097dc78b7-maas-api-xks-ca\") pod \"07af07d3-1594-4d42-9b63-af5097dc78b7\" (UID: \"07af07d3-1594-4d42-9b63-af5097dc78b7\") " Oct 30 09:59:22.087616 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:22.087543 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07af07d3-1594-4d42-9b63-af5097dc78b7-kube-api-access-qn9k8\" (UniqueName: \"kubernetes.io/projected/07af07d3-1594-4d42-9b63-af5097dc78b7-kube-api-access-qn9k8\") pod \"07af07d3-1594-4d42-9b63-af5097dc78b7\" (UID: \"07af07d3-1594-4d42-9b63-af5097dc78b7\") " Oct 30 09:59:22.087661 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:22.087613 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07af07d3-1594-4d42-9b63-af5097dc78b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07af07d3-1594-4d42-9b63-af5097dc78b7-maas-api-openshift-ca\") pod \"07af07d3-1594-4d42-9b63-af5097dc78b7\" (UID: \"07af07d3-1594-4d42-9b63-af5097dc78b7\") " Oct 30 09:59:22.088084 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:22.088059 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07af07d3-1594-4d42-9b63-af5097dc78b7-maas-api-openshift-ca" pod "07af07d3-1594-4d42-9b63-af5097dc78b7" (UID: "07af07d3-1594-4d42-9b63-af5097dc78b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:59:22.089893 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:22.089868 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07af07d3-1594-4d42-9b63-af5097dc78b7-maas-api-xks-ca" pod "07af07d3-1594-4d42-9b63-af5097dc78b7" (UID: "07af07d3-1594-4d42-9b63-af5097dc78b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:59:22.090013 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:22.089919 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07af07d3-1594-4d42-9b63-af5097dc78b7-kube-api-access-qn9k8" pod "07af07d3-1594-4d42-9b63-af5097dc78b7" (UID: "07af07d3-1594-4d42-9b63-af5097dc78b7"). InnerVolumeSpecName "kube-api-access-qn9k8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:59:22.189643 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:22.189612 4562 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07af07d3-1594-4d42-9b63-af5097dc78b7-maas-api-openshift-ca\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:59:22.189643 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:22.189641 4562 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07af07d3-1594-4d42-9b63-af5097dc78b7-maas-api-xks-ca\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:59:22.189643 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:22.189650 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn9k8\" (UniqueName: \"kubernetes.io/projected/07af07d3-1594-4d42-9b63-af5097dc78b7-kube-api-access-qn9k8\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:59:22.841244 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:22.841212 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a479df-bf6sg" Oct 30 09:59:22.841683 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:22.841247 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a479df-bf6sg" event={"ID":"07af07d3-1594-4d42-9b63-af5097dc78b7","Type":"ContainerDied","Data":"4622823683e184f2e2015244733b7f6e1e2e5db0b23e22e879fa508426aaebb3"} Oct 30 09:59:22.841683 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:22.841295 4562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4622823683e184f2e2015244733b7f6e1e2e5db0b23e22e879fa508426aaebb3" Oct 30 09:59:23.601570 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.601509 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq" Oct 30 09:59:23.629041 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.629014 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg" Oct 30 09:59:23.718211 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.718169 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/37d8a054-775f-4a08-baff-90631bbec3be-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37d8a054-775f-4a08-baff-90631bbec3be-istio-podinfo\") pod \"37d8a054-775f-4a08-baff-90631bbec3be\" (UID: \"37d8a054-775f-4a08-baff-90631bbec3be\") " Oct 30 09:59:23.718437 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.718277 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72746dcd-3b5e-46fe-8ec7-eb177b70be60-istio-token\" (UniqueName: \"kubernetes.io/projected/72746dcd-3b5e-46fe-8ec7-eb177b70be60-istio-token\") pod \"72746dcd-3b5e-46fe-8ec7-eb177b70be60\" (UID: \"72746dcd-3b5e-46fe-8ec7-eb177b70be60\") " Oct 30 09:59:23.718437 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.718310 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72746dcd-3b5e-46fe-8ec7-eb177b70be60-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72746dcd-3b5e-46fe-8ec7-eb177b70be60-istiod-ca-cert\") pod \"72746dcd-3b5e-46fe-8ec7-eb177b70be60\" (UID: \"72746dcd-3b5e-46fe-8ec7-eb177b70be60\") " Oct 30 09:59:23.718437 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.718356 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37d8a054-775f-4a08-baff-90631bbec3be-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37d8a054-775f-4a08-baff-90631bbec3be-workload-socket\") pod \"37d8a054-775f-4a08-baff-90631bbec3be\" (UID: \"37d8a054-775f-4a08-baff-90631bbec3be\") " Oct 30 09:59:23.718632 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.718480 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72746dcd-3b5e-46fe-8ec7-eb177b70be60-kube-api-access-l725g\" (UniqueName: \"kubernetes.io/projected/72746dcd-3b5e-46fe-8ec7-eb177b70be60-kube-api-access-l725g\") pod \"72746dcd-3b5e-46fe-8ec7-eb177b70be60\" (UID: \"72746dcd-3b5e-46fe-8ec7-eb177b70be60\") " Oct 30 09:59:23.718632 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.718509 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72746dcd-3b5e-46fe-8ec7-eb177b70be60-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72746dcd-3b5e-46fe-8ec7-eb177b70be60-workload-socket\") pod \"72746dcd-3b5e-46fe-8ec7-eb177b70be60\" (UID: \"72746dcd-3b5e-46fe-8ec7-eb177b70be60\") " Oct 30 09:59:23.718632 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.718543 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37d8a054-775f-4a08-baff-90631bbec3be-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37d8a054-775f-4a08-baff-90631bbec3be-istio-data\") pod \"37d8a054-775f-4a08-baff-90631bbec3be\" (UID: \"37d8a054-775f-4a08-baff-90631bbec3be\") " Oct 30 09:59:23.718632 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.718573 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37d8a054-775f-4a08-baff-90631bbec3be-kube-api-access-bm9cp\" (UniqueName: \"kubernetes.io/projected/37d8a054-775f-4a08-baff-90631bbec3be-kube-api-access-bm9cp\") pod \"37d8a054-775f-4a08-baff-90631bbec3be\" (UID: \"37d8a054-775f-4a08-baff-90631bbec3be\") " Oct 30 09:59:23.718632 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.718605 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37d8a054-775f-4a08-baff-90631bbec3be-istio-token\" (UniqueName: \"kubernetes.io/projected/37d8a054-775f-4a08-baff-90631bbec3be-istio-token\") pod \"37d8a054-775f-4a08-baff-90631bbec3be\" (UID: \"37d8a054-775f-4a08-baff-90631bbec3be\") " Oct 30 09:59:23.719126 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.718641 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72746dcd-3b5e-46fe-8ec7-eb177b70be60-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72746dcd-3b5e-46fe-8ec7-eb177b70be60-credential-socket\") pod \"72746dcd-3b5e-46fe-8ec7-eb177b70be60\" (UID: \"72746dcd-3b5e-46fe-8ec7-eb177b70be60\") " Oct 30 09:59:23.719126 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.718661 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37d8a054-775f-4a08-baff-90631bbec3be-workload-socket" pod "37d8a054-775f-4a08-baff-90631bbec3be" (UID: "37d8a054-775f-4a08-baff-90631bbec3be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:23.719126 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.718670 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37d8a054-775f-4a08-baff-90631bbec3be-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37d8a054-775f-4a08-baff-90631bbec3be-istio-envoy\") pod \"37d8a054-775f-4a08-baff-90631bbec3be\" (UID: \"37d8a054-775f-4a08-baff-90631bbec3be\") " Oct 30 09:59:23.719126 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.718738 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37d8a054-775f-4a08-baff-90631bbec3be-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37d8a054-775f-4a08-baff-90631bbec3be-workload-certs\") pod \"37d8a054-775f-4a08-baff-90631bbec3be\" (UID: \"37d8a054-775f-4a08-baff-90631bbec3be\") " Oct 30 09:59:23.719126 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.718831 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72746dcd-3b5e-46fe-8ec7-eb177b70be60-istiod-ca-cert" pod "72746dcd-3b5e-46fe-8ec7-eb177b70be60" (UID: "72746dcd-3b5e-46fe-8ec7-eb177b70be60"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:59:23.719126 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.718867 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72746dcd-3b5e-46fe-8ec7-eb177b70be60-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72746dcd-3b5e-46fe-8ec7-eb177b70be60-istio-envoy\") pod \"72746dcd-3b5e-46fe-8ec7-eb177b70be60\" (UID: \"72746dcd-3b5e-46fe-8ec7-eb177b70be60\") " Oct 30 09:59:23.719126 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.718917 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72746dcd-3b5e-46fe-8ec7-eb177b70be60-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72746dcd-3b5e-46fe-8ec7-eb177b70be60-istio-data\") pod \"72746dcd-3b5e-46fe-8ec7-eb177b70be60\" (UID: \"72746dcd-3b5e-46fe-8ec7-eb177b70be60\") " Oct 30 09:59:23.719126 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.718930 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37d8a054-775f-4a08-baff-90631bbec3be-istio-data" pod "37d8a054-775f-4a08-baff-90631bbec3be" (UID: "37d8a054-775f-4a08-baff-90631bbec3be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:23.719126 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.718958 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/72746dcd-3b5e-46fe-8ec7-eb177b70be60-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72746dcd-3b5e-46fe-8ec7-eb177b70be60-istio-podinfo\") pod \"72746dcd-3b5e-46fe-8ec7-eb177b70be60\" (UID: \"72746dcd-3b5e-46fe-8ec7-eb177b70be60\") " Oct 30 09:59:23.719126 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.719006 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37d8a054-775f-4a08-baff-90631bbec3be-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37d8a054-775f-4a08-baff-90631bbec3be-istiod-ca-cert\") pod \"37d8a054-775f-4a08-baff-90631bbec3be\" (UID: \"37d8a054-775f-4a08-baff-90631bbec3be\") " Oct 30 09:59:23.719126 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.719058 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37d8a054-775f-4a08-baff-90631bbec3be-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37d8a054-775f-4a08-baff-90631bbec3be-credential-socket\") pod \"37d8a054-775f-4a08-baff-90631bbec3be\" (UID: \"37d8a054-775f-4a08-baff-90631bbec3be\") " Oct 30 09:59:23.719126 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.719089 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72746dcd-3b5e-46fe-8ec7-eb177b70be60-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72746dcd-3b5e-46fe-8ec7-eb177b70be60-workload-certs\") pod \"72746dcd-3b5e-46fe-8ec7-eb177b70be60\" (UID: \"72746dcd-3b5e-46fe-8ec7-eb177b70be60\") " Oct 30 09:59:23.719917 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.719241 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72746dcd-3b5e-46fe-8ec7-eb177b70be60-workload-socket" pod "72746dcd-3b5e-46fe-8ec7-eb177b70be60" (UID: "72746dcd-3b5e-46fe-8ec7-eb177b70be60"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:23.719917 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.719715 4562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72746dcd-3b5e-46fe-8ec7-eb177b70be60-workload-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:59:23.719917 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.719737 4562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37d8a054-775f-4a08-baff-90631bbec3be-istio-data\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:59:23.719917 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.719750 4562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72746dcd-3b5e-46fe-8ec7-eb177b70be60-istiod-ca-cert\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:59:23.719917 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.719762 4562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37d8a054-775f-4a08-baff-90631bbec3be-workload-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:59:23.720208 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.719916 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37d8a054-775f-4a08-baff-90631bbec3be-credential-socket" pod "37d8a054-775f-4a08-baff-90631bbec3be" (UID: "37d8a054-775f-4a08-baff-90631bbec3be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:23.720208 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.719969 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72746dcd-3b5e-46fe-8ec7-eb177b70be60-workload-certs" pod "72746dcd-3b5e-46fe-8ec7-eb177b70be60" (UID: "72746dcd-3b5e-46fe-8ec7-eb177b70be60"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:23.720208 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.720165 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72746dcd-3b5e-46fe-8ec7-eb177b70be60-credential-socket" pod "72746dcd-3b5e-46fe-8ec7-eb177b70be60" (UID: "72746dcd-3b5e-46fe-8ec7-eb177b70be60"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:23.720369 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.720240 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37d8a054-775f-4a08-baff-90631bbec3be-istiod-ca-cert" pod "37d8a054-775f-4a08-baff-90631bbec3be" (UID: "37d8a054-775f-4a08-baff-90631bbec3be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:59:23.720463 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.720446 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37d8a054-775f-4a08-baff-90631bbec3be-workload-certs" pod "37d8a054-775f-4a08-baff-90631bbec3be" (UID: "37d8a054-775f-4a08-baff-90631bbec3be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:23.720564 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.720519 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72746dcd-3b5e-46fe-8ec7-eb177b70be60-istio-data" pod "72746dcd-3b5e-46fe-8ec7-eb177b70be60" (UID: "72746dcd-3b5e-46fe-8ec7-eb177b70be60"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:23.722128 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.722101 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72746dcd-3b5e-46fe-8ec7-eb177b70be60-istio-token" pod "72746dcd-3b5e-46fe-8ec7-eb177b70be60" (UID: "72746dcd-3b5e-46fe-8ec7-eb177b70be60"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:59:23.722499 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.722439 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37d8a054-775f-4a08-baff-90631bbec3be-istio-podinfo" pod "37d8a054-775f-4a08-baff-90631bbec3be" (UID: "37d8a054-775f-4a08-baff-90631bbec3be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:59:23.722499 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.722455 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37d8a054-775f-4a08-baff-90631bbec3be-istio-envoy" pod "37d8a054-775f-4a08-baff-90631bbec3be" (UID: "37d8a054-775f-4a08-baff-90631bbec3be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:23.722871 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.722848 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72746dcd-3b5e-46fe-8ec7-eb177b70be60-istio-envoy" pod "72746dcd-3b5e-46fe-8ec7-eb177b70be60" (UID: "72746dcd-3b5e-46fe-8ec7-eb177b70be60"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 09:59:23.722963 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.722926 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37d8a054-775f-4a08-baff-90631bbec3be-istio-token" pod "37d8a054-775f-4a08-baff-90631bbec3be" (UID: "37d8a054-775f-4a08-baff-90631bbec3be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:59:23.723335 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.723294 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72746dcd-3b5e-46fe-8ec7-eb177b70be60-istio-podinfo" pod "72746dcd-3b5e-46fe-8ec7-eb177b70be60" (UID: "72746dcd-3b5e-46fe-8ec7-eb177b70be60"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 09:59:23.723975 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.723949 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37d8a054-775f-4a08-baff-90631bbec3be-kube-api-access-bm9cp" pod "37d8a054-775f-4a08-baff-90631bbec3be" (UID: "37d8a054-775f-4a08-baff-90631bbec3be"). InnerVolumeSpecName "kube-api-access-bm9cp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:59:23.724537 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.724509 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72746dcd-3b5e-46fe-8ec7-eb177b70be60-kube-api-access-l725g" pod "72746dcd-3b5e-46fe-8ec7-eb177b70be60" (UID: "72746dcd-3b5e-46fe-8ec7-eb177b70be60"). InnerVolumeSpecName "kube-api-access-l725g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:59:23.821308 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.821279 4562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72746dcd-3b5e-46fe-8ec7-eb177b70be60-istio-envoy\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:59:23.821308 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.821306 4562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72746dcd-3b5e-46fe-8ec7-eb177b70be60-istio-data\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:59:23.821507 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.821315 4562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72746dcd-3b5e-46fe-8ec7-eb177b70be60-istio-podinfo\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:59:23.821507 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.821323 4562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37d8a054-775f-4a08-baff-90631bbec3be-istiod-ca-cert\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:59:23.821507 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.821332 4562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37d8a054-775f-4a08-baff-90631bbec3be-credential-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:59:23.821507 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.821341 4562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72746dcd-3b5e-46fe-8ec7-eb177b70be60-workload-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:59:23.821507 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.821360 4562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37d8a054-775f-4a08-baff-90631bbec3be-istio-podinfo\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:59:23.821507 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.821368 4562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72746dcd-3b5e-46fe-8ec7-eb177b70be60-istio-token\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:59:23.821507 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.821377 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l725g\" (UniqueName: \"kubernetes.io/projected/72746dcd-3b5e-46fe-8ec7-eb177b70be60-kube-api-access-l725g\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:59:23.821507 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.821413 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bm9cp\" (UniqueName: \"kubernetes.io/projected/37d8a054-775f-4a08-baff-90631bbec3be-kube-api-access-bm9cp\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:59:23.821507 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.821428 4562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37d8a054-775f-4a08-baff-90631bbec3be-istio-token\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:59:23.821507 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.821443 4562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72746dcd-3b5e-46fe-8ec7-eb177b70be60-credential-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:59:23.821507 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.821455 4562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37d8a054-775f-4a08-baff-90631bbec3be-istio-envoy\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:59:23.821507 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.821463 4562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37d8a054-775f-4a08-baff-90631bbec3be-workload-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:59:23.848223 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.848195 4562 generic.go:356] "Generic (PLEG): container finished" podID="37d8a054-775f-4a08-baff-90631bbec3be" containerID="299d3835c6120870e50dde60384201ac3fe0a834ee5a57d76e0d2dd58e810a0b" exitCode=0 Oct 30 09:59:23.848606 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.848239 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq" Oct 30 09:59:23.848606 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.848279 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq" event={"ID":"37d8a054-775f-4a08-baff-90631bbec3be","Type":"ContainerDied","Data":"299d3835c6120870e50dde60384201ac3fe0a834ee5a57d76e0d2dd58e810a0b"} Oct 30 09:59:23.848606 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.848354 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq" event={"ID":"37d8a054-775f-4a08-baff-90631bbec3be","Type":"ContainerDied","Data":"4c6e560bb3b7fa42a82c43957773baf42f74297c4710dddd7b6a3850ed45637e"} Oct 30 09:59:23.848606 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.848405 4562 scope.go:122] "RemoveContainer" containerID="299d3835c6120870e50dde60384201ac3fe0a834ee5a57d76e0d2dd58e810a0b" Oct 30 09:59:23.849703 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.849679 4562 generic.go:356] "Generic (PLEG): container finished" podID="72746dcd-3b5e-46fe-8ec7-eb177b70be60" containerID="a27353c0140cd23dd26d1ca41ca64d69838bda49128753571f1898e9e671319c" exitCode=0 Oct 30 09:59:23.849772 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.849721 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg" Oct 30 09:59:23.849772 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.849753 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg" event={"ID":"72746dcd-3b5e-46fe-8ec7-eb177b70be60","Type":"ContainerDied","Data":"a27353c0140cd23dd26d1ca41ca64d69838bda49128753571f1898e9e671319c"} Oct 30 09:59:23.849864 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.849806 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg" event={"ID":"72746dcd-3b5e-46fe-8ec7-eb177b70be60","Type":"ContainerDied","Data":"4fda04e728ba066cc8ad6bc0748cf203df3f51d1f8cff17373a10f1ddf0c32b7"} Oct 30 09:59:23.879306 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.879288 4562 scope.go:122] "RemoveContainer" containerID="299d3835c6120870e50dde60384201ac3fe0a834ee5a57d76e0d2dd58e810a0b" Oct 30 09:59:23.879601 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:59:23.879583 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"299d3835c6120870e50dde60384201ac3fe0a834ee5a57d76e0d2dd58e810a0b\": container with ID starting with 299d3835c6120870e50dde60384201ac3fe0a834ee5a57d76e0d2dd58e810a0b not found: ID does not exist" containerID="299d3835c6120870e50dde60384201ac3fe0a834ee5a57d76e0d2dd58e810a0b" Oct 30 09:59:23.879654 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.879610 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"299d3835c6120870e50dde60384201ac3fe0a834ee5a57d76e0d2dd58e810a0b"} err="failed to get container status \"299d3835c6120870e50dde60384201ac3fe0a834ee5a57d76e0d2dd58e810a0b\": rpc error: code = NotFound desc = could not find container \"299d3835c6120870e50dde60384201ac3fe0a834ee5a57d76e0d2dd58e810a0b\": container with ID starting with 299d3835c6120870e50dde60384201ac3fe0a834ee5a57d76e0d2dd58e810a0b not found: ID does not exist" Oct 30 09:59:23.879654 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.879627 4562 scope.go:122] "RemoveContainer" containerID="a27353c0140cd23dd26d1ca41ca64d69838bda49128753571f1898e9e671319c" Oct 30 09:59:23.888567 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.888546 4562 scope.go:122] "RemoveContainer" containerID="a27353c0140cd23dd26d1ca41ca64d69838bda49128753571f1898e9e671319c" Oct 30 09:59:23.888851 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:59:23.888831 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a27353c0140cd23dd26d1ca41ca64d69838bda49128753571f1898e9e671319c\": container with ID starting with a27353c0140cd23dd26d1ca41ca64d69838bda49128753571f1898e9e671319c not found: ID does not exist" containerID="a27353c0140cd23dd26d1ca41ca64d69838bda49128753571f1898e9e671319c" Oct 30 09:59:23.888936 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.888857 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a27353c0140cd23dd26d1ca41ca64d69838bda49128753571f1898e9e671319c"} err="failed to get container status \"a27353c0140cd23dd26d1ca41ca64d69838bda49128753571f1898e9e671319c\": rpc error: code = NotFound desc = could not find container \"a27353c0140cd23dd26d1ca41ca64d69838bda49128753571f1898e9e671319c\": container with ID starting with a27353c0140cd23dd26d1ca41ca64d69838bda49128753571f1898e9e671319c not found: ID does not exist" Oct 30 09:59:23.922255 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.922203 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg"] Oct 30 09:59:23.925125 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.925099 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-hhffg"] Oct 30 09:59:23.962430 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.962341 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq"] Oct 30 09:59:23.967440 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:23.967413 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-82ccfd-openshift-default-854d9c95f-6vwsq"] Oct 30 09:59:24.008068 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:24.008046 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82ccfd-6b4c8757ff-rc85k" Oct 30 09:59:24.124557 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:24.124475 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e7713d8-4bdf-4134-9361-e4c0bca6b525-kube-api-access-8q22z\" (UniqueName: \"kubernetes.io/projected/2e7713d8-4bdf-4134-9361-e4c0bca6b525-kube-api-access-8q22z\") pod \"2e7713d8-4bdf-4134-9361-e4c0bca6b525\" (UID: \"2e7713d8-4bdf-4134-9361-e4c0bca6b525\") " Oct 30 09:59:24.124557 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:24.124547 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e7713d8-4bdf-4134-9361-e4c0bca6b525-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e7713d8-4bdf-4134-9361-e4c0bca6b525-plugins-config-volume\") pod \"2e7713d8-4bdf-4134-9361-e4c0bca6b525\" (UID: \"2e7713d8-4bdf-4134-9361-e4c0bca6b525\") " Oct 30 09:59:24.124903 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:24.124879 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e7713d8-4bdf-4134-9361-e4c0bca6b525-plugins-config-volume" pod "2e7713d8-4bdf-4134-9361-e4c0bca6b525" (UID: "2e7713d8-4bdf-4134-9361-e4c0bca6b525"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:59:24.126661 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:24.126642 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e7713d8-4bdf-4134-9361-e4c0bca6b525-kube-api-access-8q22z" pod "2e7713d8-4bdf-4134-9361-e4c0bca6b525" (UID: "2e7713d8-4bdf-4134-9361-e4c0bca6b525"). InnerVolumeSpecName "kube-api-access-8q22z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:59:24.225806 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:24.225779 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e7713d8-4bdf-4134-9361-e4c0bca6b525-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:59:24.225806 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:24.225804 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8q22z\" (UniqueName: \"kubernetes.io/projected/2e7713d8-4bdf-4134-9361-e4c0bca6b525-kube-api-access-8q22z\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:59:24.858667 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:24.858632 4562 generic.go:356] "Generic (PLEG): container finished" podID="2e7713d8-4bdf-4134-9361-e4c0bca6b525" containerID="a8aa32d2203aa520c798dd93955b1022bf6d685380f68ad131ac93d6a131cc31" exitCode=0 Oct 30 09:59:24.859122 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:24.858670 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82ccfd-6b4c8757ff-rc85k" Oct 30 09:59:24.859122 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:24.858695 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82ccfd-6b4c8757ff-rc85k" event={"ID":"2e7713d8-4bdf-4134-9361-e4c0bca6b525","Type":"ContainerDied","Data":"a8aa32d2203aa520c798dd93955b1022bf6d685380f68ad131ac93d6a131cc31"} Oct 30 09:59:24.859122 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:24.858736 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-82ccfd-6b4c8757ff-rc85k" event={"ID":"2e7713d8-4bdf-4134-9361-e4c0bca6b525","Type":"ContainerDied","Data":"70f203685be4c151ba806356655974f6a237ba927f5b46f70b5f8345e7de18b9"} Oct 30 09:59:24.859122 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:24.858759 4562 scope.go:122] "RemoveContainer" containerID="a8aa32d2203aa520c798dd93955b1022bf6d685380f68ad131ac93d6a131cc31" Oct 30 09:59:24.868450 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:24.868428 4562 scope.go:122] "RemoveContainer" containerID="a8aa32d2203aa520c798dd93955b1022bf6d685380f68ad131ac93d6a131cc31" Oct 30 09:59:24.868716 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:59:24.868697 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8aa32d2203aa520c798dd93955b1022bf6d685380f68ad131ac93d6a131cc31\": container with ID starting with a8aa32d2203aa520c798dd93955b1022bf6d685380f68ad131ac93d6a131cc31 not found: ID does not exist" containerID="a8aa32d2203aa520c798dd93955b1022bf6d685380f68ad131ac93d6a131cc31" Oct 30 09:59:24.868778 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:24.868724 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8aa32d2203aa520c798dd93955b1022bf6d685380f68ad131ac93d6a131cc31"} err="failed to get container status \"a8aa32d2203aa520c798dd93955b1022bf6d685380f68ad131ac93d6a131cc31\": rpc error: code = NotFound desc = could not find container \"a8aa32d2203aa520c798dd93955b1022bf6d685380f68ad131ac93d6a131cc31\": container with ID starting with a8aa32d2203aa520c798dd93955b1022bf6d685380f68ad131ac93d6a131cc31 not found: ID does not exist" Oct 30 09:59:24.911109 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:24.911083 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-82ccfd-6b4c8757ff-rc85k"] Oct 30 09:59:24.923922 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:24.923899 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-82ccfd-6b4c8757ff-rc85k"] Oct 30 09:59:25.873509 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:25.873474 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e7713d8-4bdf-4134-9361-e4c0bca6b525" path="/var/lib/kubelet/pods/2e7713d8-4bdf-4134-9361-e4c0bca6b525/volumes" Oct 30 09:59:25.873955 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:25.873941 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37d8a054-775f-4a08-baff-90631bbec3be" path="/var/lib/kubelet/pods/37d8a054-775f-4a08-baff-90631bbec3be/volumes" Oct 30 09:59:25.874347 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:25.874335 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72746dcd-3b5e-46fe-8ec7-eb177b70be60" path="/var/lib/kubelet/pods/72746dcd-3b5e-46fe-8ec7-eb177b70be60/volumes" Oct 30 09:59:28.595669 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:59:28.595635 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 09:59:29.097545 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:29.097512 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3a61e1-fq8l2"] Oct 30 09:59:29.098030 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:29.098016 4562 cpu_manager.go:414] "containerMap: removing container" podUID="07af07d3-1594-4d42-9b63-af5097dc78b7" containerName="revoke-keys" Oct 30 09:59:29.098088 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:29.098032 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07af07d3-1594-4d42-9b63-af5097dc78b7" containerName="revoke-keys" Oct 30 09:59:29.098088 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:29.098063 4562 cpu_manager.go:414] "containerMap: removing container" podUID="37d8a054-775f-4a08-baff-90631bbec3be" containerName="istio-proxy" Oct 30 09:59:29.098088 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:29.098069 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37d8a054-775f-4a08-baff-90631bbec3be" containerName="istio-proxy" Oct 30 09:59:29.098088 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:29.098079 4562 cpu_manager.go:414] "containerMap: removing container" podUID="72746dcd-3b5e-46fe-8ec7-eb177b70be60" containerName="istio-proxy" Oct 30 09:59:29.098088 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:29.098084 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72746dcd-3b5e-46fe-8ec7-eb177b70be60" containerName="istio-proxy" Oct 30 09:59:29.098239 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:29.098097 4562 cpu_manager.go:414] "containerMap: removing container" podUID="2e7713d8-4bdf-4134-9361-e4c0bca6b525" containerName="payload-pre-processing" Oct 30 09:59:29.098239 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:29.098102 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e7713d8-4bdf-4134-9361-e4c0bca6b525" containerName="payload-pre-processing" Oct 30 09:59:29.098239 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:29.098172 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="37d8a054-775f-4a08-baff-90631bbec3be" containerName="istio-proxy" Oct 30 09:59:29.098239 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:29.098183 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="72746dcd-3b5e-46fe-8ec7-eb177b70be60" containerName="istio-proxy" Oct 30 09:59:29.098239 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:29.098190 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e7713d8-4bdf-4134-9361-e4c0bca6b525" containerName="payload-pre-processing" Oct 30 09:59:29.098239 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:29.098196 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="07af07d3-1594-4d42-9b63-af5097dc78b7" containerName="revoke-keys" Oct 30 09:59:29.103117 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:29.103098 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3a61e1-fq8l2" Oct 30 09:59:29.118886 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:29.118853 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3a61e1-fq8l2"] Oct 30 09:59:29.128946 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:29.128928 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:59:29.129104 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:29.129088 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:59:29.207763 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:29.207734 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s82fz\" (UniqueName: \"kubernetes.io/projected/a11d9bcd-94a9-434c-b2c2-5918ca0d2504-kube-api-access-s82fz\") pod \"maas-api-revoke-keys-e2e-worker-w4-3a61e1-fq8l2\" (UID: \"a11d9bcd-94a9-434c-b2c2-5918ca0d2504\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3a61e1-fq8l2" Oct 30 09:59:29.207920 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:29.207782 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a11d9bcd-94a9-434c-b2c2-5918ca0d2504-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-3a61e1-fq8l2\" (UID: \"a11d9bcd-94a9-434c-b2c2-5918ca0d2504\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3a61e1-fq8l2" Oct 30 09:59:29.207997 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:29.207969 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a11d9bcd-94a9-434c-b2c2-5918ca0d2504-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-3a61e1-fq8l2\" (UID: \"a11d9bcd-94a9-434c-b2c2-5918ca0d2504\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3a61e1-fq8l2" Oct 30 09:59:29.309156 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:29.309121 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a11d9bcd-94a9-434c-b2c2-5918ca0d2504-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-3a61e1-fq8l2\" (UID: \"a11d9bcd-94a9-434c-b2c2-5918ca0d2504\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3a61e1-fq8l2" Oct 30 09:59:29.309324 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:29.309175 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a11d9bcd-94a9-434c-b2c2-5918ca0d2504-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-3a61e1-fq8l2\" (UID: \"a11d9bcd-94a9-434c-b2c2-5918ca0d2504\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3a61e1-fq8l2" Oct 30 09:59:29.309324 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:29.309273 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s82fz\" (UniqueName: \"kubernetes.io/projected/a11d9bcd-94a9-434c-b2c2-5918ca0d2504-kube-api-access-s82fz\") pod \"maas-api-revoke-keys-e2e-worker-w4-3a61e1-fq8l2\" (UID: \"a11d9bcd-94a9-434c-b2c2-5918ca0d2504\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3a61e1-fq8l2" Oct 30 09:59:29.309862 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:29.309836 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a11d9bcd-94a9-434c-b2c2-5918ca0d2504-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-3a61e1-fq8l2\" (UID: \"a11d9bcd-94a9-434c-b2c2-5918ca0d2504\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3a61e1-fq8l2" Oct 30 09:59:29.311678 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:29.311636 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a11d9bcd-94a9-434c-b2c2-5918ca0d2504-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-3a61e1-fq8l2\" (UID: \"a11d9bcd-94a9-434c-b2c2-5918ca0d2504\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3a61e1-fq8l2" Oct 30 09:59:29.332731 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:29.332686 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s82fz\" (UniqueName: \"kubernetes.io/projected/a11d9bcd-94a9-434c-b2c2-5918ca0d2504-kube-api-access-s82fz\") pod \"maas-api-revoke-keys-e2e-worker-w4-3a61e1-fq8l2\" (UID: \"a11d9bcd-94a9-434c-b2c2-5918ca0d2504\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3a61e1-fq8l2" Oct 30 09:59:29.415800 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:29.415704 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3a61e1-fq8l2" Oct 30 09:59:29.714263 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:59:29.714216 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda11d9bcd_94a9_434c_b2c2_5918ca0d2504.slice/crio-0c920577de0694c1e631eab74cab499c3d48b1e8efd30a9660522059cb0a11ed WatchSource:0}: Error finding container 0c920577de0694c1e631eab74cab499c3d48b1e8efd30a9660522059cb0a11ed: Status 404 returned error can't find the container with id 0c920577de0694c1e631eab74cab499c3d48b1e8efd30a9660522059cb0a11ed Oct 30 09:59:29.714814 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:29.714787 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3a61e1-fq8l2"] Oct 30 09:59:29.888889 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:29.888848 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3a61e1-fq8l2" event={"ID":"a11d9bcd-94a9-434c-b2c2-5918ca0d2504","Type":"ContainerStarted","Data":"0c920577de0694c1e631eab74cab499c3d48b1e8efd30a9660522059cb0a11ed"} Oct 30 09:59:30.896482 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:30.896375 4562 generic.go:356] "Generic (PLEG): container finished" podID="a11d9bcd-94a9-434c-b2c2-5918ca0d2504" containerID="f4cf9dcd1cdd722f491635684453cdb3efdb4da51c58bcbad5670f7bea1e38e2" exitCode=0 Oct 30 09:59:30.896845 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:30.896479 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3a61e1-fq8l2" event={"ID":"a11d9bcd-94a9-434c-b2c2-5918ca0d2504","Type":"ContainerDied","Data":"f4cf9dcd1cdd722f491635684453cdb3efdb4da51c58bcbad5670f7bea1e38e2"} Oct 30 09:59:32.032799 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:32.032774 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3a61e1-fq8l2" Oct 30 09:59:32.139699 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:32.139670 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a11d9bcd-94a9-434c-b2c2-5918ca0d2504-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a11d9bcd-94a9-434c-b2c2-5918ca0d2504-maas-api-openshift-ca\") pod \"a11d9bcd-94a9-434c-b2c2-5918ca0d2504\" (UID: \"a11d9bcd-94a9-434c-b2c2-5918ca0d2504\") " Oct 30 09:59:32.139879 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:32.139856 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a11d9bcd-94a9-434c-b2c2-5918ca0d2504-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a11d9bcd-94a9-434c-b2c2-5918ca0d2504-maas-api-xks-ca\") pod \"a11d9bcd-94a9-434c-b2c2-5918ca0d2504\" (UID: \"a11d9bcd-94a9-434c-b2c2-5918ca0d2504\") " Oct 30 09:59:32.139929 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:32.139889 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a11d9bcd-94a9-434c-b2c2-5918ca0d2504-kube-api-access-s82fz\" (UniqueName: \"kubernetes.io/projected/a11d9bcd-94a9-434c-b2c2-5918ca0d2504-kube-api-access-s82fz\") pod \"a11d9bcd-94a9-434c-b2c2-5918ca0d2504\" (UID: \"a11d9bcd-94a9-434c-b2c2-5918ca0d2504\") " Oct 30 09:59:32.139999 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:32.139976 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a11d9bcd-94a9-434c-b2c2-5918ca0d2504-maas-api-openshift-ca" pod "a11d9bcd-94a9-434c-b2c2-5918ca0d2504" (UID: "a11d9bcd-94a9-434c-b2c2-5918ca0d2504"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:59:32.140342 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:32.140325 4562 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a11d9bcd-94a9-434c-b2c2-5918ca0d2504-maas-api-openshift-ca\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:59:32.141991 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:32.141964 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a11d9bcd-94a9-434c-b2c2-5918ca0d2504-maas-api-xks-ca" pod "a11d9bcd-94a9-434c-b2c2-5918ca0d2504" (UID: "a11d9bcd-94a9-434c-b2c2-5918ca0d2504"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:59:32.142131 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:32.142060 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a11d9bcd-94a9-434c-b2c2-5918ca0d2504-kube-api-access-s82fz" pod "a11d9bcd-94a9-434c-b2c2-5918ca0d2504" (UID: "a11d9bcd-94a9-434c-b2c2-5918ca0d2504"). InnerVolumeSpecName "kube-api-access-s82fz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:59:32.241279 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:32.241248 4562 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a11d9bcd-94a9-434c-b2c2-5918ca0d2504-maas-api-xks-ca\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:59:32.241279 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:32.241276 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s82fz\" (UniqueName: \"kubernetes.io/projected/a11d9bcd-94a9-434c-b2c2-5918ca0d2504-kube-api-access-s82fz\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:59:32.908605 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:32.908569 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3a61e1-fq8l2" event={"ID":"a11d9bcd-94a9-434c-b2c2-5918ca0d2504","Type":"ContainerDied","Data":"0c920577de0694c1e631eab74cab499c3d48b1e8efd30a9660522059cb0a11ed"} Oct 30 09:59:32.908605 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:32.908598 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3a61e1-fq8l2" Oct 30 09:59:32.908822 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:32.908616 4562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c920577de0694c1e631eab74cab499c3d48b1e8efd30a9660522059cb0a11ed" Oct 30 09:59:39.561415 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:39.561366 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-3a61e1-6489f9fdfb-lt57n"] Oct 30 09:59:39.595408 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:39.595362 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3a61e1-689c7f9675-6b84c"] Oct 30 09:59:40.487593 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:40.487554 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a479df-bf6sg"] Oct 30 09:59:40.489813 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:40.489787 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a479df-bf6sg"] Oct 30 09:59:41.873129 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:41.873093 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07af07d3-1594-4d42-9b63-af5097dc78b7" path="/var/lib/kubelet/pods/07af07d3-1594-4d42-9b63-af5097dc78b7/volumes" Oct 30 09:59:44.561721 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:44.561668 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-3a61e1-6489f9fdfb-lt57n" podUID="30a13eb2-b7e5-42ff-b5e9-993a20d588aa" containerName="payload-processing" containerID="cri-o://ce60eae137911ea5c9203c6cfc80f5938a12a48479ff47c86ff38f63d6757cc6" gracePeriod=25 Oct 30 09:59:44.596018 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:44.595979 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3a61e1-689c7f9675-6b84c" podUID="d1195ec8-c914-4d63-aca9-85e8b74b5d44" containerName="payload-pre-processing" containerID="cri-o://d9a285e72bb8d4fff6dff6fb999172014528e8505fc6faa6f129a7b80ca37644" gracePeriod=25 Oct 30 09:59:47.817191 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:47.817154 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a525bd-f7v9q"] Oct 30 09:59:47.817679 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:47.817663 4562 cpu_manager.go:414] "containerMap: removing container" podUID="a11d9bcd-94a9-434c-b2c2-5918ca0d2504" containerName="revoke-keys" Oct 30 09:59:47.817726 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:47.817680 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a11d9bcd-94a9-434c-b2c2-5918ca0d2504" containerName="revoke-keys" Oct 30 09:59:47.817766 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:47.817753 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="a11d9bcd-94a9-434c-b2c2-5918ca0d2504" containerName="revoke-keys" Oct 30 09:59:47.821101 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:47.821085 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a525bd-f7v9q" Oct 30 09:59:47.845218 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:47.845185 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a525bd-f7v9q"] Oct 30 09:59:47.854294 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:47.854266 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 09:59:47.854508 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:47.854353 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 09:59:47.911334 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:47.911305 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8rrn\" (UniqueName: \"kubernetes.io/projected/3ec08754-556b-426a-8625-67125a70caab-kube-api-access-h8rrn\") pod \"maas-api-revoke-keys-e2e-extmod-a-a525bd-f7v9q\" (UID: \"3ec08754-556b-426a-8625-67125a70caab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a525bd-f7v9q" Oct 30 09:59:47.911500 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:47.911478 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ec08754-556b-426a-8625-67125a70caab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a525bd-f7v9q\" (UID: \"3ec08754-556b-426a-8625-67125a70caab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a525bd-f7v9q" Oct 30 09:59:47.911622 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:47.911597 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ec08754-556b-426a-8625-67125a70caab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a525bd-f7v9q\" (UID: \"3ec08754-556b-426a-8625-67125a70caab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a525bd-f7v9q" Oct 30 09:59:48.012745 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:48.012707 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ec08754-556b-426a-8625-67125a70caab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a525bd-f7v9q\" (UID: \"3ec08754-556b-426a-8625-67125a70caab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a525bd-f7v9q" Oct 30 09:59:48.012942 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:48.012764 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ec08754-556b-426a-8625-67125a70caab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a525bd-f7v9q\" (UID: \"3ec08754-556b-426a-8625-67125a70caab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a525bd-f7v9q" Oct 30 09:59:48.012942 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:48.012841 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8rrn\" (UniqueName: \"kubernetes.io/projected/3ec08754-556b-426a-8625-67125a70caab-kube-api-access-h8rrn\") pod \"maas-api-revoke-keys-e2e-extmod-a-a525bd-f7v9q\" (UID: \"3ec08754-556b-426a-8625-67125a70caab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a525bd-f7v9q" Oct 30 09:59:48.013640 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:48.013616 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ec08754-556b-426a-8625-67125a70caab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a525bd-f7v9q\" (UID: \"3ec08754-556b-426a-8625-67125a70caab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a525bd-f7v9q" Oct 30 09:59:48.015220 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:48.015177 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ec08754-556b-426a-8625-67125a70caab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a525bd-f7v9q\" (UID: \"3ec08754-556b-426a-8625-67125a70caab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a525bd-f7v9q" Oct 30 09:59:48.032920 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:48.032889 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8rrn\" (UniqueName: \"kubernetes.io/projected/3ec08754-556b-426a-8625-67125a70caab-kube-api-access-h8rrn\") pod \"maas-api-revoke-keys-e2e-extmod-a-a525bd-f7v9q\" (UID: \"3ec08754-556b-426a-8625-67125a70caab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a525bd-f7v9q" Oct 30 09:59:48.134552 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:48.134472 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a525bd-f7v9q" Oct 30 09:59:48.430231 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:59:48.430196 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ec08754_556b_426a_8625_67125a70caab.slice/crio-302bc707b7d29b431540e267bfdf93015d6120bf2bf25dc4c3c509ccb97f3492 WatchSource:0}: Error finding container 302bc707b7d29b431540e267bfdf93015d6120bf2bf25dc4c3c509ccb97f3492: Status 404 returned error can't find the container with id 302bc707b7d29b431540e267bfdf93015d6120bf2bf25dc4c3c509ccb97f3492 Oct 30 09:59:48.430366 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:48.430336 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a525bd-f7v9q"] Oct 30 09:59:49.010566 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:49.010485 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a525bd-f7v9q" event={"ID":"3ec08754-556b-426a-8625-67125a70caab","Type":"ContainerStarted","Data":"8e3ba03660a7f9780a2b28937bcbb970b4571f36428ad2d85701ac8bfd2bd47a"} Oct 30 09:59:49.010566 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:49.010538 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a525bd-f7v9q" event={"ID":"3ec08754-556b-426a-8625-67125a70caab","Type":"ContainerStarted","Data":"302bc707b7d29b431540e267bfdf93015d6120bf2bf25dc4c3c509ccb97f3492"} Oct 30 09:59:49.051683 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:49.051629 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a525bd-f7v9q" podStartSLOduration=2.051616599 podStartE2EDuration="2.051616599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 09:59:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 09:59:49.048923203 +0000 UTC m=+3949.310484206" watchObservedRunningTime="2026-10-30 09:59:49.051616599 +0000 UTC m=+3949.313177593" Oct 30 09:59:49.571898 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:49.571861 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2459ab93-vwz95"] Oct 30 09:59:49.577020 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:49.576991 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2459ab93-vwz95" Oct 30 09:59:49.596573 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:49.596538 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2459ab93-vwz95"] Oct 30 09:59:49.632336 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:49.632264 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efa60339-43cc-408f-8b23-1160c89f9477-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2459ab93-vwz95\" (UID: \"efa60339-43cc-408f-8b23-1160c89f9477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2459ab93-vwz95" Oct 30 09:59:49.632497 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:49.632409 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mhh8\" (UniqueName: \"kubernetes.io/projected/efa60339-43cc-408f-8b23-1160c89f9477-kube-api-access-4mhh8\") pod \"maas-api-revoke-sub-2459ab93-vwz95\" (UID: \"efa60339-43cc-408f-8b23-1160c89f9477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2459ab93-vwz95" Oct 30 09:59:49.632577 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:49.632552 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efa60339-43cc-408f-8b23-1160c89f9477-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2459ab93-vwz95\" (UID: \"efa60339-43cc-408f-8b23-1160c89f9477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2459ab93-vwz95" Oct 30 09:59:49.733954 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:49.733920 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mhh8\" (UniqueName: \"kubernetes.io/projected/efa60339-43cc-408f-8b23-1160c89f9477-kube-api-access-4mhh8\") pod \"maas-api-revoke-sub-2459ab93-vwz95\" (UID: \"efa60339-43cc-408f-8b23-1160c89f9477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2459ab93-vwz95" Oct 30 09:59:49.734120 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:49.734010 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efa60339-43cc-408f-8b23-1160c89f9477-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2459ab93-vwz95\" (UID: \"efa60339-43cc-408f-8b23-1160c89f9477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2459ab93-vwz95" Oct 30 09:59:49.734120 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:49.734035 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efa60339-43cc-408f-8b23-1160c89f9477-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2459ab93-vwz95\" (UID: \"efa60339-43cc-408f-8b23-1160c89f9477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2459ab93-vwz95" Oct 30 09:59:49.734680 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:49.734659 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efa60339-43cc-408f-8b23-1160c89f9477-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2459ab93-vwz95\" (UID: \"efa60339-43cc-408f-8b23-1160c89f9477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2459ab93-vwz95" Oct 30 09:59:49.736338 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:49.736315 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efa60339-43cc-408f-8b23-1160c89f9477-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2459ab93-vwz95\" (UID: \"efa60339-43cc-408f-8b23-1160c89f9477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2459ab93-vwz95" Oct 30 09:59:49.752915 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:49.752896 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mhh8\" (UniqueName: \"kubernetes.io/projected/efa60339-43cc-408f-8b23-1160c89f9477-kube-api-access-4mhh8\") pod \"maas-api-revoke-sub-2459ab93-vwz95\" (UID: \"efa60339-43cc-408f-8b23-1160c89f9477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2459ab93-vwz95" Oct 30 09:59:49.891546 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:49.891474 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2459ab93-vwz95" Oct 30 09:59:50.017523 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:50.017487 4562 generic.go:356] "Generic (PLEG): container finished" podID="3ec08754-556b-426a-8625-67125a70caab" containerID="8e3ba03660a7f9780a2b28937bcbb970b4571f36428ad2d85701ac8bfd2bd47a" exitCode=0 Oct 30 09:59:50.017950 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:50.017549 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a525bd-f7v9q" event={"ID":"3ec08754-556b-426a-8625-67125a70caab","Type":"ContainerDied","Data":"8e3ba03660a7f9780a2b28937bcbb970b4571f36428ad2d85701ac8bfd2bd47a"} Oct 30 09:59:50.199809 ip-10-0-143-244 kubenswrapper[4562]: W1030 09:59:50.199780 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefa60339_43cc_408f_8b23_1160c89f9477.slice/crio-9120722434ac403aac913df9e6edc3efd17af8761fffb16c9d8c083f7f9d2314 WatchSource:0}: Error finding container 9120722434ac403aac913df9e6edc3efd17af8761fffb16c9d8c083f7f9d2314: Status 404 returned error can't find the container with id 9120722434ac403aac913df9e6edc3efd17af8761fffb16c9d8c083f7f9d2314 Oct 30 09:59:50.200283 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:50.200258 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2459ab93-vwz95"] Oct 30 09:59:51.025638 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:51.025609 4562 generic.go:356] "Generic (PLEG): container finished" podID="efa60339-43cc-408f-8b23-1160c89f9477" containerID="176660bdd0c2c890441ef9d7e4f8adf2ee690130418d36318c6ad245247b6b17" exitCode=0 Oct 30 09:59:51.025980 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:51.025674 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2459ab93-vwz95" event={"ID":"efa60339-43cc-408f-8b23-1160c89f9477","Type":"ContainerDied","Data":"176660bdd0c2c890441ef9d7e4f8adf2ee690130418d36318c6ad245247b6b17"} Oct 30 09:59:51.025980 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:51.025740 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2459ab93-vwz95" event={"ID":"efa60339-43cc-408f-8b23-1160c89f9477","Type":"ContainerStarted","Data":"9120722434ac403aac913df9e6edc3efd17af8761fffb16c9d8c083f7f9d2314"} Oct 30 09:59:51.175641 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:51.175609 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a525bd-f7v9q" Oct 30 09:59:51.249212 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:51.249127 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ec08754-556b-426a-8625-67125a70caab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ec08754-556b-426a-8625-67125a70caab-maas-api-xks-ca\") pod \"3ec08754-556b-426a-8625-67125a70caab\" (UID: \"3ec08754-556b-426a-8625-67125a70caab\") " Oct 30 09:59:51.249212 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:51.249193 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ec08754-556b-426a-8625-67125a70caab-kube-api-access-h8rrn\" (UniqueName: \"kubernetes.io/projected/3ec08754-556b-426a-8625-67125a70caab-kube-api-access-h8rrn\") pod \"3ec08754-556b-426a-8625-67125a70caab\" (UID: \"3ec08754-556b-426a-8625-67125a70caab\") " Oct 30 09:59:51.249547 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:51.249305 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ec08754-556b-426a-8625-67125a70caab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ec08754-556b-426a-8625-67125a70caab-maas-api-openshift-ca\") pod \"3ec08754-556b-426a-8625-67125a70caab\" (UID: \"3ec08754-556b-426a-8625-67125a70caab\") " Oct 30 09:59:51.249833 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:51.249801 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ec08754-556b-426a-8625-67125a70caab-maas-api-openshift-ca" pod "3ec08754-556b-426a-8625-67125a70caab" (UID: "3ec08754-556b-426a-8625-67125a70caab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:59:51.251407 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:51.251356 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ec08754-556b-426a-8625-67125a70caab-maas-api-xks-ca" pod "3ec08754-556b-426a-8625-67125a70caab" (UID: "3ec08754-556b-426a-8625-67125a70caab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:59:51.251549 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:51.251531 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ec08754-556b-426a-8625-67125a70caab-kube-api-access-h8rrn" pod "3ec08754-556b-426a-8625-67125a70caab" (UID: "3ec08754-556b-426a-8625-67125a70caab"). InnerVolumeSpecName "kube-api-access-h8rrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:59:51.350650 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:51.350616 4562 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ec08754-556b-426a-8625-67125a70caab-maas-api-xks-ca\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:59:51.350650 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:51.350643 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8rrn\" (UniqueName: \"kubernetes.io/projected/3ec08754-556b-426a-8625-67125a70caab-kube-api-access-h8rrn\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:59:51.350650 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:51.350652 4562 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ec08754-556b-426a-8625-67125a70caab-maas-api-openshift-ca\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:59:52.033139 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:52.033107 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a525bd-f7v9q" Oct 30 09:59:52.033574 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:52.033138 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a525bd-f7v9q" event={"ID":"3ec08754-556b-426a-8625-67125a70caab","Type":"ContainerDied","Data":"302bc707b7d29b431540e267bfdf93015d6120bf2bf25dc4c3c509ccb97f3492"} Oct 30 09:59:52.033574 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:52.033192 4562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="302bc707b7d29b431540e267bfdf93015d6120bf2bf25dc4c3c509ccb97f3492" Oct 30 09:59:52.192072 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:52.192043 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2459ab93-vwz95" Oct 30 09:59:52.268545 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:52.268513 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efa60339-43cc-408f-8b23-1160c89f9477-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efa60339-43cc-408f-8b23-1160c89f9477-maas-api-xks-ca\") pod \"efa60339-43cc-408f-8b23-1160c89f9477\" (UID: \"efa60339-43cc-408f-8b23-1160c89f9477\") " Oct 30 09:59:52.268731 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:52.268631 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efa60339-43cc-408f-8b23-1160c89f9477-kube-api-access-4mhh8\" (UniqueName: \"kubernetes.io/projected/efa60339-43cc-408f-8b23-1160c89f9477-kube-api-access-4mhh8\") pod \"efa60339-43cc-408f-8b23-1160c89f9477\" (UID: \"efa60339-43cc-408f-8b23-1160c89f9477\") " Oct 30 09:59:52.268731 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:52.268719 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efa60339-43cc-408f-8b23-1160c89f9477-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efa60339-43cc-408f-8b23-1160c89f9477-maas-api-openshift-ca\") pod \"efa60339-43cc-408f-8b23-1160c89f9477\" (UID: \"efa60339-43cc-408f-8b23-1160c89f9477\") " Oct 30 09:59:52.269096 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:52.269073 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efa60339-43cc-408f-8b23-1160c89f9477-maas-api-openshift-ca" pod "efa60339-43cc-408f-8b23-1160c89f9477" (UID: "efa60339-43cc-408f-8b23-1160c89f9477"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 09:59:52.270947 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:52.270921 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efa60339-43cc-408f-8b23-1160c89f9477-kube-api-access-4mhh8" pod "efa60339-43cc-408f-8b23-1160c89f9477" (UID: "efa60339-43cc-408f-8b23-1160c89f9477"). InnerVolumeSpecName "kube-api-access-4mhh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 09:59:52.271182 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:52.271157 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efa60339-43cc-408f-8b23-1160c89f9477-maas-api-xks-ca" pod "efa60339-43cc-408f-8b23-1160c89f9477" (UID: "efa60339-43cc-408f-8b23-1160c89f9477"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 09:59:52.315553 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:52.315519 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-3a61e1-6489f9fdfb-lt57n" podUID="30a13eb2-b7e5-42ff-b5e9-993a20d588aa" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:59:52.315672 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:52.315645 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3a61e1-689c7f9675-6b84c" podUID="d1195ec8-c914-4d63-aca9-85e8b74b5d44" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 09:59:52.369878 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:52.369847 4562 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efa60339-43cc-408f-8b23-1160c89f9477-maas-api-openshift-ca\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:59:52.369878 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:52.369876 4562 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efa60339-43cc-408f-8b23-1160c89f9477-maas-api-xks-ca\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:59:52.370052 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:52.369885 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mhh8\" (UniqueName: \"kubernetes.io/projected/efa60339-43cc-408f-8b23-1160c89f9477-kube-api-access-4mhh8\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 09:59:53.041883 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:53.041847 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2459ab93-vwz95" Oct 30 09:59:53.042303 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:53.041887 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2459ab93-vwz95" event={"ID":"efa60339-43cc-408f-8b23-1160c89f9477","Type":"ContainerDied","Data":"9120722434ac403aac913df9e6edc3efd17af8761fffb16c9d8c083f7f9d2314"} Oct 30 09:59:53.042303 ip-10-0-143-244 kubenswrapper[4562]: I1030 09:59:53.041938 4562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9120722434ac403aac913df9e6edc3efd17af8761fffb16c9d8c083f7f9d2314" Oct 30 09:59:58.596117 ip-10-0-143-244 kubenswrapper[4562]: E1030 09:59:58.596082 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 10:00:00.140595 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:00.140560 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889240-bc26j"] Oct 30 10:00:00.141240 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:00.141217 4562 cpu_manager.go:414] "containerMap: removing container" podUID="3ec08754-556b-426a-8625-67125a70caab" containerName="revoke-keys" Oct 30 10:00:00.141359 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:00.141242 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ec08754-556b-426a-8625-67125a70caab" containerName="revoke-keys" Oct 30 10:00:00.141359 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:00.141300 4562 cpu_manager.go:414] "containerMap: removing container" podUID="efa60339-43cc-408f-8b23-1160c89f9477" containerName="revoke-keys" Oct 30 10:00:00.141359 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:00.141309 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efa60339-43cc-408f-8b23-1160c89f9477" containerName="revoke-keys" Oct 30 10:00:00.141566 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:00.141455 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="efa60339-43cc-408f-8b23-1160c89f9477" containerName="revoke-keys" Oct 30 10:00:00.141566 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:00.141474 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ec08754-556b-426a-8625-67125a70caab" containerName="revoke-keys" Oct 30 10:00:00.145037 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:00.145014 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889240-gvt2h"] Oct 30 10:00:00.145155 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:00.145138 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889240-bc26j" Oct 30 10:00:00.148549 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:00.148534 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889240-gvt2h" Oct 30 10:00:00.161344 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:00.161307 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a525bd-29889240-l2v6p"] Oct 30 10:00:00.166852 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:00.166833 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a525bd-29889240-l2v6p" Oct 30 10:00:00.167092 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:00.167070 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889240-bc26j"] Oct 30 10:00:00.171991 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:00.171968 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 10:00:00.179952 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:00.179172 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889240-gvt2h"] Oct 30 10:00:00.191241 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:00.190614 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a525bd-29889240-l2v6p"] Oct 30 10:00:00.258029 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:00.258001 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03ef2570-c3b8-48c9-9632-de3c09ea41dc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a525bd-29889240-l2v6p\" (UID: \"03ef2570-c3b8-48c9-9632-de3c09ea41dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a525bd-29889240-l2v6p" Oct 30 10:00:00.258166 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:00.258066 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2w9r\" (UniqueName: \"kubernetes.io/projected/e2232a56-3384-41a6-961d-ec5b4ce0db03-kube-api-access-b2w9r\") pod \"maas-api-key-cleanup-29889240-bc26j\" (UID: \"e2232a56-3384-41a6-961d-ec5b4ce0db03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889240-bc26j" Oct 30 10:00:00.258225 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:00.258163 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc6684d6-8c9b-4a9d-b2bc-9f825424b546-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889240-gvt2h\" (UID: \"cc6684d6-8c9b-4a9d-b2bc-9f825424b546\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889240-gvt2h" Oct 30 10:00:00.258225 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:00.258196 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chxx7\" (UniqueName: \"kubernetes.io/projected/03ef2570-c3b8-48c9-9632-de3c09ea41dc-kube-api-access-chxx7\") pod \"maas-api-key-cleanup-e2e-extmod-a-a525bd-29889240-l2v6p\" (UID: \"03ef2570-c3b8-48c9-9632-de3c09ea41dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a525bd-29889240-l2v6p" Oct 30 10:00:00.258306 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:00.258251 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2232a56-3384-41a6-961d-ec5b4ce0db03-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889240-bc26j\" (UID: \"e2232a56-3384-41a6-961d-ec5b4ce0db03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889240-bc26j" Oct 30 10:00:00.258306 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:00.258286 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rthrp\" (UniqueName: \"kubernetes.io/projected/cc6684d6-8c9b-4a9d-b2bc-9f825424b546-kube-api-access-rthrp\") pod \"maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889240-gvt2h\" (UID: \"cc6684d6-8c9b-4a9d-b2bc-9f825424b546\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889240-gvt2h" Oct 30 10:00:00.358895 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:00.358848 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2w9r\" (UniqueName: \"kubernetes.io/projected/e2232a56-3384-41a6-961d-ec5b4ce0db03-kube-api-access-b2w9r\") pod \"maas-api-key-cleanup-29889240-bc26j\" (UID: \"e2232a56-3384-41a6-961d-ec5b4ce0db03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889240-bc26j" Oct 30 10:00:00.359080 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:00.358934 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc6684d6-8c9b-4a9d-b2bc-9f825424b546-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889240-gvt2h\" (UID: \"cc6684d6-8c9b-4a9d-b2bc-9f825424b546\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889240-gvt2h" Oct 30 10:00:00.359080 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:00.358978 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chxx7\" (UniqueName: \"kubernetes.io/projected/03ef2570-c3b8-48c9-9632-de3c09ea41dc-kube-api-access-chxx7\") pod \"maas-api-key-cleanup-e2e-extmod-a-a525bd-29889240-l2v6p\" (UID: \"03ef2570-c3b8-48c9-9632-de3c09ea41dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a525bd-29889240-l2v6p" Oct 30 10:00:00.359080 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:00.359030 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2232a56-3384-41a6-961d-ec5b4ce0db03-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889240-bc26j\" (UID: \"e2232a56-3384-41a6-961d-ec5b4ce0db03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889240-bc26j" Oct 30 10:00:00.359080 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:00.359058 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rthrp\" (UniqueName: \"kubernetes.io/projected/cc6684d6-8c9b-4a9d-b2bc-9f825424b546-kube-api-access-rthrp\") pod \"maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889240-gvt2h\" (UID: \"cc6684d6-8c9b-4a9d-b2bc-9f825424b546\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889240-gvt2h" Oct 30 10:00:00.359313 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:00.359115 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03ef2570-c3b8-48c9-9632-de3c09ea41dc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a525bd-29889240-l2v6p\" (UID: \"03ef2570-c3b8-48c9-9632-de3c09ea41dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a525bd-29889240-l2v6p" Oct 30 10:00:00.359692 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:00.359666 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc6684d6-8c9b-4a9d-b2bc-9f825424b546-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889240-gvt2h\" (UID: \"cc6684d6-8c9b-4a9d-b2bc-9f825424b546\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889240-gvt2h" Oct 30 10:00:00.359793 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:00.359717 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2232a56-3384-41a6-961d-ec5b4ce0db03-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889240-bc26j\" (UID: \"e2232a56-3384-41a6-961d-ec5b4ce0db03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889240-bc26j" Oct 30 10:00:00.359865 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:00.359817 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03ef2570-c3b8-48c9-9632-de3c09ea41dc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a525bd-29889240-l2v6p\" (UID: \"03ef2570-c3b8-48c9-9632-de3c09ea41dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a525bd-29889240-l2v6p" Oct 30 10:00:00.378438 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:00.378416 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rthrp\" (UniqueName: \"kubernetes.io/projected/cc6684d6-8c9b-4a9d-b2bc-9f825424b546-kube-api-access-rthrp\") pod \"maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889240-gvt2h\" (UID: \"cc6684d6-8c9b-4a9d-b2bc-9f825424b546\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889240-gvt2h" Oct 30 10:00:00.378531 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:00.378421 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chxx7\" (UniqueName: \"kubernetes.io/projected/03ef2570-c3b8-48c9-9632-de3c09ea41dc-kube-api-access-chxx7\") pod \"maas-api-key-cleanup-e2e-extmod-a-a525bd-29889240-l2v6p\" (UID: \"03ef2570-c3b8-48c9-9632-de3c09ea41dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a525bd-29889240-l2v6p" Oct 30 10:00:00.378531 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:00.378422 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2w9r\" (UniqueName: \"kubernetes.io/projected/e2232a56-3384-41a6-961d-ec5b4ce0db03-kube-api-access-b2w9r\") pod \"maas-api-key-cleanup-29889240-bc26j\" (UID: \"e2232a56-3384-41a6-961d-ec5b4ce0db03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889240-bc26j" Oct 30 10:00:00.459493 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:00.459462 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889240-bc26j" Oct 30 10:00:00.467688 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:00.467660 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889240-gvt2h" Oct 30 10:00:00.482713 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:00.482683 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a525bd-29889240-l2v6p" Oct 30 10:00:00.768168 ip-10-0-143-244 kubenswrapper[4562]: W1030 10:00:00.768139 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2232a56_3384_41a6_961d_ec5b4ce0db03.slice/crio-601ce9d1d2cbd0ad1e30b8100a2f8910d4f35d40dad64862e028e4b7d564d19b WatchSource:0}: Error finding container 601ce9d1d2cbd0ad1e30b8100a2f8910d4f35d40dad64862e028e4b7d564d19b: Status 404 returned error can't find the container with id 601ce9d1d2cbd0ad1e30b8100a2f8910d4f35d40dad64862e028e4b7d564d19b Oct 30 10:00:00.768677 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:00.768654 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889240-bc26j"] Oct 30 10:00:00.778526 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:00.778499 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889240-gvt2h"] Oct 30 10:00:00.778768 ip-10-0-143-244 kubenswrapper[4562]: W1030 10:00:00.778720 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc6684d6_8c9b_4a9d_b2bc_9f825424b546.slice/crio-44a0dd60ed828f29eda920aef59e659a504d4ad898d21c5964ce0a40f98e62a9 WatchSource:0}: Error finding container 44a0dd60ed828f29eda920aef59e659a504d4ad898d21c5964ce0a40f98e62a9: Status 404 returned error can't find the container with id 44a0dd60ed828f29eda920aef59e659a504d4ad898d21c5964ce0a40f98e62a9 Oct 30 10:00:00.809109 ip-10-0-143-244 kubenswrapper[4562]: W1030 10:00:00.809084 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03ef2570_c3b8_48c9_9632_de3c09ea41dc.slice/crio-2963af955314dcc0afe418c20521176d98757f7de983c3957649b0c891c496ad WatchSource:0}: Error finding container 2963af955314dcc0afe418c20521176d98757f7de983c3957649b0c891c496ad: Status 404 returned error can't find the container with id 2963af955314dcc0afe418c20521176d98757f7de983c3957649b0c891c496ad Oct 30 10:00:00.809714 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:00.809690 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a525bd-29889240-l2v6p"] Oct 30 10:00:01.104370 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:01.104340 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889240-gvt2h" event={"ID":"cc6684d6-8c9b-4a9d-b2bc-9f825424b546","Type":"ContainerStarted","Data":"79eeab0e6205b8c782036c85244c129c1320f8383aa6696758e23b8b4bf16bc6"} Oct 30 10:00:01.104520 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:01.104430 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889240-gvt2h" event={"ID":"cc6684d6-8c9b-4a9d-b2bc-9f825424b546","Type":"ContainerStarted","Data":"44a0dd60ed828f29eda920aef59e659a504d4ad898d21c5964ce0a40f98e62a9"} Oct 30 10:00:01.105958 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:01.105929 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a525bd-29889240-l2v6p" event={"ID":"03ef2570-c3b8-48c9-9632-de3c09ea41dc","Type":"ContainerStarted","Data":"2eadd35450f7fd405bd1d6830738755092f6cc07ba2c0cf401456d8b932ce7d4"} Oct 30 10:00:01.106077 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:01.105970 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a525bd-29889240-l2v6p" event={"ID":"03ef2570-c3b8-48c9-9632-de3c09ea41dc","Type":"ContainerStarted","Data":"2963af955314dcc0afe418c20521176d98757f7de983c3957649b0c891c496ad"} Oct 30 10:00:01.107157 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:01.107132 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889240-bc26j" event={"ID":"e2232a56-3384-41a6-961d-ec5b4ce0db03","Type":"ContainerStarted","Data":"601ce9d1d2cbd0ad1e30b8100a2f8910d4f35d40dad64862e028e4b7d564d19b"} Oct 30 10:00:01.166454 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:01.166411 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889240-gvt2h" podStartSLOduration=1.166397533 podStartE2EDuration="1.166397533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 10:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:00:01.164723422 +0000 UTC m=+3961.426284410" watchObservedRunningTime="2026-10-30 10:00:01.166397533 +0000 UTC m=+3961.427958579" Oct 30 10:00:01.205937 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:01.205892 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a525bd-29889240-l2v6p" podStartSLOduration=1.205879703 podStartE2EDuration="1.205879703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 10:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:00:01.205108473 +0000 UTC m=+3961.466669459" watchObservedRunningTime="2026-10-30 10:00:01.205879703 +0000 UTC m=+3961.467440732" Oct 30 10:00:02.116396 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:02.116340 4562 generic.go:356] "Generic (PLEG): container finished" podID="e2232a56-3384-41a6-961d-ec5b4ce0db03" containerID="82d7de15d25a18eb9a395b1ce3bf292dcadab768b35fd47770d7f0cc160190ed" exitCode=60 Oct 30 10:00:02.116604 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:02.116412 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889240-bc26j" event={"ID":"e2232a56-3384-41a6-961d-ec5b4ce0db03","Type":"ContainerDied","Data":"82d7de15d25a18eb9a395b1ce3bf292dcadab768b35fd47770d7f0cc160190ed"} Oct 30 10:00:02.116757 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:02.116740 4562 scope.go:122] "RemoveContainer" containerID="82d7de15d25a18eb9a395b1ce3bf292dcadab768b35fd47770d7f0cc160190ed" Oct 30 10:00:02.118319 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:02.118281 4562 generic.go:356] "Generic (PLEG): container finished" podID="cc6684d6-8c9b-4a9d-b2bc-9f825424b546" containerID="79eeab0e6205b8c782036c85244c129c1320f8383aa6696758e23b8b4bf16bc6" exitCode=60 Oct 30 10:00:02.118440 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:02.118333 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889240-gvt2h" event={"ID":"cc6684d6-8c9b-4a9d-b2bc-9f825424b546","Type":"ContainerDied","Data":"79eeab0e6205b8c782036c85244c129c1320f8383aa6696758e23b8b4bf16bc6"} Oct 30 10:00:02.118673 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:02.118659 4562 scope.go:122] "RemoveContainer" containerID="79eeab0e6205b8c782036c85244c129c1320f8383aa6696758e23b8b4bf16bc6" Oct 30 10:00:02.120328 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:02.120271 4562 generic.go:356] "Generic (PLEG): container finished" podID="03ef2570-c3b8-48c9-9632-de3c09ea41dc" containerID="2eadd35450f7fd405bd1d6830738755092f6cc07ba2c0cf401456d8b932ce7d4" exitCode=60 Oct 30 10:00:02.120416 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:02.120325 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a525bd-29889240-l2v6p" event={"ID":"03ef2570-c3b8-48c9-9632-de3c09ea41dc","Type":"ContainerDied","Data":"2eadd35450f7fd405bd1d6830738755092f6cc07ba2c0cf401456d8b932ce7d4"} Oct 30 10:00:02.120608 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:02.120548 4562 scope.go:122] "RemoveContainer" containerID="2eadd35450f7fd405bd1d6830738755092f6cc07ba2c0cf401456d8b932ce7d4" Oct 30 10:00:02.315564 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:02.315528 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-3a61e1-6489f9fdfb-lt57n" podUID="30a13eb2-b7e5-42ff-b5e9-993a20d588aa" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 10:00:02.315941 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:02.315529 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3a61e1-689c7f9675-6b84c" podUID="d1195ec8-c914-4d63-aca9-85e8b74b5d44" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 10:00:03.128328 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:03.128295 4562 generic.go:356] "Generic (PLEG): container finished" podID="03ef2570-c3b8-48c9-9632-de3c09ea41dc" containerID="339f0200ef68e6fc98d5b7f4c0ca5ef1bf6646ebdd1d64acca9836629dd4e48c" exitCode=60 Oct 30 10:00:03.128568 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:03.128372 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a525bd-29889240-l2v6p" event={"ID":"03ef2570-c3b8-48c9-9632-de3c09ea41dc","Type":"ContainerDied","Data":"339f0200ef68e6fc98d5b7f4c0ca5ef1bf6646ebdd1d64acca9836629dd4e48c"} Oct 30 10:00:03.128568 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:03.128440 4562 scope.go:122] "RemoveContainer" containerID="2eadd35450f7fd405bd1d6830738755092f6cc07ba2c0cf401456d8b932ce7d4" Oct 30 10:00:03.128698 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:03.128672 4562 scope.go:122] "RemoveContainer" containerID="339f0200ef68e6fc98d5b7f4c0ca5ef1bf6646ebdd1d64acca9836629dd4e48c" Oct 30 10:00:03.128994 ip-10-0-143-244 kubenswrapper[4562]: E1030 10:00:03.128968 4562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a525bd-29889240-l2v6p_odh-ai-gateway-infra(03ef2570-c3b8-48c9-9632-de3c09ea41dc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a525bd-29889240-l2v6p" podUID="03ef2570-c3b8-48c9-9632-de3c09ea41dc" Oct 30 10:00:03.130617 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:03.130594 4562 generic.go:356] "Generic (PLEG): container finished" podID="e2232a56-3384-41a6-961d-ec5b4ce0db03" containerID="7f03d7fb6b4c3ca48349c33693d60095caada0f46ca705b96b35995da3cc1f67" exitCode=60 Oct 30 10:00:03.130726 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:03.130676 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889240-bc26j" event={"ID":"e2232a56-3384-41a6-961d-ec5b4ce0db03","Type":"ContainerDied","Data":"7f03d7fb6b4c3ca48349c33693d60095caada0f46ca705b96b35995da3cc1f67"} Oct 30 10:00:03.130983 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:03.130966 4562 scope.go:122] "RemoveContainer" containerID="7f03d7fb6b4c3ca48349c33693d60095caada0f46ca705b96b35995da3cc1f67" Oct 30 10:00:03.131227 ip-10-0-143-244 kubenswrapper[4562]: E1030 10:00:03.131210 4562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889240-bc26j_odh-ai-gateway-infra(e2232a56-3384-41a6-961d-ec5b4ce0db03)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889240-bc26j" podUID="e2232a56-3384-41a6-961d-ec5b4ce0db03" Oct 30 10:00:03.132630 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:03.132608 4562 generic.go:356] "Generic (PLEG): container finished" podID="cc6684d6-8c9b-4a9d-b2bc-9f825424b546" containerID="bdd24801e20c9bf871653556c843eb5cb6c7bdbaa3430e469888a0224af7741d" exitCode=60 Oct 30 10:00:03.132730 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:03.132688 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889240-gvt2h" event={"ID":"cc6684d6-8c9b-4a9d-b2bc-9f825424b546","Type":"ContainerDied","Data":"bdd24801e20c9bf871653556c843eb5cb6c7bdbaa3430e469888a0224af7741d"} Oct 30 10:00:03.132871 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:03.132858 4562 scope.go:122] "RemoveContainer" containerID="bdd24801e20c9bf871653556c843eb5cb6c7bdbaa3430e469888a0224af7741d" Oct 30 10:00:03.133080 ip-10-0-143-244 kubenswrapper[4562]: E1030 10:00:03.133064 4562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889240-gvt2h_odh-ai-gateway-infra(cc6684d6-8c9b-4a9d-b2bc-9f825424b546)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889240-gvt2h" podUID="cc6684d6-8c9b-4a9d-b2bc-9f825424b546" Oct 30 10:00:03.169248 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:03.166016 4562 scope.go:122] "RemoveContainer" containerID="82d7de15d25a18eb9a395b1ce3bf292dcadab768b35fd47770d7f0cc160190ed" Oct 30 10:00:03.181661 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:03.181634 4562 scope.go:122] "RemoveContainer" containerID="79eeab0e6205b8c782036c85244c129c1320f8383aa6696758e23b8b4bf16bc6" Oct 30 10:00:03.645245 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:03.645203 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889240-gvt2h"] Oct 30 10:00:04.140585 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:04.140557 4562 scope.go:122] "RemoveContainer" containerID="bdd24801e20c9bf871653556c843eb5cb6c7bdbaa3430e469888a0224af7741d" Oct 30 10:00:04.140864 ip-10-0-143-244 kubenswrapper[4562]: E1030 10:00:04.140842 4562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889240-gvt2h_odh-ai-gateway-infra(cc6684d6-8c9b-4a9d-b2bc-9f825424b546)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889240-gvt2h" podUID="cc6684d6-8c9b-4a9d-b2bc-9f825424b546" Oct 30 10:00:04.142308 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:04.142287 4562 scope.go:122] "RemoveContainer" containerID="339f0200ef68e6fc98d5b7f4c0ca5ef1bf6646ebdd1d64acca9836629dd4e48c" Oct 30 10:00:04.142592 ip-10-0-143-244 kubenswrapper[4562]: E1030 10:00:04.142563 4562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a525bd-29889240-l2v6p_odh-ai-gateway-infra(03ef2570-c3b8-48c9-9632-de3c09ea41dc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a525bd-29889240-l2v6p" podUID="03ef2570-c3b8-48c9-9632-de3c09ea41dc" Oct 30 10:00:04.143931 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:04.143913 4562 scope.go:122] "RemoveContainer" containerID="7f03d7fb6b4c3ca48349c33693d60095caada0f46ca705b96b35995da3cc1f67" Oct 30 10:00:04.144133 ip-10-0-143-244 kubenswrapper[4562]: E1030 10:00:04.144114 4562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889240-bc26j_odh-ai-gateway-infra(e2232a56-3384-41a6-961d-ec5b4ce0db03)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889240-bc26j" podUID="e2232a56-3384-41a6-961d-ec5b4ce0db03" Oct 30 10:00:04.225156 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:04.225116 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3a61e1-fq8l2"] Oct 30 10:00:04.235346 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:04.235312 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3a61e1-fq8l2"] Oct 30 10:00:04.860670 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:04.860641 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-3a61e1-6489f9fdfb-lt57n" Oct 30 10:00:04.890373 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:04.890352 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3a61e1-689c7f9675-6b84c" Oct 30 10:00:05.008292 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:05.008255 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1195ec8-c914-4d63-aca9-85e8b74b5d44-kube-api-access-62f4c\" (UniqueName: \"kubernetes.io/projected/d1195ec8-c914-4d63-aca9-85e8b74b5d44-kube-api-access-62f4c\") pod \"d1195ec8-c914-4d63-aca9-85e8b74b5d44\" (UID: \"d1195ec8-c914-4d63-aca9-85e8b74b5d44\") " Oct 30 10:00:05.008496 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:05.008313 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30a13eb2-b7e5-42ff-b5e9-993a20d588aa-kube-api-access-tpbcq\" (UniqueName: \"kubernetes.io/projected/30a13eb2-b7e5-42ff-b5e9-993a20d588aa-kube-api-access-tpbcq\") pod \"30a13eb2-b7e5-42ff-b5e9-993a20d588aa\" (UID: \"30a13eb2-b7e5-42ff-b5e9-993a20d588aa\") " Oct 30 10:00:05.008555 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:05.008499 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30a13eb2-b7e5-42ff-b5e9-993a20d588aa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30a13eb2-b7e5-42ff-b5e9-993a20d588aa-plugins-config-volume\") pod \"30a13eb2-b7e5-42ff-b5e9-993a20d588aa\" (UID: \"30a13eb2-b7e5-42ff-b5e9-993a20d588aa\") " Oct 30 10:00:05.008659 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:05.008639 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1195ec8-c914-4d63-aca9-85e8b74b5d44-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1195ec8-c914-4d63-aca9-85e8b74b5d44-plugins-config-volume\") pod \"d1195ec8-c914-4d63-aca9-85e8b74b5d44\" (UID: \"d1195ec8-c914-4d63-aca9-85e8b74b5d44\") " Oct 30 10:00:05.008963 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:05.008936 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30a13eb2-b7e5-42ff-b5e9-993a20d588aa-plugins-config-volume" pod "30a13eb2-b7e5-42ff-b5e9-993a20d588aa" (UID: "30a13eb2-b7e5-42ff-b5e9-993a20d588aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:00:05.009071 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:05.008997 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1195ec8-c914-4d63-aca9-85e8b74b5d44-plugins-config-volume" pod "d1195ec8-c914-4d63-aca9-85e8b74b5d44" (UID: "d1195ec8-c914-4d63-aca9-85e8b74b5d44"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:00:05.009178 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:05.009157 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30a13eb2-b7e5-42ff-b5e9-993a20d588aa-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:00:05.009219 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:05.009186 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1195ec8-c914-4d63-aca9-85e8b74b5d44-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:00:05.010459 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:05.010434 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30a13eb2-b7e5-42ff-b5e9-993a20d588aa-kube-api-access-tpbcq" pod "30a13eb2-b7e5-42ff-b5e9-993a20d588aa" (UID: "30a13eb2-b7e5-42ff-b5e9-993a20d588aa"). InnerVolumeSpecName "kube-api-access-tpbcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:00:05.010513 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:05.010464 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1195ec8-c914-4d63-aca9-85e8b74b5d44-kube-api-access-62f4c" pod "d1195ec8-c914-4d63-aca9-85e8b74b5d44" (UID: "d1195ec8-c914-4d63-aca9-85e8b74b5d44"). InnerVolumeSpecName "kube-api-access-62f4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:00:05.110688 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:05.110606 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62f4c\" (UniqueName: \"kubernetes.io/projected/d1195ec8-c914-4d63-aca9-85e8b74b5d44-kube-api-access-62f4c\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:00:05.110688 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:05.110635 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpbcq\" (UniqueName: \"kubernetes.io/projected/30a13eb2-b7e5-42ff-b5e9-993a20d588aa-kube-api-access-tpbcq\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:00:05.151520 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:05.151488 4562 generic.go:356] "Generic (PLEG): container finished" podID="30a13eb2-b7e5-42ff-b5e9-993a20d588aa" containerID="ce60eae137911ea5c9203c6cfc80f5938a12a48479ff47c86ff38f63d6757cc6" exitCode=0 Oct 30 10:00:05.151691 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:05.151545 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-3a61e1-6489f9fdfb-lt57n" Oct 30 10:00:05.151691 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:05.151577 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-3a61e1-6489f9fdfb-lt57n" event={"ID":"30a13eb2-b7e5-42ff-b5e9-993a20d588aa","Type":"ContainerDied","Data":"ce60eae137911ea5c9203c6cfc80f5938a12a48479ff47c86ff38f63d6757cc6"} Oct 30 10:00:05.151691 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:05.151656 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-3a61e1-6489f9fdfb-lt57n" event={"ID":"30a13eb2-b7e5-42ff-b5e9-993a20d588aa","Type":"ContainerDied","Data":"7003f94e05bd38f7906ffb21f46b445fad024f56a572afd8c83086250bb978e7"} Oct 30 10:00:05.151691 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:05.151685 4562 scope.go:122] "RemoveContainer" containerID="ce60eae137911ea5c9203c6cfc80f5938a12a48479ff47c86ff38f63d6757cc6" Oct 30 10:00:05.152966 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:05.152945 4562 generic.go:356] "Generic (PLEG): container finished" podID="d1195ec8-c914-4d63-aca9-85e8b74b5d44" containerID="d9a285e72bb8d4fff6dff6fb999172014528e8505fc6faa6f129a7b80ca37644" exitCode=0 Oct 30 10:00:05.153059 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:05.153009 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3a61e1-689c7f9675-6b84c" event={"ID":"d1195ec8-c914-4d63-aca9-85e8b74b5d44","Type":"ContainerDied","Data":"d9a285e72bb8d4fff6dff6fb999172014528e8505fc6faa6f129a7b80ca37644"} Oct 30 10:00:05.153125 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:05.153084 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3a61e1-689c7f9675-6b84c" event={"ID":"d1195ec8-c914-4d63-aca9-85e8b74b5d44","Type":"ContainerDied","Data":"046161ad3843e1e9bbd19be909781e2cfd5257b97ef50eb88920d1a14f817d62"} Oct 30 10:00:05.153125 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:05.153021 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3a61e1-689c7f9675-6b84c" Oct 30 10:00:05.162907 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:05.162805 4562 scope.go:122] "RemoveContainer" containerID="ce60eae137911ea5c9203c6cfc80f5938a12a48479ff47c86ff38f63d6757cc6" Oct 30 10:00:05.163302 ip-10-0-143-244 kubenswrapper[4562]: E1030 10:00:05.163277 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce60eae137911ea5c9203c6cfc80f5938a12a48479ff47c86ff38f63d6757cc6\": container with ID starting with ce60eae137911ea5c9203c6cfc80f5938a12a48479ff47c86ff38f63d6757cc6 not found: ID does not exist" containerID="ce60eae137911ea5c9203c6cfc80f5938a12a48479ff47c86ff38f63d6757cc6" Oct 30 10:00:05.163424 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:05.163312 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce60eae137911ea5c9203c6cfc80f5938a12a48479ff47c86ff38f63d6757cc6"} err="failed to get container status \"ce60eae137911ea5c9203c6cfc80f5938a12a48479ff47c86ff38f63d6757cc6\": rpc error: code = NotFound desc = could not find container \"ce60eae137911ea5c9203c6cfc80f5938a12a48479ff47c86ff38f63d6757cc6\": container with ID starting with ce60eae137911ea5c9203c6cfc80f5938a12a48479ff47c86ff38f63d6757cc6 not found: ID does not exist" Oct 30 10:00:05.163424 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:05.163334 4562 scope.go:122] "RemoveContainer" containerID="d9a285e72bb8d4fff6dff6fb999172014528e8505fc6faa6f129a7b80ca37644" Oct 30 10:00:05.173276 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:05.173259 4562 scope.go:122] "RemoveContainer" containerID="d9a285e72bb8d4fff6dff6fb999172014528e8505fc6faa6f129a7b80ca37644" Oct 30 10:00:05.173618 ip-10-0-143-244 kubenswrapper[4562]: E1030 10:00:05.173594 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9a285e72bb8d4fff6dff6fb999172014528e8505fc6faa6f129a7b80ca37644\": container with ID starting with d9a285e72bb8d4fff6dff6fb999172014528e8505fc6faa6f129a7b80ca37644 not found: ID does not exist" containerID="d9a285e72bb8d4fff6dff6fb999172014528e8505fc6faa6f129a7b80ca37644" Oct 30 10:00:05.173707 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:05.173624 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9a285e72bb8d4fff6dff6fb999172014528e8505fc6faa6f129a7b80ca37644"} err="failed to get container status \"d9a285e72bb8d4fff6dff6fb999172014528e8505fc6faa6f129a7b80ca37644\": rpc error: code = NotFound desc = could not find container \"d9a285e72bb8d4fff6dff6fb999172014528e8505fc6faa6f129a7b80ca37644\": container with ID starting with d9a285e72bb8d4fff6dff6fb999172014528e8505fc6faa6f129a7b80ca37644 not found: ID does not exist" Oct 30 10:00:05.204242 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:05.204211 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-3a61e1-6489f9fdfb-lt57n"] Oct 30 10:00:05.208748 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:05.208722 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-3a61e1-6489f9fdfb-lt57n"] Oct 30 10:00:05.244397 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:05.244353 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3a61e1-689c7f9675-6b84c"] Oct 30 10:00:05.249467 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:05.249427 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3a61e1-689c7f9675-6b84c"] Oct 30 10:00:05.284720 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:05.284422 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889240-gvt2h" Oct 30 10:00:05.414893 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:05.414813 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc6684d6-8c9b-4a9d-b2bc-9f825424b546-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc6684d6-8c9b-4a9d-b2bc-9f825424b546-maas-api-service-ca\") pod \"cc6684d6-8c9b-4a9d-b2bc-9f825424b546\" (UID: \"cc6684d6-8c9b-4a9d-b2bc-9f825424b546\") " Oct 30 10:00:05.414893 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:05.414882 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc6684d6-8c9b-4a9d-b2bc-9f825424b546-kube-api-access-rthrp\" (UniqueName: \"kubernetes.io/projected/cc6684d6-8c9b-4a9d-b2bc-9f825424b546-kube-api-access-rthrp\") pod \"cc6684d6-8c9b-4a9d-b2bc-9f825424b546\" (UID: \"cc6684d6-8c9b-4a9d-b2bc-9f825424b546\") " Oct 30 10:00:05.415172 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:05.415150 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc6684d6-8c9b-4a9d-b2bc-9f825424b546-maas-api-service-ca" pod "cc6684d6-8c9b-4a9d-b2bc-9f825424b546" (UID: "cc6684d6-8c9b-4a9d-b2bc-9f825424b546"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:00:05.417061 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:05.417036 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc6684d6-8c9b-4a9d-b2bc-9f825424b546-kube-api-access-rthrp" pod "cc6684d6-8c9b-4a9d-b2bc-9f825424b546" (UID: "cc6684d6-8c9b-4a9d-b2bc-9f825424b546"). InnerVolumeSpecName "kube-api-access-rthrp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:00:05.516575 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:05.516537 4562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc6684d6-8c9b-4a9d-b2bc-9f825424b546-maas-api-service-ca\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:00:05.516575 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:05.516567 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rthrp\" (UniqueName: \"kubernetes.io/projected/cc6684d6-8c9b-4a9d-b2bc-9f825424b546-kube-api-access-rthrp\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:00:05.734597 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:05.734560 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a525bd-29889240-l2v6p"] Oct 30 10:00:05.867161 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:05.867121 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a525bd-29889240-l2v6p" Oct 30 10:00:05.874249 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:05.874222 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30a13eb2-b7e5-42ff-b5e9-993a20d588aa" path="/var/lib/kubelet/pods/30a13eb2-b7e5-42ff-b5e9-993a20d588aa/volumes" Oct 30 10:00:05.874681 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:05.874666 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a11d9bcd-94a9-434c-b2c2-5918ca0d2504" path="/var/lib/kubelet/pods/a11d9bcd-94a9-434c-b2c2-5918ca0d2504/volumes" Oct 30 10:00:05.875108 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:05.875089 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1195ec8-c914-4d63-aca9-85e8b74b5d44" path="/var/lib/kubelet/pods/d1195ec8-c914-4d63-aca9-85e8b74b5d44/volumes" Oct 30 10:00:06.033486 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:06.033376 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03ef2570-c3b8-48c9-9632-de3c09ea41dc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03ef2570-c3b8-48c9-9632-de3c09ea41dc-maas-api-service-ca\") pod \"03ef2570-c3b8-48c9-9632-de3c09ea41dc\" (UID: \"03ef2570-c3b8-48c9-9632-de3c09ea41dc\") " Oct 30 10:00:06.033663 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:06.033605 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03ef2570-c3b8-48c9-9632-de3c09ea41dc-kube-api-access-chxx7\" (UniqueName: \"kubernetes.io/projected/03ef2570-c3b8-48c9-9632-de3c09ea41dc-kube-api-access-chxx7\") pod \"03ef2570-c3b8-48c9-9632-de3c09ea41dc\" (UID: \"03ef2570-c3b8-48c9-9632-de3c09ea41dc\") " Oct 30 10:00:06.033775 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:06.033747 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03ef2570-c3b8-48c9-9632-de3c09ea41dc-maas-api-service-ca" pod "03ef2570-c3b8-48c9-9632-de3c09ea41dc" (UID: "03ef2570-c3b8-48c9-9632-de3c09ea41dc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:00:06.034065 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:06.034050 4562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03ef2570-c3b8-48c9-9632-de3c09ea41dc-maas-api-service-ca\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:00:06.035692 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:06.035671 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03ef2570-c3b8-48c9-9632-de3c09ea41dc-kube-api-access-chxx7" pod "03ef2570-c3b8-48c9-9632-de3c09ea41dc" (UID: "03ef2570-c3b8-48c9-9632-de3c09ea41dc"). InnerVolumeSpecName "kube-api-access-chxx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:00:06.136305 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:06.135883 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chxx7\" (UniqueName: \"kubernetes.io/projected/03ef2570-c3b8-48c9-9632-de3c09ea41dc-kube-api-access-chxx7\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:00:06.160443 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:06.160408 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889240-gvt2h" Oct 30 10:00:06.160443 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:06.160413 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889240-gvt2h" event={"ID":"cc6684d6-8c9b-4a9d-b2bc-9f825424b546","Type":"ContainerDied","Data":"44a0dd60ed828f29eda920aef59e659a504d4ad898d21c5964ce0a40f98e62a9"} Oct 30 10:00:06.160727 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:06.160490 4562 scope.go:122] "RemoveContainer" containerID="bdd24801e20c9bf871653556c843eb5cb6c7bdbaa3430e469888a0224af7741d" Oct 30 10:00:06.163916 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:06.163896 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a525bd-29889240-l2v6p" Oct 30 10:00:06.164042 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:06.163895 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a525bd-29889240-l2v6p" event={"ID":"03ef2570-c3b8-48c9-9632-de3c09ea41dc","Type":"ContainerDied","Data":"2963af955314dcc0afe418c20521176d98757f7de983c3957649b0c891c496ad"} Oct 30 10:00:06.171473 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:06.171458 4562 scope.go:122] "RemoveContainer" containerID="339f0200ef68e6fc98d5b7f4c0ca5ef1bf6646ebdd1d64acca9836629dd4e48c" Oct 30 10:00:06.215278 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:06.215254 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a525bd-29889240-l2v6p"] Oct 30 10:00:06.221680 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:06.221656 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a525bd-29889240-l2v6p"] Oct 30 10:00:06.267422 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:06.267372 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889240-gvt2h"] Oct 30 10:00:06.269572 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:06.269544 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3a61e1-29889240-gvt2h"] Oct 30 10:00:06.303222 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:06.303155 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a525bd-f7v9q"] Oct 30 10:00:06.305193 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:06.305172 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a525bd-f7v9q"] Oct 30 10:00:07.873501 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:07.873468 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03ef2570-c3b8-48c9-9632-de3c09ea41dc" path="/var/lib/kubelet/pods/03ef2570-c3b8-48c9-9632-de3c09ea41dc/volumes" Oct 30 10:00:07.873922 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:07.873908 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ec08754-556b-426a-8625-67125a70caab" path="/var/lib/kubelet/pods/3ec08754-556b-426a-8625-67125a70caab/volumes" Oct 30 10:00:07.874324 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:07.874308 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc6684d6-8c9b-4a9d-b2bc-9f825424b546" path="/var/lib/kubelet/pods/cc6684d6-8c9b-4a9d-b2bc-9f825424b546/volumes" Oct 30 10:00:12.842473 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:12.842442 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9"] Oct 30 10:00:12.843149 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:12.843115 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9" podUID="f4c52d5e-0a44-479d-bfce-072508f5527a" containerName="istio-proxy" containerID="cri-o://f085a92738e9188c7ebb3623a0491b2758556fa495f56eed5f600e252cea6cde" gracePeriod=30 Oct 30 10:00:12.848757 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:12.848721 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd"] Oct 30 10:00:12.849158 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:12.849121 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd" podUID="9ed52c04-442a-4a04-8a72-c40062a5e3ba" containerName="istio-proxy" containerID="cri-o://6866fbc01b29285c1c8232c865f27766be9819dda948a40292c79c33fd74df4c" gracePeriod=30 Oct 30 10:00:13.619891 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:13.619858 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24"] Oct 30 10:00:13.620102 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:13.620074 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24" podUID="18686e38-6651-454d-b9ad-996ffebe2b91" containerName="istio-proxy" containerID="cri-o://c41e759d6fc5c724727ce44f55e9a2177cb97954f052d3db913c105f08f52e26" gracePeriod=30 Oct 30 10:00:14.867426 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:14.867361 4562 scope.go:122] "RemoveContainer" containerID="7f03d7fb6b4c3ca48349c33693d60095caada0f46ca705b96b35995da3cc1f67" Oct 30 10:00:15.216627 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:15.216593 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889240-bc26j" event={"ID":"e2232a56-3384-41a6-961d-ec5b4ce0db03","Type":"ContainerStarted","Data":"89e6b5d1423573fb4d29abd2ee9c0f083f850260d54b0ed969b2e557c4b8778d"} Oct 30 10:00:15.258772 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:15.258721 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889240-bc26j" podStartSLOduration=15.258707942000001 podStartE2EDuration="15.258707942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 10:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:00:15.257530297 +0000 UTC m=+3975.519091284" watchObservedRunningTime="2026-10-30 10:00:15.258707942 +0000 UTC m=+3975.520268929" Oct 30 10:00:16.224185 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:16.224152 4562 generic.go:356] "Generic (PLEG): container finished" podID="e2232a56-3384-41a6-961d-ec5b4ce0db03" containerID="89e6b5d1423573fb4d29abd2ee9c0f083f850260d54b0ed969b2e557c4b8778d" exitCode=60 Oct 30 10:00:16.224635 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:16.224222 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889240-bc26j" event={"ID":"e2232a56-3384-41a6-961d-ec5b4ce0db03","Type":"ContainerDied","Data":"89e6b5d1423573fb4d29abd2ee9c0f083f850260d54b0ed969b2e557c4b8778d"} Oct 30 10:00:16.224635 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:16.224283 4562 scope.go:122] "RemoveContainer" containerID="7f03d7fb6b4c3ca48349c33693d60095caada0f46ca705b96b35995da3cc1f67" Oct 30 10:00:16.224635 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:16.224625 4562 scope.go:122] "RemoveContainer" containerID="89e6b5d1423573fb4d29abd2ee9c0f083f850260d54b0ed969b2e557c4b8778d" Oct 30 10:00:16.224896 ip-10-0-143-244 kubenswrapper[4562]: E1030 10:00:16.224876 4562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889240-bc26j_odh-ai-gateway-infra(e2232a56-3384-41a6-961d-ec5b4ce0db03)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889240-bc26j" podUID="e2232a56-3384-41a6-961d-ec5b4ce0db03" Oct 30 10:00:16.270102 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:16.270071 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889240-bc26j"] Oct 30 10:00:17.384012 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:17.383984 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889240-bc26j" Oct 30 10:00:17.459330 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:17.459299 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2232a56-3384-41a6-961d-ec5b4ce0db03-kube-api-access-b2w9r\" (UniqueName: \"kubernetes.io/projected/e2232a56-3384-41a6-961d-ec5b4ce0db03-kube-api-access-b2w9r\") pod \"e2232a56-3384-41a6-961d-ec5b4ce0db03\" (UID: \"e2232a56-3384-41a6-961d-ec5b4ce0db03\") " Oct 30 10:00:17.459544 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:17.459430 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2232a56-3384-41a6-961d-ec5b4ce0db03-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2232a56-3384-41a6-961d-ec5b4ce0db03-maas-api-service-ca\") pod \"e2232a56-3384-41a6-961d-ec5b4ce0db03\" (UID: \"e2232a56-3384-41a6-961d-ec5b4ce0db03\") " Oct 30 10:00:17.459845 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:17.459823 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2232a56-3384-41a6-961d-ec5b4ce0db03-maas-api-service-ca" pod "e2232a56-3384-41a6-961d-ec5b4ce0db03" (UID: "e2232a56-3384-41a6-961d-ec5b4ce0db03"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:00:17.461535 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:17.461508 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2232a56-3384-41a6-961d-ec5b4ce0db03-kube-api-access-b2w9r" pod "e2232a56-3384-41a6-961d-ec5b4ce0db03" (UID: "e2232a56-3384-41a6-961d-ec5b4ce0db03"). InnerVolumeSpecName "kube-api-access-b2w9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:00:17.560454 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:17.560365 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2w9r\" (UniqueName: \"kubernetes.io/projected/e2232a56-3384-41a6-961d-ec5b4ce0db03-kube-api-access-b2w9r\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:00:17.560454 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:17.560409 4562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2232a56-3384-41a6-961d-ec5b4ce0db03-maas-api-service-ca\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:00:18.126080 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.126050 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd" Oct 30 10:00:18.143477 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.143451 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9" Oct 30 10:00:18.166719 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.166687 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4c52d5e-0a44-479d-bfce-072508f5527a-kube-api-access-pnqsq\" (UniqueName: \"kubernetes.io/projected/f4c52d5e-0a44-479d-bfce-072508f5527a-kube-api-access-pnqsq\") pod \"f4c52d5e-0a44-479d-bfce-072508f5527a\" (UID: \"f4c52d5e-0a44-479d-bfce-072508f5527a\") " Oct 30 10:00:18.166886 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.166752 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ed52c04-442a-4a04-8a72-c40062a5e3ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ed52c04-442a-4a04-8a72-c40062a5e3ba-istio-envoy\") pod \"9ed52c04-442a-4a04-8a72-c40062a5e3ba\" (UID: \"9ed52c04-442a-4a04-8a72-c40062a5e3ba\") " Oct 30 10:00:18.166886 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.166771 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ed52c04-442a-4a04-8a72-c40062a5e3ba-istio-token\" (UniqueName: \"kubernetes.io/projected/9ed52c04-442a-4a04-8a72-c40062a5e3ba-istio-token\") pod \"9ed52c04-442a-4a04-8a72-c40062a5e3ba\" (UID: \"9ed52c04-442a-4a04-8a72-c40062a5e3ba\") " Oct 30 10:00:18.166886 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.166805 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4c52d5e-0a44-479d-bfce-072508f5527a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4c52d5e-0a44-479d-bfce-072508f5527a-istiod-ca-cert\") pod \"f4c52d5e-0a44-479d-bfce-072508f5527a\" (UID: \"f4c52d5e-0a44-479d-bfce-072508f5527a\") " Oct 30 10:00:18.166886 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.166824 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9ed52c04-442a-4a04-8a72-c40062a5e3ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ed52c04-442a-4a04-8a72-c40062a5e3ba-istio-podinfo\") pod \"9ed52c04-442a-4a04-8a72-c40062a5e3ba\" (UID: \"9ed52c04-442a-4a04-8a72-c40062a5e3ba\") " Oct 30 10:00:18.166886 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.166860 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4c52d5e-0a44-479d-bfce-072508f5527a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4c52d5e-0a44-479d-bfce-072508f5527a-workload-socket\") pod \"f4c52d5e-0a44-479d-bfce-072508f5527a\" (UID: \"f4c52d5e-0a44-479d-bfce-072508f5527a\") " Oct 30 10:00:18.167113 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.166896 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ed52c04-442a-4a04-8a72-c40062a5e3ba-kube-api-access-gstff\" (UniqueName: \"kubernetes.io/projected/9ed52c04-442a-4a04-8a72-c40062a5e3ba-kube-api-access-gstff\") pod \"9ed52c04-442a-4a04-8a72-c40062a5e3ba\" (UID: \"9ed52c04-442a-4a04-8a72-c40062a5e3ba\") " Oct 30 10:00:18.167113 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.166923 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ed52c04-442a-4a04-8a72-c40062a5e3ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ed52c04-442a-4a04-8a72-c40062a5e3ba-credential-socket\") pod \"9ed52c04-442a-4a04-8a72-c40062a5e3ba\" (UID: \"9ed52c04-442a-4a04-8a72-c40062a5e3ba\") " Oct 30 10:00:18.167113 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.166991 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4c52d5e-0a44-479d-bfce-072508f5527a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4c52d5e-0a44-479d-bfce-072508f5527a-istio-data\") pod \"f4c52d5e-0a44-479d-bfce-072508f5527a\" (UID: \"f4c52d5e-0a44-479d-bfce-072508f5527a\") " Oct 30 10:00:18.167113 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.167020 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ed52c04-442a-4a04-8a72-c40062a5e3ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ed52c04-442a-4a04-8a72-c40062a5e3ba-istio-data\") pod \"9ed52c04-442a-4a04-8a72-c40062a5e3ba\" (UID: \"9ed52c04-442a-4a04-8a72-c40062a5e3ba\") " Oct 30 10:00:18.167113 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.167086 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4c52d5e-0a44-479d-bfce-072508f5527a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4c52d5e-0a44-479d-bfce-072508f5527a-workload-certs\") pod \"f4c52d5e-0a44-479d-bfce-072508f5527a\" (UID: \"f4c52d5e-0a44-479d-bfce-072508f5527a\") " Oct 30 10:00:18.167415 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.167118 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ed52c04-442a-4a04-8a72-c40062a5e3ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ed52c04-442a-4a04-8a72-c40062a5e3ba-istiod-ca-cert\") pod \"9ed52c04-442a-4a04-8a72-c40062a5e3ba\" (UID: \"9ed52c04-442a-4a04-8a72-c40062a5e3ba\") " Oct 30 10:00:18.167415 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.167204 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4c52d5e-0a44-479d-bfce-072508f5527a-istio-token\" (UniqueName: \"kubernetes.io/projected/f4c52d5e-0a44-479d-bfce-072508f5527a-istio-token\") pod \"f4c52d5e-0a44-479d-bfce-072508f5527a\" (UID: \"f4c52d5e-0a44-479d-bfce-072508f5527a\") " Oct 30 10:00:18.167415 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.167220 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4c52d5e-0a44-479d-bfce-072508f5527a-istiod-ca-cert" pod "f4c52d5e-0a44-479d-bfce-072508f5527a" (UID: "f4c52d5e-0a44-479d-bfce-072508f5527a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:00:18.167415 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.167235 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4c52d5e-0a44-479d-bfce-072508f5527a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4c52d5e-0a44-479d-bfce-072508f5527a-istio-envoy\") pod \"f4c52d5e-0a44-479d-bfce-072508f5527a\" (UID: \"f4c52d5e-0a44-479d-bfce-072508f5527a\") " Oct 30 10:00:18.167415 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.167266 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f4c52d5e-0a44-479d-bfce-072508f5527a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4c52d5e-0a44-479d-bfce-072508f5527a-istio-podinfo\") pod \"f4c52d5e-0a44-479d-bfce-072508f5527a\" (UID: \"f4c52d5e-0a44-479d-bfce-072508f5527a\") " Oct 30 10:00:18.167415 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.167334 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ed52c04-442a-4a04-8a72-c40062a5e3ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ed52c04-442a-4a04-8a72-c40062a5e3ba-workload-certs\") pod \"9ed52c04-442a-4a04-8a72-c40062a5e3ba\" (UID: \"9ed52c04-442a-4a04-8a72-c40062a5e3ba\") " Oct 30 10:00:18.167415 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.167413 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4c52d5e-0a44-479d-bfce-072508f5527a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4c52d5e-0a44-479d-bfce-072508f5527a-credential-socket\") pod \"f4c52d5e-0a44-479d-bfce-072508f5527a\" (UID: \"f4c52d5e-0a44-479d-bfce-072508f5527a\") " Oct 30 10:00:18.167800 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.167446 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ed52c04-442a-4a04-8a72-c40062a5e3ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ed52c04-442a-4a04-8a72-c40062a5e3ba-workload-socket\") pod \"9ed52c04-442a-4a04-8a72-c40062a5e3ba\" (UID: \"9ed52c04-442a-4a04-8a72-c40062a5e3ba\") " Oct 30 10:00:18.167938 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.167913 4562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4c52d5e-0a44-479d-bfce-072508f5527a-istiod-ca-cert\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:00:18.168495 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.168466 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ed52c04-442a-4a04-8a72-c40062a5e3ba-istiod-ca-cert" pod "9ed52c04-442a-4a04-8a72-c40062a5e3ba" (UID: "9ed52c04-442a-4a04-8a72-c40062a5e3ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:00:18.168825 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.168769 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4c52d5e-0a44-479d-bfce-072508f5527a-credential-socket" pod "f4c52d5e-0a44-479d-bfce-072508f5527a" (UID: "f4c52d5e-0a44-479d-bfce-072508f5527a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:00:18.168825 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.168778 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ed52c04-442a-4a04-8a72-c40062a5e3ba-workload-certs" pod "9ed52c04-442a-4a04-8a72-c40062a5e3ba" (UID: "9ed52c04-442a-4a04-8a72-c40062a5e3ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:00:18.168825 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.168796 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ed52c04-442a-4a04-8a72-c40062a5e3ba-istio-data" pod "9ed52c04-442a-4a04-8a72-c40062a5e3ba" (UID: "9ed52c04-442a-4a04-8a72-c40062a5e3ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:00:18.169063 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.168861 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4c52d5e-0a44-479d-bfce-072508f5527a-workload-socket" pod "f4c52d5e-0a44-479d-bfce-072508f5527a" (UID: "f4c52d5e-0a44-479d-bfce-072508f5527a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:00:18.169063 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.168972 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4c52d5e-0a44-479d-bfce-072508f5527a-istio-data" pod "f4c52d5e-0a44-479d-bfce-072508f5527a" (UID: "f4c52d5e-0a44-479d-bfce-072508f5527a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:00:18.169063 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.169027 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4c52d5e-0a44-479d-bfce-072508f5527a-workload-certs" pod "f4c52d5e-0a44-479d-bfce-072508f5527a" (UID: "f4c52d5e-0a44-479d-bfce-072508f5527a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:00:18.170247 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.170221 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ed52c04-442a-4a04-8a72-c40062a5e3ba-credential-socket" pod "9ed52c04-442a-4a04-8a72-c40062a5e3ba" (UID: "9ed52c04-442a-4a04-8a72-c40062a5e3ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:00:18.170335 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.170289 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ed52c04-442a-4a04-8a72-c40062a5e3ba-workload-socket" pod "9ed52c04-442a-4a04-8a72-c40062a5e3ba" (UID: "9ed52c04-442a-4a04-8a72-c40062a5e3ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:00:18.170422 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.170356 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4c52d5e-0a44-479d-bfce-072508f5527a-kube-api-access-pnqsq" pod "f4c52d5e-0a44-479d-bfce-072508f5527a" (UID: "f4c52d5e-0a44-479d-bfce-072508f5527a"). InnerVolumeSpecName "kube-api-access-pnqsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:00:18.170483 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.170450 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ed52c04-442a-4a04-8a72-c40062a5e3ba-kube-api-access-gstff" pod "9ed52c04-442a-4a04-8a72-c40062a5e3ba" (UID: "9ed52c04-442a-4a04-8a72-c40062a5e3ba"). InnerVolumeSpecName "kube-api-access-gstff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:00:18.170483 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.170467 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ed52c04-442a-4a04-8a72-c40062a5e3ba-istio-token" pod "9ed52c04-442a-4a04-8a72-c40062a5e3ba" (UID: "9ed52c04-442a-4a04-8a72-c40062a5e3ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:00:18.170943 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.170918 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ed52c04-442a-4a04-8a72-c40062a5e3ba-istio-envoy" pod "9ed52c04-442a-4a04-8a72-c40062a5e3ba" (UID: "9ed52c04-442a-4a04-8a72-c40062a5e3ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:00:18.171120 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.171106 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ed52c04-442a-4a04-8a72-c40062a5e3ba-istio-podinfo" pod "9ed52c04-442a-4a04-8a72-c40062a5e3ba" (UID: "9ed52c04-442a-4a04-8a72-c40062a5e3ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 10:00:18.171747 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.171728 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f4c52d5e-0a44-479d-bfce-072508f5527a-istio-podinfo" pod "f4c52d5e-0a44-479d-bfce-072508f5527a" (UID: "f4c52d5e-0a44-479d-bfce-072508f5527a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 10:00:18.172066 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.172039 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4c52d5e-0a44-479d-bfce-072508f5527a-istio-token" pod "f4c52d5e-0a44-479d-bfce-072508f5527a" (UID: "f4c52d5e-0a44-479d-bfce-072508f5527a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:00:18.172904 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.172885 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4c52d5e-0a44-479d-bfce-072508f5527a-istio-envoy" pod "f4c52d5e-0a44-479d-bfce-072508f5527a" (UID: "f4c52d5e-0a44-479d-bfce-072508f5527a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:00:18.241078 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.241036 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889240-bc26j" Oct 30 10:00:18.241078 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.241052 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889240-bc26j" event={"ID":"e2232a56-3384-41a6-961d-ec5b4ce0db03","Type":"ContainerDied","Data":"601ce9d1d2cbd0ad1e30b8100a2f8910d4f35d40dad64862e028e4b7d564d19b"} Oct 30 10:00:18.241335 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.241159 4562 scope.go:122] "RemoveContainer" containerID="89e6b5d1423573fb4d29abd2ee9c0f083f850260d54b0ed969b2e557c4b8778d" Oct 30 10:00:18.243515 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.243487 4562 generic.go:356] "Generic (PLEG): container finished" podID="f4c52d5e-0a44-479d-bfce-072508f5527a" containerID="f085a92738e9188c7ebb3623a0491b2758556fa495f56eed5f600e252cea6cde" exitCode=0 Oct 30 10:00:18.243651 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.243605 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9" event={"ID":"f4c52d5e-0a44-479d-bfce-072508f5527a","Type":"ContainerDied","Data":"f085a92738e9188c7ebb3623a0491b2758556fa495f56eed5f600e252cea6cde"} Oct 30 10:00:18.243713 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.243652 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9" event={"ID":"f4c52d5e-0a44-479d-bfce-072508f5527a","Type":"ContainerDied","Data":"302ddc7252d3415e00f841874973bd44a309559a07ce4d6b5913e5747574a61c"} Oct 30 10:00:18.243976 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.243945 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9" Oct 30 10:00:18.245223 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.245196 4562 generic.go:356] "Generic (PLEG): container finished" podID="9ed52c04-442a-4a04-8a72-c40062a5e3ba" containerID="6866fbc01b29285c1c8232c865f27766be9819dda948a40292c79c33fd74df4c" exitCode=0 Oct 30 10:00:18.245314 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.245254 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd" Oct 30 10:00:18.245314 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.245270 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd" event={"ID":"9ed52c04-442a-4a04-8a72-c40062a5e3ba","Type":"ContainerDied","Data":"6866fbc01b29285c1c8232c865f27766be9819dda948a40292c79c33fd74df4c"} Oct 30 10:00:18.245452 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.245313 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd" event={"ID":"9ed52c04-442a-4a04-8a72-c40062a5e3ba","Type":"ContainerDied","Data":"ec0b04ad109f07ebccb9a1553dbb326b91e1e45749abf658bb5bfa104b0584f5"} Oct 30 10:00:18.254617 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.254592 4562 scope.go:122] "RemoveContainer" containerID="f085a92738e9188c7ebb3623a0491b2758556fa495f56eed5f600e252cea6cde" Oct 30 10:00:18.264224 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.264204 4562 scope.go:122] "RemoveContainer" containerID="f085a92738e9188c7ebb3623a0491b2758556fa495f56eed5f600e252cea6cde" Oct 30 10:00:18.264513 ip-10-0-143-244 kubenswrapper[4562]: E1030 10:00:18.264493 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f085a92738e9188c7ebb3623a0491b2758556fa495f56eed5f600e252cea6cde\": container with ID starting with f085a92738e9188c7ebb3623a0491b2758556fa495f56eed5f600e252cea6cde not found: ID does not exist" containerID="f085a92738e9188c7ebb3623a0491b2758556fa495f56eed5f600e252cea6cde" Oct 30 10:00:18.264576 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.264521 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f085a92738e9188c7ebb3623a0491b2758556fa495f56eed5f600e252cea6cde"} err="failed to get container status \"f085a92738e9188c7ebb3623a0491b2758556fa495f56eed5f600e252cea6cde\": rpc error: code = NotFound desc = could not find container \"f085a92738e9188c7ebb3623a0491b2758556fa495f56eed5f600e252cea6cde\": container with ID starting with f085a92738e9188c7ebb3623a0491b2758556fa495f56eed5f600e252cea6cde not found: ID does not exist" Oct 30 10:00:18.264576 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.264539 4562 scope.go:122] "RemoveContainer" containerID="6866fbc01b29285c1c8232c865f27766be9819dda948a40292c79c33fd74df4c" Oct 30 10:00:18.269360 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.269335 4562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ed52c04-442a-4a04-8a72-c40062a5e3ba-workload-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:00:18.269711 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.269364 4562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4c52d5e-0a44-479d-bfce-072508f5527a-credential-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:00:18.269711 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.269593 4562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ed52c04-442a-4a04-8a72-c40062a5e3ba-workload-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:00:18.269711 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.269622 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnqsq\" (UniqueName: \"kubernetes.io/projected/f4c52d5e-0a44-479d-bfce-072508f5527a-kube-api-access-pnqsq\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:00:18.269711 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.269636 4562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ed52c04-442a-4a04-8a72-c40062a5e3ba-istio-envoy\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:00:18.269711 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.269650 4562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ed52c04-442a-4a04-8a72-c40062a5e3ba-istio-token\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:00:18.269711 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.269662 4562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ed52c04-442a-4a04-8a72-c40062a5e3ba-istio-podinfo\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:00:18.269711 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.269675 4562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4c52d5e-0a44-479d-bfce-072508f5527a-workload-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:00:18.269711 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.269691 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gstff\" (UniqueName: \"kubernetes.io/projected/9ed52c04-442a-4a04-8a72-c40062a5e3ba-kube-api-access-gstff\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:00:18.269711 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.269704 4562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ed52c04-442a-4a04-8a72-c40062a5e3ba-credential-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:00:18.270039 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.269719 4562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4c52d5e-0a44-479d-bfce-072508f5527a-istio-data\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:00:18.270039 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.269731 4562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ed52c04-442a-4a04-8a72-c40062a5e3ba-istio-data\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:00:18.270039 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.269743 4562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4c52d5e-0a44-479d-bfce-072508f5527a-workload-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:00:18.270039 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.269757 4562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ed52c04-442a-4a04-8a72-c40062a5e3ba-istiod-ca-cert\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:00:18.270039 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.269770 4562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4c52d5e-0a44-479d-bfce-072508f5527a-istio-token\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:00:18.270039 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.269782 4562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4c52d5e-0a44-479d-bfce-072508f5527a-istio-envoy\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:00:18.270039 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.269795 4562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4c52d5e-0a44-479d-bfce-072508f5527a-istio-podinfo\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:00:18.273531 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.273514 4562 scope.go:122] "RemoveContainer" containerID="6866fbc01b29285c1c8232c865f27766be9819dda948a40292c79c33fd74df4c" Oct 30 10:00:18.273818 ip-10-0-143-244 kubenswrapper[4562]: E1030 10:00:18.273799 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6866fbc01b29285c1c8232c865f27766be9819dda948a40292c79c33fd74df4c\": container with ID starting with 6866fbc01b29285c1c8232c865f27766be9819dda948a40292c79c33fd74df4c not found: ID does not exist" containerID="6866fbc01b29285c1c8232c865f27766be9819dda948a40292c79c33fd74df4c" Oct 30 10:00:18.273883 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.273826 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6866fbc01b29285c1c8232c865f27766be9819dda948a40292c79c33fd74df4c"} err="failed to get container status \"6866fbc01b29285c1c8232c865f27766be9819dda948a40292c79c33fd74df4c\": rpc error: code = NotFound desc = could not find container \"6866fbc01b29285c1c8232c865f27766be9819dda948a40292c79c33fd74df4c\": container with ID starting with 6866fbc01b29285c1c8232c865f27766be9819dda948a40292c79c33fd74df4c not found: ID does not exist" Oct 30 10:00:18.305007 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.304976 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889240-bc26j"] Oct 30 10:00:18.311605 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.311577 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889240-bc26j"] Oct 30 10:00:18.346540 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.346504 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd"] Oct 30 10:00:18.348591 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.348569 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd"] Oct 30 10:00:18.382496 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.382424 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9"] Oct 30 10:00:18.389807 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.389775 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3a61e1-openshift-default-76c56d5b47-jtxk9"] Oct 30 10:00:18.881443 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.881415 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24" Oct 30 10:00:18.977184 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.977152 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/18686e38-6651-454d-b9ad-996ffebe2b91-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18686e38-6651-454d-b9ad-996ffebe2b91-istio-podinfo\") pod \"18686e38-6651-454d-b9ad-996ffebe2b91\" (UID: \"18686e38-6651-454d-b9ad-996ffebe2b91\") " Oct 30 10:00:18.977372 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.977208 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18686e38-6651-454d-b9ad-996ffebe2b91-kube-api-access-lxdzg\" (UniqueName: \"kubernetes.io/projected/18686e38-6651-454d-b9ad-996ffebe2b91-kube-api-access-lxdzg\") pod \"18686e38-6651-454d-b9ad-996ffebe2b91\" (UID: \"18686e38-6651-454d-b9ad-996ffebe2b91\") " Oct 30 10:00:18.977372 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.977280 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18686e38-6651-454d-b9ad-996ffebe2b91-istio-token\" (UniqueName: \"kubernetes.io/projected/18686e38-6651-454d-b9ad-996ffebe2b91-istio-token\") pod \"18686e38-6651-454d-b9ad-996ffebe2b91\" (UID: \"18686e38-6651-454d-b9ad-996ffebe2b91\") " Oct 30 10:00:18.977372 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.977321 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18686e38-6651-454d-b9ad-996ffebe2b91-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18686e38-6651-454d-b9ad-996ffebe2b91-istio-data\") pod \"18686e38-6651-454d-b9ad-996ffebe2b91\" (UID: \"18686e38-6651-454d-b9ad-996ffebe2b91\") " Oct 30 10:00:18.977575 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.977373 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18686e38-6651-454d-b9ad-996ffebe2b91-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18686e38-6651-454d-b9ad-996ffebe2b91-istiod-ca-cert\") pod \"18686e38-6651-454d-b9ad-996ffebe2b91\" (UID: \"18686e38-6651-454d-b9ad-996ffebe2b91\") " Oct 30 10:00:18.977575 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.977464 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18686e38-6651-454d-b9ad-996ffebe2b91-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18686e38-6651-454d-b9ad-996ffebe2b91-istio-envoy\") pod \"18686e38-6651-454d-b9ad-996ffebe2b91\" (UID: \"18686e38-6651-454d-b9ad-996ffebe2b91\") " Oct 30 10:00:18.977690 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.977586 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18686e38-6651-454d-b9ad-996ffebe2b91-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18686e38-6651-454d-b9ad-996ffebe2b91-credential-socket\") pod \"18686e38-6651-454d-b9ad-996ffebe2b91\" (UID: \"18686e38-6651-454d-b9ad-996ffebe2b91\") " Oct 30 10:00:18.977749 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.977687 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18686e38-6651-454d-b9ad-996ffebe2b91-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18686e38-6651-454d-b9ad-996ffebe2b91-workload-socket\") pod \"18686e38-6651-454d-b9ad-996ffebe2b91\" (UID: \"18686e38-6651-454d-b9ad-996ffebe2b91\") " Oct 30 10:00:18.977749 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.977737 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18686e38-6651-454d-b9ad-996ffebe2b91-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18686e38-6651-454d-b9ad-996ffebe2b91-workload-certs\") pod \"18686e38-6651-454d-b9ad-996ffebe2b91\" (UID: \"18686e38-6651-454d-b9ad-996ffebe2b91\") " Oct 30 10:00:18.977879 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.977809 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18686e38-6651-454d-b9ad-996ffebe2b91-istio-data" pod "18686e38-6651-454d-b9ad-996ffebe2b91" (UID: "18686e38-6651-454d-b9ad-996ffebe2b91"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:00:18.977937 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.977896 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18686e38-6651-454d-b9ad-996ffebe2b91-credential-socket" pod "18686e38-6651-454d-b9ad-996ffebe2b91" (UID: "18686e38-6651-454d-b9ad-996ffebe2b91"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:00:18.978069 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.978044 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18686e38-6651-454d-b9ad-996ffebe2b91-workload-certs" pod "18686e38-6651-454d-b9ad-996ffebe2b91" (UID: "18686e38-6651-454d-b9ad-996ffebe2b91"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:00:18.978164 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.978135 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18686e38-6651-454d-b9ad-996ffebe2b91-workload-socket" pod "18686e38-6651-454d-b9ad-996ffebe2b91" (UID: "18686e38-6651-454d-b9ad-996ffebe2b91"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:00:18.978235 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.978188 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18686e38-6651-454d-b9ad-996ffebe2b91-istiod-ca-cert" pod "18686e38-6651-454d-b9ad-996ffebe2b91" (UID: "18686e38-6651-454d-b9ad-996ffebe2b91"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:00:18.978298 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.978265 4562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18686e38-6651-454d-b9ad-996ffebe2b91-credential-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:00:18.978298 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.978287 4562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18686e38-6651-454d-b9ad-996ffebe2b91-workload-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:00:18.978430 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.978302 4562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18686e38-6651-454d-b9ad-996ffebe2b91-workload-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:00:18.978430 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.978316 4562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18686e38-6651-454d-b9ad-996ffebe2b91-istio-data\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:00:18.979555 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.979531 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18686e38-6651-454d-b9ad-996ffebe2b91-istio-podinfo" pod "18686e38-6651-454d-b9ad-996ffebe2b91" (UID: "18686e38-6651-454d-b9ad-996ffebe2b91"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 10:00:18.979664 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.979610 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18686e38-6651-454d-b9ad-996ffebe2b91-kube-api-access-lxdzg" pod "18686e38-6651-454d-b9ad-996ffebe2b91" (UID: "18686e38-6651-454d-b9ad-996ffebe2b91"). InnerVolumeSpecName "kube-api-access-lxdzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:00:18.979664 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.979637 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18686e38-6651-454d-b9ad-996ffebe2b91-istio-token" pod "18686e38-6651-454d-b9ad-996ffebe2b91" (UID: "18686e38-6651-454d-b9ad-996ffebe2b91"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:00:18.980042 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:18.980023 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18686e38-6651-454d-b9ad-996ffebe2b91-istio-envoy" pod "18686e38-6651-454d-b9ad-996ffebe2b91" (UID: "18686e38-6651-454d-b9ad-996ffebe2b91"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:00:19.044219 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:19.044183 4562 patch_prober.go:29] interesting pod/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 10:00:19.044368 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:19.044237 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2f69c8-openshift-default-566dd846b9-5jnbd" podUID="9ed52c04-442a-4a04-8a72-c40062a5e3ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 10:00:19.079033 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:19.079003 4562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18686e38-6651-454d-b9ad-996ffebe2b91-istiod-ca-cert\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:00:19.079033 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:19.079033 4562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18686e38-6651-454d-b9ad-996ffebe2b91-istio-envoy\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:00:19.079239 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:19.079043 4562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18686e38-6651-454d-b9ad-996ffebe2b91-istio-podinfo\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:00:19.079239 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:19.079051 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxdzg\" (UniqueName: \"kubernetes.io/projected/18686e38-6651-454d-b9ad-996ffebe2b91-kube-api-access-lxdzg\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:00:19.079239 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:19.079061 4562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18686e38-6651-454d-b9ad-996ffebe2b91-istio-token\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:00:19.254090 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:19.254008 4562 generic.go:356] "Generic (PLEG): container finished" podID="18686e38-6651-454d-b9ad-996ffebe2b91" containerID="c41e759d6fc5c724727ce44f55e9a2177cb97954f052d3db913c105f08f52e26" exitCode=0 Oct 30 10:00:19.254090 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:19.254044 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24" event={"ID":"18686e38-6651-454d-b9ad-996ffebe2b91","Type":"ContainerDied","Data":"c41e759d6fc5c724727ce44f55e9a2177cb97954f052d3db913c105f08f52e26"} Oct 30 10:00:19.254090 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:19.254065 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24" Oct 30 10:00:19.254365 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:19.254100 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24" event={"ID":"18686e38-6651-454d-b9ad-996ffebe2b91","Type":"ContainerDied","Data":"b56016622c9eaab17c4aa4d150152d152036d3d5ff625a1e3219cb0fd753e61f"} Oct 30 10:00:19.254365 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:19.254132 4562 scope.go:122] "RemoveContainer" containerID="c41e759d6fc5c724727ce44f55e9a2177cb97954f052d3db913c105f08f52e26" Oct 30 10:00:19.264611 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:19.264541 4562 scope.go:122] "RemoveContainer" containerID="8164ea6956edc7d2e43109737ac0f7bfbe74b625b3c510d1307756202a96688c" Oct 30 10:00:19.273205 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:19.273184 4562 scope.go:122] "RemoveContainer" containerID="c41e759d6fc5c724727ce44f55e9a2177cb97954f052d3db913c105f08f52e26" Oct 30 10:00:19.273485 ip-10-0-143-244 kubenswrapper[4562]: E1030 10:00:19.273459 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c41e759d6fc5c724727ce44f55e9a2177cb97954f052d3db913c105f08f52e26\": container with ID starting with c41e759d6fc5c724727ce44f55e9a2177cb97954f052d3db913c105f08f52e26 not found: ID does not exist" containerID="c41e759d6fc5c724727ce44f55e9a2177cb97954f052d3db913c105f08f52e26" Oct 30 10:00:19.273565 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:19.273494 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c41e759d6fc5c724727ce44f55e9a2177cb97954f052d3db913c105f08f52e26"} err="failed to get container status \"c41e759d6fc5c724727ce44f55e9a2177cb97954f052d3db913c105f08f52e26\": rpc error: code = NotFound desc = could not find container \"c41e759d6fc5c724727ce44f55e9a2177cb97954f052d3db913c105f08f52e26\": container with ID starting with c41e759d6fc5c724727ce44f55e9a2177cb97954f052d3db913c105f08f52e26 not found: ID does not exist" Oct 30 10:00:19.273565 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:19.273514 4562 scope.go:122] "RemoveContainer" containerID="8164ea6956edc7d2e43109737ac0f7bfbe74b625b3c510d1307756202a96688c" Oct 30 10:00:19.273798 ip-10-0-143-244 kubenswrapper[4562]: E1030 10:00:19.273777 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8164ea6956edc7d2e43109737ac0f7bfbe74b625b3c510d1307756202a96688c\": container with ID starting with 8164ea6956edc7d2e43109737ac0f7bfbe74b625b3c510d1307756202a96688c not found: ID does not exist" containerID="8164ea6956edc7d2e43109737ac0f7bfbe74b625b3c510d1307756202a96688c" Oct 30 10:00:19.273852 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:19.273808 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8164ea6956edc7d2e43109737ac0f7bfbe74b625b3c510d1307756202a96688c"} err="failed to get container status \"8164ea6956edc7d2e43109737ac0f7bfbe74b625b3c510d1307756202a96688c\": rpc error: code = NotFound desc = could not find container \"8164ea6956edc7d2e43109737ac0f7bfbe74b625b3c510d1307756202a96688c\": container with ID starting with 8164ea6956edc7d2e43109737ac0f7bfbe74b625b3c510d1307756202a96688c not found: ID does not exist" Oct 30 10:00:19.306265 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:19.306244 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24"] Oct 30 10:00:19.313146 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:19.313124 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-a525bd-openshift-default-68dcf45b7c-d9m24"] Oct 30 10:00:19.873998 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:19.873963 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18686e38-6651-454d-b9ad-996ffebe2b91" path="/var/lib/kubelet/pods/18686e38-6651-454d-b9ad-996ffebe2b91/volumes" Oct 30 10:00:19.874519 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:19.874501 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ed52c04-442a-4a04-8a72-c40062a5e3ba" path="/var/lib/kubelet/pods/9ed52c04-442a-4a04-8a72-c40062a5e3ba/volumes" Oct 30 10:00:19.874922 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:19.874908 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2232a56-3384-41a6-961d-ec5b4ce0db03" path="/var/lib/kubelet/pods/e2232a56-3384-41a6-961d-ec5b4ce0db03/volumes" Oct 30 10:00:19.875712 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:19.875697 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4c52d5e-0a44-479d-bfce-072508f5527a" path="/var/lib/kubelet/pods/f4c52d5e-0a44-479d-bfce-072508f5527a/volumes" Oct 30 10:00:24.643039 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:24.643003 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-f58f32-664f5dc478-zb7dn"] Oct 30 10:00:28.596229 ip-10-0-143-244 kubenswrapper[4562]: E1030 10:00:28.596194 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 10:00:29.643773 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:29.643708 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-f58f32-664f5dc478-zb7dn" podUID="83e4cd09-2917-4dcb-b67e-13b088428c02" containerName="payload-processing" containerID="cri-o://bbfcfd1be1f0ea448c958554dd4e441b92c88bba449d4dd44380e49d1f82c2e4" gracePeriod=25 Oct 30 10:00:38.598592 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:38.598542 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-f58f32-664f5dc478-zb7dn" podUID="83e4cd09-2917-4dcb-b67e-13b088428c02" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 10:00:48.225341 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:48.225262 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-8ebd65-84fd6bbf6d-6t7tb"] Oct 30 10:00:48.258304 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:48.258269 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8ebd65-7cb68f4d79-z5vx4"] Oct 30 10:00:48.597986 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:48.597896 4562 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-f58f32-664f5dc478-zb7dn" podUID="83e4cd09-2917-4dcb-b67e-13b088428c02" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 10:00:49.900338 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:49.900284 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-f58f32-664f5dc478-zb7dn" Oct 30 10:00:49.979715 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:49.979682 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83e4cd09-2917-4dcb-b67e-13b088428c02-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83e4cd09-2917-4dcb-b67e-13b088428c02-plugins-config-volume\") pod \"83e4cd09-2917-4dcb-b67e-13b088428c02\" (UID: \"83e4cd09-2917-4dcb-b67e-13b088428c02\") " Oct 30 10:00:49.979916 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:49.979740 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83e4cd09-2917-4dcb-b67e-13b088428c02-kube-api-access-5djrd\" (UniqueName: \"kubernetes.io/projected/83e4cd09-2917-4dcb-b67e-13b088428c02-kube-api-access-5djrd\") pod \"83e4cd09-2917-4dcb-b67e-13b088428c02\" (UID: \"83e4cd09-2917-4dcb-b67e-13b088428c02\") " Oct 30 10:00:49.980070 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:49.980046 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83e4cd09-2917-4dcb-b67e-13b088428c02-plugins-config-volume" pod "83e4cd09-2917-4dcb-b67e-13b088428c02" (UID: "83e4cd09-2917-4dcb-b67e-13b088428c02"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:00:49.980176 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:49.980159 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83e4cd09-2917-4dcb-b67e-13b088428c02-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:00:49.981951 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:49.981918 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83e4cd09-2917-4dcb-b67e-13b088428c02-kube-api-access-5djrd" pod "83e4cd09-2917-4dcb-b67e-13b088428c02" (UID: "83e4cd09-2917-4dcb-b67e-13b088428c02"). InnerVolumeSpecName "kube-api-access-5djrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:00:50.080845 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:50.080813 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5djrd\" (UniqueName: \"kubernetes.io/projected/83e4cd09-2917-4dcb-b67e-13b088428c02-kube-api-access-5djrd\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:00:50.446226 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:50.446193 4562 generic.go:356] "Generic (PLEG): container finished" podID="83e4cd09-2917-4dcb-b67e-13b088428c02" containerID="bbfcfd1be1f0ea448c958554dd4e441b92c88bba449d4dd44380e49d1f82c2e4" exitCode=0 Oct 30 10:00:50.446412 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:50.446248 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-f58f32-664f5dc478-zb7dn" Oct 30 10:00:50.446412 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:50.446278 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-f58f32-664f5dc478-zb7dn" event={"ID":"83e4cd09-2917-4dcb-b67e-13b088428c02","Type":"ContainerDied","Data":"bbfcfd1be1f0ea448c958554dd4e441b92c88bba449d4dd44380e49d1f82c2e4"} Oct 30 10:00:50.446412 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:50.446327 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-f58f32-664f5dc478-zb7dn" event={"ID":"83e4cd09-2917-4dcb-b67e-13b088428c02","Type":"ContainerDied","Data":"e8ae1ffb40b132fbaa6b9eaa395ef5b1562847a98945f0cd306919dc53e7d330"} Oct 30 10:00:50.446412 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:50.446352 4562 scope.go:122] "RemoveContainer" containerID="bbfcfd1be1f0ea448c958554dd4e441b92c88bba449d4dd44380e49d1f82c2e4" Oct 30 10:00:50.458841 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:50.458817 4562 scope.go:122] "RemoveContainer" containerID="bbfcfd1be1f0ea448c958554dd4e441b92c88bba449d4dd44380e49d1f82c2e4" Oct 30 10:00:50.459103 ip-10-0-143-244 kubenswrapper[4562]: E1030 10:00:50.459080 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbfcfd1be1f0ea448c958554dd4e441b92c88bba449d4dd44380e49d1f82c2e4\": container with ID starting with bbfcfd1be1f0ea448c958554dd4e441b92c88bba449d4dd44380e49d1f82c2e4 not found: ID does not exist" containerID="bbfcfd1be1f0ea448c958554dd4e441b92c88bba449d4dd44380e49d1f82c2e4" Oct 30 10:00:50.459177 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:50.459115 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbfcfd1be1f0ea448c958554dd4e441b92c88bba449d4dd44380e49d1f82c2e4"} err="failed to get container status \"bbfcfd1be1f0ea448c958554dd4e441b92c88bba449d4dd44380e49d1f82c2e4\": rpc error: code = NotFound desc = could not find container \"bbfcfd1be1f0ea448c958554dd4e441b92c88bba449d4dd44380e49d1f82c2e4\": container with ID starting with bbfcfd1be1f0ea448c958554dd4e441b92c88bba449d4dd44380e49d1f82c2e4 not found: ID does not exist" Oct 30 10:00:50.499292 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:50.499266 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-f58f32-664f5dc478-zb7dn"] Oct 30 10:00:50.512276 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:50.512242 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-f58f32-664f5dc478-zb7dn"] Oct 30 10:00:51.873893 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:51.873857 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83e4cd09-2917-4dcb-b67e-13b088428c02" path="/var/lib/kubelet/pods/83e4cd09-2917-4dcb-b67e-13b088428c02/volumes" Oct 30 10:00:53.226232 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:53.226189 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8ebd65-84fd6bbf6d-6t7tb" podUID="74fa48b5-9049-4bb8-978d-5b7773f0b93e" containerName="payload-processing" containerID="cri-o://6ceec0c628cceb12390d6fde7406af207acf002324fb1b08e54168782215119c" gracePeriod=25 Oct 30 10:00:53.258881 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:53.258838 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8ebd65-7cb68f4d79-z5vx4" podUID="aadf57f9-3206-4de6-87af-c42d14e3af5f" containerName="payload-pre-processing" containerID="cri-o://e9e03a5a690efa822a04db830e494a779dd97fddaa48b5d46d60e3e3515e5536" gracePeriod=25 Oct 30 10:00:53.468739 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:53.468698 4562 generic.go:356] "Generic (PLEG): container finished" podID="74fa48b5-9049-4bb8-978d-5b7773f0b93e" containerID="6ceec0c628cceb12390d6fde7406af207acf002324fb1b08e54168782215119c" exitCode=0 Oct 30 10:00:53.468940 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:53.468804 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8ebd65-84fd6bbf6d-6t7tb" event={"ID":"74fa48b5-9049-4bb8-978d-5b7773f0b93e","Type":"ContainerDied","Data":"6ceec0c628cceb12390d6fde7406af207acf002324fb1b08e54168782215119c"} Oct 30 10:00:53.470940 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:53.470907 4562 generic.go:356] "Generic (PLEG): container finished" podID="aadf57f9-3206-4de6-87af-c42d14e3af5f" containerID="e9e03a5a690efa822a04db830e494a779dd97fddaa48b5d46d60e3e3515e5536" exitCode=0 Oct 30 10:00:53.471079 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:53.470989 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8ebd65-7cb68f4d79-z5vx4" event={"ID":"aadf57f9-3206-4de6-87af-c42d14e3af5f","Type":"ContainerDied","Data":"e9e03a5a690efa822a04db830e494a779dd97fddaa48b5d46d60e3e3515e5536"} Oct 30 10:00:53.524736 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:53.524708 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8ebd65-84fd6bbf6d-6t7tb" Oct 30 10:00:53.552753 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:53.552730 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8ebd65-7cb68f4d79-z5vx4" Oct 30 10:00:53.611086 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:53.611051 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aadf57f9-3206-4de6-87af-c42d14e3af5f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aadf57f9-3206-4de6-87af-c42d14e3af5f-plugins-config-volume\") pod \"aadf57f9-3206-4de6-87af-c42d14e3af5f\" (UID: \"aadf57f9-3206-4de6-87af-c42d14e3af5f\") " Oct 30 10:00:53.611281 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:53.611105 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74fa48b5-9049-4bb8-978d-5b7773f0b93e-kube-api-access-7chlk\" (UniqueName: \"kubernetes.io/projected/74fa48b5-9049-4bb8-978d-5b7773f0b93e-kube-api-access-7chlk\") pod \"74fa48b5-9049-4bb8-978d-5b7773f0b93e\" (UID: \"74fa48b5-9049-4bb8-978d-5b7773f0b93e\") " Oct 30 10:00:53.611281 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:53.611132 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aadf57f9-3206-4de6-87af-c42d14e3af5f-kube-api-access-vggbw\" (UniqueName: \"kubernetes.io/projected/aadf57f9-3206-4de6-87af-c42d14e3af5f-kube-api-access-vggbw\") pod \"aadf57f9-3206-4de6-87af-c42d14e3af5f\" (UID: \"aadf57f9-3206-4de6-87af-c42d14e3af5f\") " Oct 30 10:00:53.612132 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:53.611446 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74fa48b5-9049-4bb8-978d-5b7773f0b93e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74fa48b5-9049-4bb8-978d-5b7773f0b93e-plugins-config-volume\") pod \"74fa48b5-9049-4bb8-978d-5b7773f0b93e\" (UID: \"74fa48b5-9049-4bb8-978d-5b7773f0b93e\") " Oct 30 10:00:53.612132 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:53.611526 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aadf57f9-3206-4de6-87af-c42d14e3af5f-plugins-config-volume" pod "aadf57f9-3206-4de6-87af-c42d14e3af5f" (UID: "aadf57f9-3206-4de6-87af-c42d14e3af5f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:00:53.612132 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:53.611877 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74fa48b5-9049-4bb8-978d-5b7773f0b93e-plugins-config-volume" pod "74fa48b5-9049-4bb8-978d-5b7773f0b93e" (UID: "74fa48b5-9049-4bb8-978d-5b7773f0b93e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:00:53.612132 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:53.612095 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aadf57f9-3206-4de6-87af-c42d14e3af5f-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:00:53.612132 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:53.612112 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74fa48b5-9049-4bb8-978d-5b7773f0b93e-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:00:53.614623 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:53.614595 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aadf57f9-3206-4de6-87af-c42d14e3af5f-kube-api-access-vggbw" pod "aadf57f9-3206-4de6-87af-c42d14e3af5f" (UID: "aadf57f9-3206-4de6-87af-c42d14e3af5f"). InnerVolumeSpecName "kube-api-access-vggbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:00:53.616220 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:53.616199 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74fa48b5-9049-4bb8-978d-5b7773f0b93e-kube-api-access-7chlk" pod "74fa48b5-9049-4bb8-978d-5b7773f0b93e" (UID: "74fa48b5-9049-4bb8-978d-5b7773f0b93e"). InnerVolumeSpecName "kube-api-access-7chlk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:00:53.713063 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:53.713015 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7chlk\" (UniqueName: \"kubernetes.io/projected/74fa48b5-9049-4bb8-978d-5b7773f0b93e-kube-api-access-7chlk\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:00:53.713063 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:53.713058 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vggbw\" (UniqueName: \"kubernetes.io/projected/aadf57f9-3206-4de6-87af-c42d14e3af5f-kube-api-access-vggbw\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:00:54.478830 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:54.478788 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8ebd65-84fd6bbf6d-6t7tb" Oct 30 10:00:54.478830 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:54.478814 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8ebd65-84fd6bbf6d-6t7tb" event={"ID":"74fa48b5-9049-4bb8-978d-5b7773f0b93e","Type":"ContainerDied","Data":"e78f61201edb971d9e84d2ac59f64eae21e244402a958ca455013cff4a42e30e"} Oct 30 10:00:54.479378 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:54.478886 4562 scope.go:122] "RemoveContainer" containerID="6ceec0c628cceb12390d6fde7406af207acf002324fb1b08e54168782215119c" Oct 30 10:00:54.480353 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:54.480325 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8ebd65-7cb68f4d79-z5vx4" Oct 30 10:00:54.480492 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:54.480364 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8ebd65-7cb68f4d79-z5vx4" event={"ID":"aadf57f9-3206-4de6-87af-c42d14e3af5f","Type":"ContainerDied","Data":"e8d5e752e24e8cd94d2436ba000e3ae8c1cf998e6a2e46e3c70b75493bc2f853"} Oct 30 10:00:54.488742 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:54.488716 4562 scope.go:122] "RemoveContainer" containerID="e9e03a5a690efa822a04db830e494a779dd97fddaa48b5d46d60e3e3515e5536" Oct 30 10:00:54.535847 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:54.535826 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-8ebd65-84fd6bbf6d-6t7tb"] Oct 30 10:00:54.539827 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:54.539799 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-8ebd65-84fd6bbf6d-6t7tb"] Oct 30 10:00:54.572036 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:54.572010 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8ebd65-7cb68f4d79-z5vx4"] Oct 30 10:00:54.573724 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:54.573704 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8ebd65-7cb68f4d79-z5vx4"] Oct 30 10:00:55.874577 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:55.874545 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74fa48b5-9049-4bb8-978d-5b7773f0b93e" path="/var/lib/kubelet/pods/74fa48b5-9049-4bb8-978d-5b7773f0b93e/volumes" Oct 30 10:00:55.874985 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:00:55.874944 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aadf57f9-3206-4de6-87af-c42d14e3af5f" path="/var/lib/kubelet/pods/aadf57f9-3206-4de6-87af-c42d14e3af5f/volumes" Oct 30 10:00:58.596048 ip-10-0-143-244 kubenswrapper[4562]: E1030 10:00:58.596011 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 10:01:13.221154 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:13.221117 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-388653-576db78cbd-t8ncz"] Oct 30 10:01:13.255203 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:13.255165 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-388653-56dfb7cb47-8th58"] Oct 30 10:01:13.271257 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:13.271227 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-388653-6f7cf8fd57-686xr"] Oct 30 10:01:13.271512 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:13.271486 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-388653-6f7cf8fd57-686xr" podUID="14eb89e0-861c-41a9-a245-e97adfa39f25" containerName="maas-api" containerID="cri-o://d9df37c272b79b292f23c7b3a4d2873d358422e4f8bc84e32245c65a941709a4" gracePeriod=30 Oct 30 10:01:13.511942 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:13.511915 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-388653-6f7cf8fd57-686xr" Oct 30 10:01:13.579318 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:13.579274 4562 generic.go:356] "Generic (PLEG): container finished" podID="14eb89e0-861c-41a9-a245-e97adfa39f25" containerID="d9df37c272b79b292f23c7b3a4d2873d358422e4f8bc84e32245c65a941709a4" exitCode=0 Oct 30 10:01:13.579523 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:13.579329 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-388653-6f7cf8fd57-686xr" Oct 30 10:01:13.579523 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:13.579366 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-388653-6f7cf8fd57-686xr" event={"ID":"14eb89e0-861c-41a9-a245-e97adfa39f25","Type":"ContainerDied","Data":"d9df37c272b79b292f23c7b3a4d2873d358422e4f8bc84e32245c65a941709a4"} Oct 30 10:01:13.579523 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:13.579488 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-388653-6f7cf8fd57-686xr" event={"ID":"14eb89e0-861c-41a9-a245-e97adfa39f25","Type":"ContainerDied","Data":"5e01a245c66f377d95cea92de964f5d17b6b51e4a7926034353c1588de0a9369"} Oct 30 10:01:13.579653 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:13.579523 4562 scope.go:122] "RemoveContainer" containerID="d9df37c272b79b292f23c7b3a4d2873d358422e4f8bc84e32245c65a941709a4" Oct 30 10:01:13.590029 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:13.590005 4562 scope.go:122] "RemoveContainer" containerID="d9df37c272b79b292f23c7b3a4d2873d358422e4f8bc84e32245c65a941709a4" Oct 30 10:01:13.590324 ip-10-0-143-244 kubenswrapper[4562]: E1030 10:01:13.590302 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9df37c272b79b292f23c7b3a4d2873d358422e4f8bc84e32245c65a941709a4\": container with ID starting with d9df37c272b79b292f23c7b3a4d2873d358422e4f8bc84e32245c65a941709a4 not found: ID does not exist" containerID="d9df37c272b79b292f23c7b3a4d2873d358422e4f8bc84e32245c65a941709a4" Oct 30 10:01:13.590434 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:13.590335 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9df37c272b79b292f23c7b3a4d2873d358422e4f8bc84e32245c65a941709a4"} err="failed to get container status \"d9df37c272b79b292f23c7b3a4d2873d358422e4f8bc84e32245c65a941709a4\": rpc error: code = NotFound desc = could not find container \"d9df37c272b79b292f23c7b3a4d2873d358422e4f8bc84e32245c65a941709a4\": container with ID starting with d9df37c272b79b292f23c7b3a4d2873d358422e4f8bc84e32245c65a941709a4 not found: ID does not exist" Oct 30 10:01:13.615255 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:13.615232 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14eb89e0-861c-41a9-a245-e97adfa39f25-kube-api-access-zvz7g\" (UniqueName: \"kubernetes.io/projected/14eb89e0-861c-41a9-a245-e97adfa39f25-kube-api-access-zvz7g\") pod \"14eb89e0-861c-41a9-a245-e97adfa39f25\" (UID: \"14eb89e0-861c-41a9-a245-e97adfa39f25\") " Oct 30 10:01:13.615410 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:13.615271 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14eb89e0-861c-41a9-a245-e97adfa39f25-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14eb89e0-861c-41a9-a245-e97adfa39f25-maas-api-tls\") pod \"14eb89e0-861c-41a9-a245-e97adfa39f25\" (UID: \"14eb89e0-861c-41a9-a245-e97adfa39f25\") " Oct 30 10:01:13.615410 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:13.615338 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14eb89e0-861c-41a9-a245-e97adfa39f25-metrics-certs\" (UniqueName: \"kubernetes.io/secret/14eb89e0-861c-41a9-a245-e97adfa39f25-metrics-certs\") pod \"14eb89e0-861c-41a9-a245-e97adfa39f25\" (UID: \"14eb89e0-861c-41a9-a245-e97adfa39f25\") " Oct 30 10:01:13.615537 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:13.615418 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14eb89e0-861c-41a9-a245-e97adfa39f25-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14eb89e0-861c-41a9-a245-e97adfa39f25-openshift-service-ca\") pod \"14eb89e0-861c-41a9-a245-e97adfa39f25\" (UID: \"14eb89e0-861c-41a9-a245-e97adfa39f25\") " Oct 30 10:01:13.615845 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:13.615820 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14eb89e0-861c-41a9-a245-e97adfa39f25-openshift-service-ca" pod "14eb89e0-861c-41a9-a245-e97adfa39f25" (UID: "14eb89e0-861c-41a9-a245-e97adfa39f25"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:01:13.617334 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:13.617314 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14eb89e0-861c-41a9-a245-e97adfa39f25-maas-api-tls" pod "14eb89e0-861c-41a9-a245-e97adfa39f25" (UID: "14eb89e0-861c-41a9-a245-e97adfa39f25"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:01:13.617463 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:13.617412 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14eb89e0-861c-41a9-a245-e97adfa39f25-metrics-certs" pod "14eb89e0-861c-41a9-a245-e97adfa39f25" (UID: "14eb89e0-861c-41a9-a245-e97adfa39f25"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:01:13.617537 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:13.617517 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14eb89e0-861c-41a9-a245-e97adfa39f25-kube-api-access-zvz7g" pod "14eb89e0-861c-41a9-a245-e97adfa39f25" (UID: "14eb89e0-861c-41a9-a245-e97adfa39f25"). InnerVolumeSpecName "kube-api-access-zvz7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:01:13.716581 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:13.716544 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvz7g\" (UniqueName: \"kubernetes.io/projected/14eb89e0-861c-41a9-a245-e97adfa39f25-kube-api-access-zvz7g\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:01:13.716581 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:13.716580 4562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14eb89e0-861c-41a9-a245-e97adfa39f25-maas-api-tls\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:01:13.716827 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:13.716593 4562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14eb89e0-861c-41a9-a245-e97adfa39f25-metrics-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:01:13.716827 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:13.716606 4562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14eb89e0-861c-41a9-a245-e97adfa39f25-openshift-service-ca\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:01:13.919651 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:13.919619 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-388653-6f7cf8fd57-686xr"] Oct 30 10:01:13.922687 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:13.922661 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-388653-6f7cf8fd57-686xr"] Oct 30 10:01:15.874075 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:15.874037 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14eb89e0-861c-41a9-a245-e97adfa39f25" path="/var/lib/kubelet/pods/14eb89e0-861c-41a9-a245-e97adfa39f25/volumes" Oct 30 10:01:18.255489 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:18.255434 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-388653-56dfb7cb47-8th58" podUID="23911a6c-f976-496a-b75d-759bbc201b44" containerName="payload-pre-processing" containerID="cri-o://030fb21e03d183dbc0933944c3b21947a9e1db492d2614dffe0db0bf02a16ae8" gracePeriod=25 Oct 30 10:01:18.507237 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:18.507179 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-388653-56dfb7cb47-8th58" Oct 30 10:01:18.579907 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:18.579865 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-388653-576db78cbd-t8ncz" podUID="f367c180-d0c3-4266-ba10-d982197e5bd8" containerName="payload-processing" containerID="cri-o://ccdde6a697f8d77a059a68bddbb243cac687f59b6e2d3b284bac520075869962" gracePeriod=25 Oct 30 10:01:18.606494 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:18.606463 4562 generic.go:356] "Generic (PLEG): container finished" podID="23911a6c-f976-496a-b75d-759bbc201b44" containerID="030fb21e03d183dbc0933944c3b21947a9e1db492d2614dffe0db0bf02a16ae8" exitCode=0 Oct 30 10:01:18.606634 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:18.606514 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-388653-56dfb7cb47-8th58" Oct 30 10:01:18.606634 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:18.606518 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-388653-56dfb7cb47-8th58" event={"ID":"23911a6c-f976-496a-b75d-759bbc201b44","Type":"ContainerDied","Data":"030fb21e03d183dbc0933944c3b21947a9e1db492d2614dffe0db0bf02a16ae8"} Oct 30 10:01:18.606757 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:18.606661 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-388653-56dfb7cb47-8th58" event={"ID":"23911a6c-f976-496a-b75d-759bbc201b44","Type":"ContainerDied","Data":"c8f1f707bb9a4eb63c17400649217c7dbda04645c33c78b87297071ad62f7fce"} Oct 30 10:01:18.606757 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:18.606682 4562 scope.go:122] "RemoveContainer" containerID="030fb21e03d183dbc0933944c3b21947a9e1db492d2614dffe0db0bf02a16ae8" Oct 30 10:01:18.665490 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:18.665444 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23911a6c-f976-496a-b75d-759bbc201b44-kube-api-access-9m6k7\" (UniqueName: \"kubernetes.io/projected/23911a6c-f976-496a-b75d-759bbc201b44-kube-api-access-9m6k7\") pod \"23911a6c-f976-496a-b75d-759bbc201b44\" (UID: \"23911a6c-f976-496a-b75d-759bbc201b44\") " Oct 30 10:01:18.665654 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:18.665530 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23911a6c-f976-496a-b75d-759bbc201b44-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23911a6c-f976-496a-b75d-759bbc201b44-plugins-config-volume\") pod \"23911a6c-f976-496a-b75d-759bbc201b44\" (UID: \"23911a6c-f976-496a-b75d-759bbc201b44\") " Oct 30 10:01:18.665925 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:18.665903 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23911a6c-f976-496a-b75d-759bbc201b44-plugins-config-volume" pod "23911a6c-f976-496a-b75d-759bbc201b44" (UID: "23911a6c-f976-496a-b75d-759bbc201b44"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:01:18.667693 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:18.667665 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23911a6c-f976-496a-b75d-759bbc201b44-kube-api-access-9m6k7" pod "23911a6c-f976-496a-b75d-759bbc201b44" (UID: "23911a6c-f976-496a-b75d-759bbc201b44"). InnerVolumeSpecName "kube-api-access-9m6k7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:01:18.690886 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:18.690862 4562 scope.go:122] "RemoveContainer" containerID="030fb21e03d183dbc0933944c3b21947a9e1db492d2614dffe0db0bf02a16ae8" Oct 30 10:01:18.691268 ip-10-0-143-244 kubenswrapper[4562]: E1030 10:01:18.691246 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"030fb21e03d183dbc0933944c3b21947a9e1db492d2614dffe0db0bf02a16ae8\": container with ID starting with 030fb21e03d183dbc0933944c3b21947a9e1db492d2614dffe0db0bf02a16ae8 not found: ID does not exist" containerID="030fb21e03d183dbc0933944c3b21947a9e1db492d2614dffe0db0bf02a16ae8" Oct 30 10:01:18.691328 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:18.691278 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"030fb21e03d183dbc0933944c3b21947a9e1db492d2614dffe0db0bf02a16ae8"} err="failed to get container status \"030fb21e03d183dbc0933944c3b21947a9e1db492d2614dffe0db0bf02a16ae8\": rpc error: code = NotFound desc = could not find container \"030fb21e03d183dbc0933944c3b21947a9e1db492d2614dffe0db0bf02a16ae8\": container with ID starting with 030fb21e03d183dbc0933944c3b21947a9e1db492d2614dffe0db0bf02a16ae8 not found: ID does not exist" Oct 30 10:01:18.767109 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:18.767036 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9m6k7\" (UniqueName: \"kubernetes.io/projected/23911a6c-f976-496a-b75d-759bbc201b44-kube-api-access-9m6k7\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:01:18.767109 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:18.767065 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23911a6c-f976-496a-b75d-759bbc201b44-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:01:18.828933 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:18.828908 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-388653-576db78cbd-t8ncz" Oct 30 10:01:18.952693 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:18.952657 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-388653-56dfb7cb47-8th58"] Oct 30 10:01:18.955459 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:18.955436 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-388653-56dfb7cb47-8th58"] Oct 30 10:01:18.968815 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:18.968790 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f367c180-d0c3-4266-ba10-d982197e5bd8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f367c180-d0c3-4266-ba10-d982197e5bd8-plugins-config-volume\") pod \"f367c180-d0c3-4266-ba10-d982197e5bd8\" (UID: \"f367c180-d0c3-4266-ba10-d982197e5bd8\") " Oct 30 10:01:18.968944 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:18.968850 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f367c180-d0c3-4266-ba10-d982197e5bd8-kube-api-access-5t4n4\" (UniqueName: \"kubernetes.io/projected/f367c180-d0c3-4266-ba10-d982197e5bd8-kube-api-access-5t4n4\") pod \"f367c180-d0c3-4266-ba10-d982197e5bd8\" (UID: \"f367c180-d0c3-4266-ba10-d982197e5bd8\") " Oct 30 10:01:18.969211 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:18.969189 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f367c180-d0c3-4266-ba10-d982197e5bd8-plugins-config-volume" pod "f367c180-d0c3-4266-ba10-d982197e5bd8" (UID: "f367c180-d0c3-4266-ba10-d982197e5bd8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:01:18.970917 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:18.970894 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f367c180-d0c3-4266-ba10-d982197e5bd8-kube-api-access-5t4n4" pod "f367c180-d0c3-4266-ba10-d982197e5bd8" (UID: "f367c180-d0c3-4266-ba10-d982197e5bd8"). InnerVolumeSpecName "kube-api-access-5t4n4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:01:19.070140 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:19.070061 4562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f367c180-d0c3-4266-ba10-d982197e5bd8-plugins-config-volume\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:01:19.070140 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:19.070098 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5t4n4\" (UniqueName: \"kubernetes.io/projected/f367c180-d0c3-4266-ba10-d982197e5bd8-kube-api-access-5t4n4\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:01:19.612612 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:19.612574 4562 generic.go:356] "Generic (PLEG): container finished" podID="f367c180-d0c3-4266-ba10-d982197e5bd8" containerID="ccdde6a697f8d77a059a68bddbb243cac687f59b6e2d3b284bac520075869962" exitCode=0 Oct 30 10:01:19.613109 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:19.612625 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-388653-576db78cbd-t8ncz" Oct 30 10:01:19.613109 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:19.612654 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-388653-576db78cbd-t8ncz" event={"ID":"f367c180-d0c3-4266-ba10-d982197e5bd8","Type":"ContainerDied","Data":"ccdde6a697f8d77a059a68bddbb243cac687f59b6e2d3b284bac520075869962"} Oct 30 10:01:19.613109 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:19.612713 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-388653-576db78cbd-t8ncz" event={"ID":"f367c180-d0c3-4266-ba10-d982197e5bd8","Type":"ContainerDied","Data":"bc1dfbf406d99b72894318beb1f3332d7e451f0b5184f93ee1b1718b4441019f"} Oct 30 10:01:19.613109 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:19.612737 4562 scope.go:122] "RemoveContainer" containerID="ccdde6a697f8d77a059a68bddbb243cac687f59b6e2d3b284bac520075869962" Oct 30 10:01:19.622226 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:19.622204 4562 scope.go:122] "RemoveContainer" containerID="ccdde6a697f8d77a059a68bddbb243cac687f59b6e2d3b284bac520075869962" Oct 30 10:01:19.622507 ip-10-0-143-244 kubenswrapper[4562]: E1030 10:01:19.622482 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccdde6a697f8d77a059a68bddbb243cac687f59b6e2d3b284bac520075869962\": container with ID starting with ccdde6a697f8d77a059a68bddbb243cac687f59b6e2d3b284bac520075869962 not found: ID does not exist" containerID="ccdde6a697f8d77a059a68bddbb243cac687f59b6e2d3b284bac520075869962" Oct 30 10:01:19.622566 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:19.622518 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccdde6a697f8d77a059a68bddbb243cac687f59b6e2d3b284bac520075869962"} err="failed to get container status \"ccdde6a697f8d77a059a68bddbb243cac687f59b6e2d3b284bac520075869962\": rpc error: code = NotFound desc = could not find container \"ccdde6a697f8d77a059a68bddbb243cac687f59b6e2d3b284bac520075869962\": container with ID starting with ccdde6a697f8d77a059a68bddbb243cac687f59b6e2d3b284bac520075869962 not found: ID does not exist" Oct 30 10:01:19.669091 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:19.669057 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-388653-576db78cbd-t8ncz"] Oct 30 10:01:19.673537 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:19.673511 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-388653-576db78cbd-t8ncz"] Oct 30 10:01:19.872783 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:19.872708 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23911a6c-f976-496a-b75d-759bbc201b44" path="/var/lib/kubelet/pods/23911a6c-f976-496a-b75d-759bbc201b44/volumes" Oct 30 10:01:19.873116 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:19.873103 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f367c180-d0c3-4266-ba10-d982197e5bd8" path="/var/lib/kubelet/pods/f367c180-d0c3-4266-ba10-d982197e5bd8/volumes" Oct 30 10:01:26.782756 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:26.782716 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2f69c8/distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb"] Oct 30 10:01:28.595296 ip-10-0-143-244 kubenswrapper[4562]: E1030 10:01:28.595259 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 10:01:32.538829 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:32.538787 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d8fb6cb-khkg9"] Oct 30 10:01:32.539336 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:32.539315 4562 cpu_manager.go:414] "containerMap: removing container" podUID="f4c52d5e-0a44-479d-bfce-072508f5527a" containerName="istio-proxy" Oct 30 10:01:32.539421 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:32.539341 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4c52d5e-0a44-479d-bfce-072508f5527a" containerName="istio-proxy" Oct 30 10:01:32.539421 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:32.539363 4562 cpu_manager.go:414] "containerMap: removing container" podUID="cc6684d6-8c9b-4a9d-b2bc-9f825424b546" containerName="cleanup" Oct 30 10:01:32.539421 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:32.539372 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc6684d6-8c9b-4a9d-b2bc-9f825424b546" containerName="cleanup" Oct 30 10:01:32.539421 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:32.539397 4562 cpu_manager.go:414] "containerMap: removing container" podUID="03ef2570-c3b8-48c9-9632-de3c09ea41dc" containerName="cleanup" Oct 30 10:01:32.539421 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:32.539405 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03ef2570-c3b8-48c9-9632-de3c09ea41dc" containerName="cleanup" Oct 30 10:01:32.539421 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:32.539420 4562 cpu_manager.go:414] "containerMap: removing container" podUID="e2232a56-3384-41a6-961d-ec5b4ce0db03" containerName="cleanup" Oct 30 10:01:32.539660 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:32.539426 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2232a56-3384-41a6-961d-ec5b4ce0db03" containerName="cleanup" Oct 30 10:01:32.539660 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:32.539437 4562 cpu_manager.go:414] "containerMap: removing container" podUID="30a13eb2-b7e5-42ff-b5e9-993a20d588aa" containerName="payload-processing" Oct 30 10:01:32.539660 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:32.539444 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30a13eb2-b7e5-42ff-b5e9-993a20d588aa" containerName="payload-processing" Oct 30 10:01:32.539660 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:32.539459 4562 cpu_manager.go:414] "containerMap: removing container" podUID="9ed52c04-442a-4a04-8a72-c40062a5e3ba" containerName="istio-proxy" Oct 30 10:01:32.539660 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:32.539466 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ed52c04-442a-4a04-8a72-c40062a5e3ba" containerName="istio-proxy" Oct 30 10:01:32.539660 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:32.539475 4562 cpu_manager.go:414] "containerMap: removing container" podUID="14eb89e0-861c-41a9-a245-e97adfa39f25" containerName="maas-api" Oct 30 10:01:32.539660 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:32.539483 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14eb89e0-861c-41a9-a245-e97adfa39f25" containerName="maas-api" Oct 30 10:01:32.539660 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:32.539497 4562 cpu_manager.go:414] "containerMap: removing container" podUID="f367c180-d0c3-4266-ba10-d982197e5bd8" containerName="payload-processing" Oct 30 10:01:32.539660 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:32.539504 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f367c180-d0c3-4266-ba10-d982197e5bd8" containerName="payload-processing" Oct 30 10:01:32.539660 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:32.539516 4562 cpu_manager.go:414] "containerMap: removing container" podUID="e2232a56-3384-41a6-961d-ec5b4ce0db03" containerName="cleanup" Oct 30 10:01:32.539660 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:32.539522 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2232a56-3384-41a6-961d-ec5b4ce0db03" containerName="cleanup" Oct 30 10:01:32.539660 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:32.539533 4562 cpu_manager.go:414] "containerMap: removing container" podUID="23911a6c-f976-496a-b75d-759bbc201b44" containerName="payload-pre-processing" Oct 30 10:01:32.539660 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:32.539540 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23911a6c-f976-496a-b75d-759bbc201b44" containerName="payload-pre-processing" Oct 30 10:01:32.539660 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:32.539552 4562 cpu_manager.go:414] "containerMap: removing container" podUID="cc6684d6-8c9b-4a9d-b2bc-9f825424b546" containerName="cleanup" Oct 30 10:01:32.539660 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:32.539559 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc6684d6-8c9b-4a9d-b2bc-9f825424b546" containerName="cleanup" Oct 30 10:01:32.539660 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:32.539568 4562 cpu_manager.go:414] "containerMap: removing container" podUID="e2232a56-3384-41a6-961d-ec5b4ce0db03" containerName="cleanup" Oct 30 10:01:32.539660 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:32.539575 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2232a56-3384-41a6-961d-ec5b4ce0db03" containerName="cleanup" Oct 30 10:01:32.539660 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:32.539590 4562 cpu_manager.go:414] "containerMap: removing container" podUID="74fa48b5-9049-4bb8-978d-5b7773f0b93e" containerName="payload-processing" Oct 30 10:01:32.539660 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:32.539597 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74fa48b5-9049-4bb8-978d-5b7773f0b93e" containerName="payload-processing" Oct 30 10:01:32.539660 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:32.539625 4562 cpu_manager.go:414] "containerMap: removing container" podUID="83e4cd09-2917-4dcb-b67e-13b088428c02" containerName="payload-processing" Oct 30 10:01:32.539660 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:32.539632 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83e4cd09-2917-4dcb-b67e-13b088428c02" containerName="payload-processing" Oct 30 10:01:32.539660 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:32.539645 4562 cpu_manager.go:414] "containerMap: removing container" podUID="18686e38-6651-454d-b9ad-996ffebe2b91" containerName="istio-proxy" Oct 30 10:01:32.539660 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:32.539651 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18686e38-6651-454d-b9ad-996ffebe2b91" containerName="istio-proxy" Oct 30 10:01:32.539660 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:32.539662 4562 cpu_manager.go:414] "containerMap: removing container" podUID="d1195ec8-c914-4d63-aca9-85e8b74b5d44" containerName="payload-pre-processing" Oct 30 10:01:32.539660 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:32.539669 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1195ec8-c914-4d63-aca9-85e8b74b5d44" containerName="payload-pre-processing" Oct 30 10:01:32.540662 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:32.539681 4562 cpu_manager.go:414] "containerMap: removing container" podUID="aadf57f9-3206-4de6-87af-c42d14e3af5f" containerName="payload-pre-processing" Oct 30 10:01:32.540662 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:32.539688 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aadf57f9-3206-4de6-87af-c42d14e3af5f" containerName="payload-pre-processing" Oct 30 10:01:32.540662 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:32.539794 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc6684d6-8c9b-4a9d-b2bc-9f825424b546" containerName="cleanup" Oct 30 10:01:32.540662 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:32.539806 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="03ef2570-c3b8-48c9-9632-de3c09ea41dc" containerName="cleanup" Oct 30 10:01:32.540662 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:32.539815 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2232a56-3384-41a6-961d-ec5b4ce0db03" containerName="cleanup" Oct 30 10:01:32.540662 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:32.539825 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="30a13eb2-b7e5-42ff-b5e9-993a20d588aa" containerName="payload-processing" Oct 30 10:01:32.540662 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:32.539835 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="03ef2570-c3b8-48c9-9632-de3c09ea41dc" containerName="cleanup" Oct 30 10:01:32.540662 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:32.539845 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ed52c04-442a-4a04-8a72-c40062a5e3ba" containerName="istio-proxy" Oct 30 10:01:32.540662 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:32.539855 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="14eb89e0-861c-41a9-a245-e97adfa39f25" containerName="maas-api" Oct 30 10:01:32.540662 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:32.539863 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="f367c180-d0c3-4266-ba10-d982197e5bd8" containerName="payload-processing" Oct 30 10:01:32.540662 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:32.539872 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2232a56-3384-41a6-961d-ec5b4ce0db03" containerName="cleanup" Oct 30 10:01:32.540662 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:32.539880 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="23911a6c-f976-496a-b75d-759bbc201b44" containerName="payload-pre-processing" Oct 30 10:01:32.540662 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:32.539887 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2232a56-3384-41a6-961d-ec5b4ce0db03" containerName="cleanup" Oct 30 10:01:32.540662 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:32.539897 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="74fa48b5-9049-4bb8-978d-5b7773f0b93e" containerName="payload-processing" Oct 30 10:01:32.540662 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:32.539909 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="83e4cd09-2917-4dcb-b67e-13b088428c02" containerName="payload-processing" Oct 30 10:01:32.540662 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:32.539918 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="18686e38-6651-454d-b9ad-996ffebe2b91" containerName="istio-proxy" Oct 30 10:01:32.540662 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:32.539928 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="18686e38-6651-454d-b9ad-996ffebe2b91" containerName="istio-proxy" Oct 30 10:01:32.540662 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:32.539936 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1195ec8-c914-4d63-aca9-85e8b74b5d44" containerName="payload-pre-processing" Oct 30 10:01:32.540662 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:32.539944 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="aadf57f9-3206-4de6-87af-c42d14e3af5f" containerName="payload-pre-processing" Oct 30 10:01:32.540662 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:32.539952 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4c52d5e-0a44-479d-bfce-072508f5527a" containerName="istio-proxy" Oct 30 10:01:32.545019 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:32.544994 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8fb6cb-khkg9" Oct 30 10:01:32.562708 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:32.562679 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d8fb6cb-khkg9"] Oct 30 10:01:32.605046 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:32.605005 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwfzx\" (UniqueName: \"kubernetes.io/projected/00a51d31-d280-4fda-af37-82e5b63dfde4-kube-api-access-wwfzx\") pod \"maas-api-revoke-sub-3d8fb6cb-khkg9\" (UID: \"00a51d31-d280-4fda-af37-82e5b63dfde4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8fb6cb-khkg9" Oct 30 10:01:32.605227 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:32.605069 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00a51d31-d280-4fda-af37-82e5b63dfde4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d8fb6cb-khkg9\" (UID: \"00a51d31-d280-4fda-af37-82e5b63dfde4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8fb6cb-khkg9" Oct 30 10:01:32.605334 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:32.605250 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00a51d31-d280-4fda-af37-82e5b63dfde4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d8fb6cb-khkg9\" (UID: \"00a51d31-d280-4fda-af37-82e5b63dfde4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8fb6cb-khkg9" Oct 30 10:01:32.636518 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:32.636486 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 10:01:32.636681 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:32.636519 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf9gd\"" Oct 30 10:01:32.706621 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:32.706581 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00a51d31-d280-4fda-af37-82e5b63dfde4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d8fb6cb-khkg9\" (UID: \"00a51d31-d280-4fda-af37-82e5b63dfde4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8fb6cb-khkg9" Oct 30 10:01:32.706848 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:32.706663 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwfzx\" (UniqueName: \"kubernetes.io/projected/00a51d31-d280-4fda-af37-82e5b63dfde4-kube-api-access-wwfzx\") pod \"maas-api-revoke-sub-3d8fb6cb-khkg9\" (UID: \"00a51d31-d280-4fda-af37-82e5b63dfde4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8fb6cb-khkg9" Oct 30 10:01:32.706848 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:32.706711 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00a51d31-d280-4fda-af37-82e5b63dfde4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d8fb6cb-khkg9\" (UID: \"00a51d31-d280-4fda-af37-82e5b63dfde4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8fb6cb-khkg9" Oct 30 10:01:32.707352 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:32.707328 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00a51d31-d280-4fda-af37-82e5b63dfde4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d8fb6cb-khkg9\" (UID: \"00a51d31-d280-4fda-af37-82e5b63dfde4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8fb6cb-khkg9" Oct 30 10:01:32.708992 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:32.708964 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00a51d31-d280-4fda-af37-82e5b63dfde4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d8fb6cb-khkg9\" (UID: \"00a51d31-d280-4fda-af37-82e5b63dfde4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8fb6cb-khkg9" Oct 30 10:01:32.726580 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:32.726554 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwfzx\" (UniqueName: \"kubernetes.io/projected/00a51d31-d280-4fda-af37-82e5b63dfde4-kube-api-access-wwfzx\") pod \"maas-api-revoke-sub-3d8fb6cb-khkg9\" (UID: \"00a51d31-d280-4fda-af37-82e5b63dfde4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8fb6cb-khkg9" Oct 30 10:01:32.860522 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:32.860440 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8fb6cb-khkg9" Oct 30 10:01:33.157994 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:33.157951 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d8fb6cb-khkg9"] Oct 30 10:01:33.162591 ip-10-0-143-244 kubenswrapper[4562]: W1030 10:01:33.162546 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00a51d31_d280_4fda_af37_82e5b63dfde4.slice/crio-e9896886eb1ad6f860ae5f370a64f67a540a3276a70f99b1fe97b60144fcb383 WatchSource:0}: Error finding container e9896886eb1ad6f860ae5f370a64f67a540a3276a70f99b1fe97b60144fcb383: Status 404 returned error can't find the container with id e9896886eb1ad6f860ae5f370a64f67a540a3276a70f99b1fe97b60144fcb383 Oct 30 10:01:33.688575 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:33.688535 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8fb6cb-khkg9" event={"ID":"00a51d31-d280-4fda-af37-82e5b63dfde4","Type":"ContainerStarted","Data":"1fb5b7a2742053c9cf44837177b233cf4ed7bd1a5cba27de43d85c82a718327b"} Oct 30 10:01:33.688962 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:33.688593 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8fb6cb-khkg9" event={"ID":"00a51d31-d280-4fda-af37-82e5b63dfde4","Type":"ContainerStarted","Data":"e9896886eb1ad6f860ae5f370a64f67a540a3276a70f99b1fe97b60144fcb383"} Oct 30 10:01:33.729773 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:33.729682 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8fb6cb-khkg9" podStartSLOduration=1.729669262 podStartE2EDuration="1.729669262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 10:01:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:01:33.7275776 +0000 UTC m=+4053.989138589" watchObservedRunningTime="2026-10-30 10:01:33.729669262 +0000 UTC m=+4053.991230250" Oct 30 10:01:34.696306 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:34.696273 4562 generic.go:356] "Generic (PLEG): container finished" podID="00a51d31-d280-4fda-af37-82e5b63dfde4" containerID="1fb5b7a2742053c9cf44837177b233cf4ed7bd1a5cba27de43d85c82a718327b" exitCode=0 Oct 30 10:01:34.696745 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:34.696356 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8fb6cb-khkg9" event={"ID":"00a51d31-d280-4fda-af37-82e5b63dfde4","Type":"ContainerDied","Data":"1fb5b7a2742053c9cf44837177b233cf4ed7bd1a5cba27de43d85c82a718327b"} Oct 30 10:01:35.841130 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:35.841103 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8fb6cb-khkg9" Oct 30 10:01:35.939737 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:35.939705 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00a51d31-d280-4fda-af37-82e5b63dfde4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00a51d31-d280-4fda-af37-82e5b63dfde4-maas-api-openshift-ca\") pod \"00a51d31-d280-4fda-af37-82e5b63dfde4\" (UID: \"00a51d31-d280-4fda-af37-82e5b63dfde4\") " Oct 30 10:01:35.939939 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:35.939801 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00a51d31-d280-4fda-af37-82e5b63dfde4-kube-api-access-wwfzx\" (UniqueName: \"kubernetes.io/projected/00a51d31-d280-4fda-af37-82e5b63dfde4-kube-api-access-wwfzx\") pod \"00a51d31-d280-4fda-af37-82e5b63dfde4\" (UID: \"00a51d31-d280-4fda-af37-82e5b63dfde4\") " Oct 30 10:01:35.939939 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:35.939864 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00a51d31-d280-4fda-af37-82e5b63dfde4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00a51d31-d280-4fda-af37-82e5b63dfde4-maas-api-xks-ca\") pod \"00a51d31-d280-4fda-af37-82e5b63dfde4\" (UID: \"00a51d31-d280-4fda-af37-82e5b63dfde4\") " Oct 30 10:01:35.940151 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:35.940122 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00a51d31-d280-4fda-af37-82e5b63dfde4-maas-api-openshift-ca" pod "00a51d31-d280-4fda-af37-82e5b63dfde4" (UID: "00a51d31-d280-4fda-af37-82e5b63dfde4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:01:35.940406 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:35.940234 4562 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00a51d31-d280-4fda-af37-82e5b63dfde4-maas-api-openshift-ca\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:01:35.942064 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:35.942038 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00a51d31-d280-4fda-af37-82e5b63dfde4-maas-api-xks-ca" pod "00a51d31-d280-4fda-af37-82e5b63dfde4" (UID: "00a51d31-d280-4fda-af37-82e5b63dfde4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:01:35.942294 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:35.942269 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00a51d31-d280-4fda-af37-82e5b63dfde4-kube-api-access-wwfzx" pod "00a51d31-d280-4fda-af37-82e5b63dfde4" (UID: "00a51d31-d280-4fda-af37-82e5b63dfde4"). InnerVolumeSpecName "kube-api-access-wwfzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:01:36.041246 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:36.041158 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwfzx\" (UniqueName: \"kubernetes.io/projected/00a51d31-d280-4fda-af37-82e5b63dfde4-kube-api-access-wwfzx\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:01:36.041246 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:36.041200 4562 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00a51d31-d280-4fda-af37-82e5b63dfde4-maas-api-xks-ca\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:01:36.710323 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:36.710285 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8fb6cb-khkg9" Oct 30 10:01:36.710494 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:36.710282 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8fb6cb-khkg9" event={"ID":"00a51d31-d280-4fda-af37-82e5b63dfde4","Type":"ContainerDied","Data":"e9896886eb1ad6f860ae5f370a64f67a540a3276a70f99b1fe97b60144fcb383"} Oct 30 10:01:36.710494 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:36.710449 4562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9896886eb1ad6f860ae5f370a64f67a540a3276a70f99b1fe97b60144fcb383" Oct 30 10:01:41.803146 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:41.803079 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb" podUID="a540c86f-c856-42a8-9f73-b5b9b690f42b" containerName="main" containerID="cri-o://fb0a8053bc154633938bee2dec78a3cf6dd626c8791aa8210b83a8dca615fde8" gracePeriod=45 Oct 30 10:01:42.052679 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:42.052653 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb" Oct 30 10:01:42.102161 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:42.102080 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a540c86f-c856-42a8-9f73-b5b9b690f42b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a540c86f-c856-42a8-9f73-b5b9b690f42b-tmp-dir\") pod \"a540c86f-c856-42a8-9f73-b5b9b690f42b\" (UID: \"a540c86f-c856-42a8-9f73-b5b9b690f42b\") " Oct 30 10:01:42.102161 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:42.102118 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a540c86f-c856-42a8-9f73-b5b9b690f42b-home\" (UniqueName: \"kubernetes.io/empty-dir/a540c86f-c856-42a8-9f73-b5b9b690f42b-home\") pod \"a540c86f-c856-42a8-9f73-b5b9b690f42b\" (UID: \"a540c86f-c856-42a8-9f73-b5b9b690f42b\") " Oct 30 10:01:42.102437 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:42.102171 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a540c86f-c856-42a8-9f73-b5b9b690f42b-tls-certs\" (UniqueName: \"kubernetes.io/secret/a540c86f-c856-42a8-9f73-b5b9b690f42b-tls-certs\") pod \"a540c86f-c856-42a8-9f73-b5b9b690f42b\" (UID: \"a540c86f-c856-42a8-9f73-b5b9b690f42b\") " Oct 30 10:01:42.102437 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:42.102232 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a540c86f-c856-42a8-9f73-b5b9b690f42b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a540c86f-c856-42a8-9f73-b5b9b690f42b-model-cache\") pod \"a540c86f-c856-42a8-9f73-b5b9b690f42b\" (UID: \"a540c86f-c856-42a8-9f73-b5b9b690f42b\") " Oct 30 10:01:42.102437 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:42.102304 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a540c86f-c856-42a8-9f73-b5b9b690f42b-kube-api-access-tg69t\" (UniqueName: \"kubernetes.io/projected/a540c86f-c856-42a8-9f73-b5b9b690f42b-kube-api-access-tg69t\") pod \"a540c86f-c856-42a8-9f73-b5b9b690f42b\" (UID: \"a540c86f-c856-42a8-9f73-b5b9b690f42b\") " Oct 30 10:01:42.102437 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:42.102341 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a540c86f-c856-42a8-9f73-b5b9b690f42b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a540c86f-c856-42a8-9f73-b5b9b690f42b-dshm\") pod \"a540c86f-c856-42a8-9f73-b5b9b690f42b\" (UID: \"a540c86f-c856-42a8-9f73-b5b9b690f42b\") " Oct 30 10:01:42.102437 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:42.102398 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a540c86f-c856-42a8-9f73-b5b9b690f42b-tmp-dir" pod "a540c86f-c856-42a8-9f73-b5b9b690f42b" (UID: "a540c86f-c856-42a8-9f73-b5b9b690f42b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:01:42.102437 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:42.102411 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a540c86f-c856-42a8-9f73-b5b9b690f42b-home" pod "a540c86f-c856-42a8-9f73-b5b9b690f42b" (UID: "a540c86f-c856-42a8-9f73-b5b9b690f42b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:01:42.102806 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:42.102609 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a540c86f-c856-42a8-9f73-b5b9b690f42b-model-cache" pod "a540c86f-c856-42a8-9f73-b5b9b690f42b" (UID: "a540c86f-c856-42a8-9f73-b5b9b690f42b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:01:42.102806 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:42.102698 4562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a540c86f-c856-42a8-9f73-b5b9b690f42b-tmp-dir\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:01:42.102806 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:42.102718 4562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a540c86f-c856-42a8-9f73-b5b9b690f42b-home\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:01:42.102806 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:42.102731 4562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a540c86f-c856-42a8-9f73-b5b9b690f42b-model-cache\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:01:42.104925 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:42.104898 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a540c86f-c856-42a8-9f73-b5b9b690f42b-tls-certs" pod "a540c86f-c856-42a8-9f73-b5b9b690f42b" (UID: "a540c86f-c856-42a8-9f73-b5b9b690f42b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:01:42.105052 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:42.105030 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a540c86f-c856-42a8-9f73-b5b9b690f42b-kube-api-access-tg69t" pod "a540c86f-c856-42a8-9f73-b5b9b690f42b" (UID: "a540c86f-c856-42a8-9f73-b5b9b690f42b"). InnerVolumeSpecName "kube-api-access-tg69t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:01:42.105218 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:42.105191 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a540c86f-c856-42a8-9f73-b5b9b690f42b-dshm" pod "a540c86f-c856-42a8-9f73-b5b9b690f42b" (UID: "a540c86f-c856-42a8-9f73-b5b9b690f42b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:01:42.203747 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:42.203704 4562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a540c86f-c856-42a8-9f73-b5b9b690f42b-dshm\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:01:42.203747 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:42.203745 4562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a540c86f-c856-42a8-9f73-b5b9b690f42b-tls-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:01:42.203977 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:42.203762 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg69t\" (UniqueName: \"kubernetes.io/projected/a540c86f-c856-42a8-9f73-b5b9b690f42b-kube-api-access-tg69t\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:01:42.742730 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:42.742695 4562 generic.go:356] "Generic (PLEG): container finished" podID="a540c86f-c856-42a8-9f73-b5b9b690f42b" containerID="fb0a8053bc154633938bee2dec78a3cf6dd626c8791aa8210b83a8dca615fde8" exitCode=0 Oct 30 10:01:42.742914 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:42.742740 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb" Oct 30 10:01:42.742914 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:42.742783 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb" event={"ID":"a540c86f-c856-42a8-9f73-b5b9b690f42b","Type":"ContainerDied","Data":"fb0a8053bc154633938bee2dec78a3cf6dd626c8791aa8210b83a8dca615fde8"} Oct 30 10:01:42.742914 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:42.742827 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2f69c8/distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb" event={"ID":"a540c86f-c856-42a8-9f73-b5b9b690f42b","Type":"ContainerDied","Data":"a0751bfaaaaa91b93af34962cbcebb50d7dcf847be062dfc665b211bef2818fb"} Oct 30 10:01:42.742914 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:42.742845 4562 scope.go:122] "RemoveContainer" containerID="fb0a8053bc154633938bee2dec78a3cf6dd626c8791aa8210b83a8dca615fde8" Oct 30 10:01:42.752548 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:42.752528 4562 scope.go:122] "RemoveContainer" containerID="fb0a8053bc154633938bee2dec78a3cf6dd626c8791aa8210b83a8dca615fde8" Oct 30 10:01:42.752786 ip-10-0-143-244 kubenswrapper[4562]: E1030 10:01:42.752766 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb0a8053bc154633938bee2dec78a3cf6dd626c8791aa8210b83a8dca615fde8\": container with ID starting with fb0a8053bc154633938bee2dec78a3cf6dd626c8791aa8210b83a8dca615fde8 not found: ID does not exist" containerID="fb0a8053bc154633938bee2dec78a3cf6dd626c8791aa8210b83a8dca615fde8" Oct 30 10:01:42.752861 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:42.752798 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb0a8053bc154633938bee2dec78a3cf6dd626c8791aa8210b83a8dca615fde8"} err="failed to get container status \"fb0a8053bc154633938bee2dec78a3cf6dd626c8791aa8210b83a8dca615fde8\": rpc error: code = NotFound desc = could not find container \"fb0a8053bc154633938bee2dec78a3cf6dd626c8791aa8210b83a8dca615fde8\": container with ID starting with fb0a8053bc154633938bee2dec78a3cf6dd626c8791aa8210b83a8dca615fde8 not found: ID does not exist" Oct 30 10:01:42.796756 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:42.796720 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2f69c8/distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb"] Oct 30 10:01:42.807201 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:42.807173 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-2f69c8/distinct-2-w0-2f69c8-kserve-7949d95c56-gnbvb"] Oct 30 10:01:42.928131 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:42.928095 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b8mzp"] Oct 30 10:01:42.928463 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:42.928404 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b8mzp" podUID="5ef70489-cb47-4bd5-844b-72e81331b648" containerName="istio-proxy" containerID="cri-o://fd08ce487619a41ccfa624e059fe2da212bf1be0fe10b9dec9f9941716b2ef8c" gracePeriod=30 Oct 30 10:01:43.872989 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:43.872955 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a540c86f-c856-42a8-9f73-b5b9b690f42b" path="/var/lib/kubelet/pods/a540c86f-c856-42a8-9f73-b5b9b690f42b/volumes" Oct 30 10:01:47.621535 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:47.621494 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-2f69c8-5dbb84f778-2gp27"] Oct 30 10:01:47.622044 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:47.621736 4562 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2f69c8-5dbb84f778-2gp27" podUID="48d959d9-0791-48dd-84e9-78da9baaaea3" containerName="maas-api" containerID="cri-o://9b7b2f1ec9845637e9ea3d56f608542e3a06e9569434daa701710bb2b9c15ad8" gracePeriod=30 Oct 30 10:01:47.771157 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:47.771123 4562 generic.go:356] "Generic (PLEG): container finished" podID="48d959d9-0791-48dd-84e9-78da9baaaea3" containerID="9b7b2f1ec9845637e9ea3d56f608542e3a06e9569434daa701710bb2b9c15ad8" exitCode=0 Oct 30 10:01:47.771318 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:47.771203 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2f69c8-5dbb84f778-2gp27" event={"ID":"48d959d9-0791-48dd-84e9-78da9baaaea3","Type":"ContainerDied","Data":"9b7b2f1ec9845637e9ea3d56f608542e3a06e9569434daa701710bb2b9c15ad8"} Oct 30 10:01:47.885106 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:47.885083 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2f69c8-5dbb84f778-2gp27" Oct 30 10:01:47.964030 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:47.964002 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48d959d9-0791-48dd-84e9-78da9baaaea3-kube-api-access-4wxqz\" (UniqueName: \"kubernetes.io/projected/48d959d9-0791-48dd-84e9-78da9baaaea3-kube-api-access-4wxqz\") pod \"48d959d9-0791-48dd-84e9-78da9baaaea3\" (UID: \"48d959d9-0791-48dd-84e9-78da9baaaea3\") " Oct 30 10:01:47.964139 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:47.964048 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48d959d9-0791-48dd-84e9-78da9baaaea3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/48d959d9-0791-48dd-84e9-78da9baaaea3-metrics-certs\") pod \"48d959d9-0791-48dd-84e9-78da9baaaea3\" (UID: \"48d959d9-0791-48dd-84e9-78da9baaaea3\") " Oct 30 10:01:47.964139 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:47.964130 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48d959d9-0791-48dd-84e9-78da9baaaea3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48d959d9-0791-48dd-84e9-78da9baaaea3-maas-api-tls\") pod \"48d959d9-0791-48dd-84e9-78da9baaaea3\" (UID: \"48d959d9-0791-48dd-84e9-78da9baaaea3\") " Oct 30 10:01:47.964217 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:47.964196 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48d959d9-0791-48dd-84e9-78da9baaaea3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48d959d9-0791-48dd-84e9-78da9baaaea3-openshift-service-ca\") pod \"48d959d9-0791-48dd-84e9-78da9baaaea3\" (UID: \"48d959d9-0791-48dd-84e9-78da9baaaea3\") " Oct 30 10:01:47.964677 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:47.964644 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48d959d9-0791-48dd-84e9-78da9baaaea3-openshift-service-ca" pod "48d959d9-0791-48dd-84e9-78da9baaaea3" (UID: "48d959d9-0791-48dd-84e9-78da9baaaea3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:01:47.966185 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:47.966161 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48d959d9-0791-48dd-84e9-78da9baaaea3-maas-api-tls" pod "48d959d9-0791-48dd-84e9-78da9baaaea3" (UID: "48d959d9-0791-48dd-84e9-78da9baaaea3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:01:47.966268 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:47.966203 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48d959d9-0791-48dd-84e9-78da9baaaea3-metrics-certs" pod "48d959d9-0791-48dd-84e9-78da9baaaea3" (UID: "48d959d9-0791-48dd-84e9-78da9baaaea3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:01:47.966268 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:47.966238 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48d959d9-0791-48dd-84e9-78da9baaaea3-kube-api-access-4wxqz" pod "48d959d9-0791-48dd-84e9-78da9baaaea3" (UID: "48d959d9-0791-48dd-84e9-78da9baaaea3"). InnerVolumeSpecName "kube-api-access-4wxqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:01:48.066036 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:48.066004 4562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48d959d9-0791-48dd-84e9-78da9baaaea3-openshift-service-ca\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:01:48.066036 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:48.066036 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wxqz\" (UniqueName: \"kubernetes.io/projected/48d959d9-0791-48dd-84e9-78da9baaaea3-kube-api-access-4wxqz\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:01:48.066252 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:48.066050 4562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48d959d9-0791-48dd-84e9-78da9baaaea3-metrics-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:01:48.066252 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:48.066063 4562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48d959d9-0791-48dd-84e9-78da9baaaea3-maas-api-tls\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:01:48.181126 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:48.181102 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b8mzp" Oct 30 10:01:48.267595 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:48.267559 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ef70489-cb47-4bd5-844b-72e81331b648-kube-api-access-wfmt2\" (UniqueName: \"kubernetes.io/projected/5ef70489-cb47-4bd5-844b-72e81331b648-kube-api-access-wfmt2\") pod \"5ef70489-cb47-4bd5-844b-72e81331b648\" (UID: \"5ef70489-cb47-4bd5-844b-72e81331b648\") " Oct 30 10:01:48.267818 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:48.267637 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ef70489-cb47-4bd5-844b-72e81331b648-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ef70489-cb47-4bd5-844b-72e81331b648-workload-certs\") pod \"5ef70489-cb47-4bd5-844b-72e81331b648\" (UID: \"5ef70489-cb47-4bd5-844b-72e81331b648\") " Oct 30 10:01:48.267818 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:48.267675 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ef70489-cb47-4bd5-844b-72e81331b648-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ef70489-cb47-4bd5-844b-72e81331b648-istio-data\") pod \"5ef70489-cb47-4bd5-844b-72e81331b648\" (UID: \"5ef70489-cb47-4bd5-844b-72e81331b648\") " Oct 30 10:01:48.267818 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:48.267706 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ef70489-cb47-4bd5-844b-72e81331b648-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ef70489-cb47-4bd5-844b-72e81331b648-istio-envoy\") pod \"5ef70489-cb47-4bd5-844b-72e81331b648\" (UID: \"5ef70489-cb47-4bd5-844b-72e81331b648\") " Oct 30 10:01:48.267818 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:48.267738 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ef70489-cb47-4bd5-844b-72e81331b648-istio-token\" (UniqueName: \"kubernetes.io/projected/5ef70489-cb47-4bd5-844b-72e81331b648-istio-token\") pod \"5ef70489-cb47-4bd5-844b-72e81331b648\" (UID: \"5ef70489-cb47-4bd5-844b-72e81331b648\") " Oct 30 10:01:48.267818 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:48.267799 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ef70489-cb47-4bd5-844b-72e81331b648-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef70489-cb47-4bd5-844b-72e81331b648-workload-socket\") pod \"5ef70489-cb47-4bd5-844b-72e81331b648\" (UID: \"5ef70489-cb47-4bd5-844b-72e81331b648\") " Oct 30 10:01:48.268115 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:48.267873 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ef70489-cb47-4bd5-844b-72e81331b648-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ef70489-cb47-4bd5-844b-72e81331b648-istio-podinfo\") pod \"5ef70489-cb47-4bd5-844b-72e81331b648\" (UID: \"5ef70489-cb47-4bd5-844b-72e81331b648\") " Oct 30 10:01:48.268115 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:48.267914 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ef70489-cb47-4bd5-844b-72e81331b648-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef70489-cb47-4bd5-844b-72e81331b648-credential-socket\") pod \"5ef70489-cb47-4bd5-844b-72e81331b648\" (UID: \"5ef70489-cb47-4bd5-844b-72e81331b648\") " Oct 30 10:01:48.268115 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:48.267961 4562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ef70489-cb47-4bd5-844b-72e81331b648-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ef70489-cb47-4bd5-844b-72e81331b648-istiod-ca-cert\") pod \"5ef70489-cb47-4bd5-844b-72e81331b648\" (UID: \"5ef70489-cb47-4bd5-844b-72e81331b648\") " Oct 30 10:01:48.268115 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:48.267988 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ef70489-cb47-4bd5-844b-72e81331b648-workload-certs" pod "5ef70489-cb47-4bd5-844b-72e81331b648" (UID: "5ef70489-cb47-4bd5-844b-72e81331b648"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:01:48.268115 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:48.268000 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ef70489-cb47-4bd5-844b-72e81331b648-istio-data" pod "5ef70489-cb47-4bd5-844b-72e81331b648" (UID: "5ef70489-cb47-4bd5-844b-72e81331b648"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:01:48.268416 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:48.268281 4562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ef70489-cb47-4bd5-844b-72e81331b648-workload-certs\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:01:48.268416 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:48.268302 4562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ef70489-cb47-4bd5-844b-72e81331b648-istio-data\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:01:48.268416 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:48.268356 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ef70489-cb47-4bd5-844b-72e81331b648-workload-socket" pod "5ef70489-cb47-4bd5-844b-72e81331b648" (UID: "5ef70489-cb47-4bd5-844b-72e81331b648"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:01:48.268610 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:48.268445 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ef70489-cb47-4bd5-844b-72e81331b648-credential-socket" pod "5ef70489-cb47-4bd5-844b-72e81331b648" (UID: "5ef70489-cb47-4bd5-844b-72e81331b648"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:01:48.268670 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:48.268635 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ef70489-cb47-4bd5-844b-72e81331b648-istiod-ca-cert" pod "5ef70489-cb47-4bd5-844b-72e81331b648" (UID: "5ef70489-cb47-4bd5-844b-72e81331b648"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:01:48.270092 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:48.270072 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ef70489-cb47-4bd5-844b-72e81331b648-kube-api-access-wfmt2" pod "5ef70489-cb47-4bd5-844b-72e81331b648" (UID: "5ef70489-cb47-4bd5-844b-72e81331b648"). InnerVolumeSpecName "kube-api-access-wfmt2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:01:48.270300 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:48.270274 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ef70489-cb47-4bd5-844b-72e81331b648-istio-token" pod "5ef70489-cb47-4bd5-844b-72e81331b648" (UID: "5ef70489-cb47-4bd5-844b-72e81331b648"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:01:48.270377 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:48.270361 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ef70489-cb47-4bd5-844b-72e81331b648-istio-envoy" pod "5ef70489-cb47-4bd5-844b-72e81331b648" (UID: "5ef70489-cb47-4bd5-844b-72e81331b648"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:01:48.270546 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:48.270519 4562 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ef70489-cb47-4bd5-844b-72e81331b648-istio-podinfo" pod "5ef70489-cb47-4bd5-844b-72e81331b648" (UID: "5ef70489-cb47-4bd5-844b-72e81331b648"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 10:01:48.369194 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:48.369154 4562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ef70489-cb47-4bd5-844b-72e81331b648-istiod-ca-cert\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:01:48.369194 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:48.369192 4562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfmt2\" (UniqueName: \"kubernetes.io/projected/5ef70489-cb47-4bd5-844b-72e81331b648-kube-api-access-wfmt2\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:01:48.369467 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:48.369208 4562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ef70489-cb47-4bd5-844b-72e81331b648-istio-envoy\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:01:48.369467 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:48.369220 4562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ef70489-cb47-4bd5-844b-72e81331b648-istio-token\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:01:48.369467 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:48.369234 4562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef70489-cb47-4bd5-844b-72e81331b648-workload-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:01:48.369467 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:48.369249 4562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ef70489-cb47-4bd5-844b-72e81331b648-istio-podinfo\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:01:48.369467 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:48.369262 4562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef70489-cb47-4bd5-844b-72e81331b648-credential-socket\") on node \"ip-10-0-143-244.ec2.internal\" DevicePath \"\"" Oct 30 10:01:48.778001 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:48.777966 4562 generic.go:356] "Generic (PLEG): container finished" podID="5ef70489-cb47-4bd5-844b-72e81331b648" containerID="fd08ce487619a41ccfa624e059fe2da212bf1be0fe10b9dec9f9941716b2ef8c" exitCode=0 Oct 30 10:01:48.778483 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:48.778017 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b8mzp" Oct 30 10:01:48.778483 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:48.778044 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b8mzp" event={"ID":"5ef70489-cb47-4bd5-844b-72e81331b648","Type":"ContainerDied","Data":"fd08ce487619a41ccfa624e059fe2da212bf1be0fe10b9dec9f9941716b2ef8c"} Oct 30 10:01:48.778483 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:48.778115 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b8mzp" event={"ID":"5ef70489-cb47-4bd5-844b-72e81331b648","Type":"ContainerDied","Data":"2d342b1c8fdce455ead6ff1ca1f714a6ae561b20a287b7dc3520f540b6313622"} Oct 30 10:01:48.778483 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:48.778144 4562 scope.go:122] "RemoveContainer" containerID="fd08ce487619a41ccfa624e059fe2da212bf1be0fe10b9dec9f9941716b2ef8c" Oct 30 10:01:48.779538 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:48.779500 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2f69c8-5dbb84f778-2gp27" event={"ID":"48d959d9-0791-48dd-84e9-78da9baaaea3","Type":"ContainerDied","Data":"db9b6ded7fc4139b39d74185fd654f2b355605376c721d5e0479eed48b513409"} Oct 30 10:01:48.779600 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:48.779542 4562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2f69c8-5dbb84f778-2gp27" Oct 30 10:01:48.789353 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:48.789332 4562 scope.go:122] "RemoveContainer" containerID="fd08ce487619a41ccfa624e059fe2da212bf1be0fe10b9dec9f9941716b2ef8c" Oct 30 10:01:48.789728 ip-10-0-143-244 kubenswrapper[4562]: E1030 10:01:48.789706 4562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd08ce487619a41ccfa624e059fe2da212bf1be0fe10b9dec9f9941716b2ef8c\": container with ID starting with fd08ce487619a41ccfa624e059fe2da212bf1be0fe10b9dec9f9941716b2ef8c not found: ID does not exist" containerID="fd08ce487619a41ccfa624e059fe2da212bf1be0fe10b9dec9f9941716b2ef8c" Oct 30 10:01:48.789827 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:48.789734 4562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd08ce487619a41ccfa624e059fe2da212bf1be0fe10b9dec9f9941716b2ef8c"} err="failed to get container status \"fd08ce487619a41ccfa624e059fe2da212bf1be0fe10b9dec9f9941716b2ef8c\": rpc error: code = NotFound desc = could not find container \"fd08ce487619a41ccfa624e059fe2da212bf1be0fe10b9dec9f9941716b2ef8c\": container with ID starting with fd08ce487619a41ccfa624e059fe2da212bf1be0fe10b9dec9f9941716b2ef8c not found: ID does not exist" Oct 30 10:01:48.789827 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:48.789754 4562 scope.go:122] "RemoveContainer" containerID="9b7b2f1ec9845637e9ea3d56f608542e3a06e9569434daa701710bb2b9c15ad8" Oct 30 10:01:48.843624 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:48.843593 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b8mzp"] Oct 30 10:01:48.856592 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:48.856565 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b8mzp"] Oct 30 10:01:48.890631 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:48.890555 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-2f69c8-5dbb84f778-2gp27"] Oct 30 10:01:48.896147 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:48.896121 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-2f69c8-5dbb84f778-2gp27"] Oct 30 10:01:49.872310 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:49.872276 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48d959d9-0791-48dd-84e9-78da9baaaea3" path="/var/lib/kubelet/pods/48d959d9-0791-48dd-84e9-78da9baaaea3/volumes" Oct 30 10:01:49.872770 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:49.872755 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ef70489-cb47-4bd5-844b-72e81331b648" path="/var/lib/kubelet/pods/5ef70489-cb47-4bd5-844b-72e81331b648/volumes" Oct 30 10:01:55.807294 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:55.807263 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-74d66bdc9c-wxffc_664fe1ef-e0ec-4bc3-afe1-9eac486c4ac4/authorino/0.log" Oct 30 10:01:58.492492 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:58.492460 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-5bbbc667d4-sdbtx_fd98de9c-4d7f-4f22-b042-2867781cf88c/manager/0.log" Oct 30 10:01:58.594476 ip-10-0-143-244 kubenswrapper[4562]: E1030 10:01:58.594441 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 10:01:59.651798 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:01:59.651761 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-687fd9c8d8-gt7tl_625507b9-d06d-4071-b33a-b959a6e3129a/manager/0.log" Oct 30 10:02:00.257866 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:00.257837 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-gv64t_a9a30ca0-995e-4e77-bf92-b3923ecc14c0/proxy/0.log" Oct 30 10:02:01.015699 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:01.015661 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-74d66bdc9c-wxffc_664fe1ef-e0ec-4bc3-afe1-9eac486c4ac4/authorino/0.log" Oct 30 10:02:11.905405 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:11.905356 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-5bbbc667d4-sdbtx_fd98de9c-4d7f-4f22-b042-2867781cf88c/manager/0.log" Oct 30 10:02:13.080118 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:13.080084 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-687fd9c8d8-gt7tl_625507b9-d06d-4071-b33a-b959a6e3129a/manager/0.log" Oct 30 10:02:13.669451 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:13.669419 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-gv64t_a9a30ca0-995e-4e77-bf92-b3923ecc14c0/proxy/0.log" Oct 30 10:02:14.699015 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:14.698976 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-2459ab93-vwz95_efa60339-43cc-408f-8b23-1160c89f9477/revoke-keys/0.log" Oct 30 10:02:15.094641 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:15.094570 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3d8fb6cb-khkg9_00a51d31-d280-4fda-af37-82e5b63dfde4/revoke-keys/0.log" Oct 30 10:02:15.310488 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:15.310456 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-555bd640-9nfpl_007dfc94-fbc7-456d-8760-2235a48f4277/revoke-keys/0.log" Oct 30 10:02:16.332761 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:16.332724 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-da7a0f7c-4rnj7_c0caf9cd-3ae5-4d1f-9815-71b2310cf896/revoke-keys/0.log" Oct 30 10:02:16.712021 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:16.711992 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-jc4mv_6276e9c0-0ed3-4eb3-83b2-0d64a4e3763b/postgres/0.log" Oct 30 10:02:18.344488 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:18.344461 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-74d66bdc9c-wxffc_664fe1ef-e0ec-4bc3-afe1-9eac486c4ac4/authorino/0.log" Oct 30 10:02:21.342478 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:21.342447 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-nkb2b_3bc88a93-dfe9-4dad-b1a7-76d39693168c/discovery/0.log" Oct 30 10:02:21.749568 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:21.749522 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-767d5cbf5-vx9wj_993358dc-1271-474e-b372-5e30c394340d/kube-auth-proxy/0.log" Oct 30 10:02:21.953199 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:21.953171 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-dwcq7_cb668c39-39ae-443f-9648-c9d5cc7e2d2e/istio-proxy/0.log" Oct 30 10:02:22.154511 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:22.154437 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-hl5zf_7b74384a-1957-4071-b9aa-e4f2c1e9cef0/istio-proxy/0.log" Oct 30 10:02:23.188431 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:23.188397 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-l6sj2_076fc963-efa1-46c0-ae7f-3773c788ed00/payload-pre-processing/0.log" Oct 30 10:02:23.398231 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:23.398196 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-85rhg_2f5e6dbb-9b81-4734-ba2a-5c6a677e49e8/payload-processing/0.log" Oct 30 10:02:24.620143 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:24.620109 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-g5jjc_1449660d-c4f4-49da-b916-52dd67a92dcf/main/0.log" Oct 30 10:02:24.823935 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:24.823902 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-zt6jf_697547af-60fc-4555-ac54-37c087f649e8/main/0.log" Oct 30 10:02:25.018167 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:25.018134 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-2h599_bce5ba79-e37c-4ff8-9b95-b2d27bdd3a6d/main/0.log" Oct 30 10:02:25.438021 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:25.437915 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88xq7t_0893a22f-fb50-42c5-af68-af12521387f1/main/0.log" Oct 30 10:02:25.629734 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:25.629701 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-jbjpj_cfc464d1-6f5c-4ab5-8b5a-41e439035213/main/0.log" Oct 30 10:02:28.595705 ip-10-0-143-244 kubenswrapper[4562]: E1030 10:02:28.595667 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 10:02:32.032130 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:32.032098 4562 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da7a0f7c-4rnj7"] Oct 30 10:02:32.044687 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:32.044660 4562 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da7a0f7c-4rnj7"] Oct 30 10:02:33.873231 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:33.873195 4562 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0caf9cd-3ae5-4d1f-9815-71b2310cf896" path="/var/lib/kubelet/pods/c0caf9cd-3ae5-4d1f-9815-71b2310cf896/volumes" Oct 30 10:02:36.456667 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:36.456621 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6llqt/must-gather-l5jrm"] Oct 30 10:02:36.457254 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:36.457233 4562 cpu_manager.go:414] "containerMap: removing container" podUID="18686e38-6651-454d-b9ad-996ffebe2b91" containerName="istio-proxy" Oct 30 10:02:36.457346 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:36.457255 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18686e38-6651-454d-b9ad-996ffebe2b91" containerName="istio-proxy" Oct 30 10:02:36.457346 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:36.457277 4562 cpu_manager.go:414] "containerMap: removing container" podUID="5ef70489-cb47-4bd5-844b-72e81331b648" containerName="istio-proxy" Oct 30 10:02:36.457346 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:36.457285 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ef70489-cb47-4bd5-844b-72e81331b648" containerName="istio-proxy" Oct 30 10:02:36.457346 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:36.457304 4562 cpu_manager.go:414] "containerMap: removing container" podUID="03ef2570-c3b8-48c9-9632-de3c09ea41dc" containerName="cleanup" Oct 30 10:02:36.457346 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:36.457312 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03ef2570-c3b8-48c9-9632-de3c09ea41dc" containerName="cleanup" Oct 30 10:02:36.457346 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:36.457328 4562 cpu_manager.go:414] "containerMap: removing container" podUID="a540c86f-c856-42a8-9f73-b5b9b690f42b" containerName="main" Oct 30 10:02:36.457346 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:36.457335 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a540c86f-c856-42a8-9f73-b5b9b690f42b" containerName="main" Oct 30 10:02:36.457346 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:36.457347 4562 cpu_manager.go:414] "containerMap: removing container" podUID="00a51d31-d280-4fda-af37-82e5b63dfde4" containerName="revoke-keys" Oct 30 10:02:36.457820 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:36.457355 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00a51d31-d280-4fda-af37-82e5b63dfde4" containerName="revoke-keys" Oct 30 10:02:36.457820 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:36.457376 4562 cpu_manager.go:414] "containerMap: removing container" podUID="48d959d9-0791-48dd-84e9-78da9baaaea3" containerName="maas-api" Oct 30 10:02:36.457820 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:36.457398 4562 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48d959d9-0791-48dd-84e9-78da9baaaea3" containerName="maas-api" Oct 30 10:02:36.457820 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:36.457520 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="48d959d9-0791-48dd-84e9-78da9baaaea3" containerName="maas-api" Oct 30 10:02:36.457820 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:36.457542 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ef70489-cb47-4bd5-844b-72e81331b648" containerName="istio-proxy" Oct 30 10:02:36.457820 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:36.457554 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc6684d6-8c9b-4a9d-b2bc-9f825424b546" containerName="cleanup" Oct 30 10:02:36.457820 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:36.457564 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="a540c86f-c856-42a8-9f73-b5b9b690f42b" containerName="main" Oct 30 10:02:36.457820 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:36.457574 4562 memory_manager.go:365] "RemoveStaleState removing state" podUID="00a51d31-d280-4fda-af37-82e5b63dfde4" containerName="revoke-keys" Oct 30 10:02:36.461052 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:36.461031 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6llqt/must-gather-l5jrm" Oct 30 10:02:36.477771 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:36.477712 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6llqt/must-gather-l5jrm"] Oct 30 10:02:36.480438 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:36.480413 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6llqt\"/\"openshift-service-ca.crt\"" Oct 30 10:02:36.480568 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:36.480498 4562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6llqt\"/\"kube-root-ca.crt\"" Oct 30 10:02:36.488736 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:36.488715 4562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-6llqt\"/\"default-dockercfg-tfjr2\"" Oct 30 10:02:36.599686 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:36.599651 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlgh4\" (UniqueName: \"kubernetes.io/projected/a21cd284-322d-4d9c-98b8-11079be94d94-kube-api-access-qlgh4\") pod \"must-gather-l5jrm\" (UID: \"a21cd284-322d-4d9c-98b8-11079be94d94\") " pod="openshift-must-gather-6llqt/must-gather-l5jrm" Oct 30 10:02:36.599871 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:36.599699 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a21cd284-322d-4d9c-98b8-11079be94d94-must-gather-output\") pod \"must-gather-l5jrm\" (UID: \"a21cd284-322d-4d9c-98b8-11079be94d94\") " pod="openshift-must-gather-6llqt/must-gather-l5jrm" Oct 30 10:02:36.700261 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:36.700221 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlgh4\" (UniqueName: \"kubernetes.io/projected/a21cd284-322d-4d9c-98b8-11079be94d94-kube-api-access-qlgh4\") pod \"must-gather-l5jrm\" (UID: \"a21cd284-322d-4d9c-98b8-11079be94d94\") " pod="openshift-must-gather-6llqt/must-gather-l5jrm" Oct 30 10:02:36.700485 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:36.700275 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a21cd284-322d-4d9c-98b8-11079be94d94-must-gather-output\") pod \"must-gather-l5jrm\" (UID: \"a21cd284-322d-4d9c-98b8-11079be94d94\") " pod="openshift-must-gather-6llqt/must-gather-l5jrm" Oct 30 10:02:36.700669 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:36.700647 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a21cd284-322d-4d9c-98b8-11079be94d94-must-gather-output\") pod \"must-gather-l5jrm\" (UID: \"a21cd284-322d-4d9c-98b8-11079be94d94\") " pod="openshift-must-gather-6llqt/must-gather-l5jrm" Oct 30 10:02:36.720211 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:36.720149 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlgh4\" (UniqueName: \"kubernetes.io/projected/a21cd284-322d-4d9c-98b8-11079be94d94-kube-api-access-qlgh4\") pod \"must-gather-l5jrm\" (UID: \"a21cd284-322d-4d9c-98b8-11079be94d94\") " pod="openshift-must-gather-6llqt/must-gather-l5jrm" Oct 30 10:02:36.775198 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:36.775164 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6llqt/must-gather-l5jrm" Oct 30 10:02:37.069110 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:37.069076 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6llqt/must-gather-l5jrm"] Oct 30 10:02:37.069249 ip-10-0-143-244 kubenswrapper[4562]: W1030 10:02:37.069152 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda21cd284_322d_4d9c_98b8_11079be94d94.slice/crio-803c7ac209b6a9c5f88b4b70599241a789d7ce9ed6702157c72021514b189149 WatchSource:0}: Error finding container 803c7ac209b6a9c5f88b4b70599241a789d7ce9ed6702157c72021514b189149: Status 404 returned error can't find the container with id 803c7ac209b6a9c5f88b4b70599241a789d7ce9ed6702157c72021514b189149 Oct 30 10:02:37.071828 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:37.071808 4562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 10:02:38.016230 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:38.016192 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6llqt/must-gather-l5jrm" event={"ID":"a21cd284-322d-4d9c-98b8-11079be94d94","Type":"ContainerStarted","Data":"bf8b8ef021916282f3a6fb6eebad26ef1d2ccf62857d0bb51d3cdb50d2ab804a"} Oct 30 10:02:38.016578 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:38.016252 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6llqt/must-gather-l5jrm" event={"ID":"a21cd284-322d-4d9c-98b8-11079be94d94","Type":"ContainerStarted","Data":"803c7ac209b6a9c5f88b4b70599241a789d7ce9ed6702157c72021514b189149"} Oct 30 10:02:39.022656 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:39.022619 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6llqt/must-gather-l5jrm" event={"ID":"a21cd284-322d-4d9c-98b8-11079be94d94","Type":"ContainerStarted","Data":"696ce0abcf4a1672d421d06baa380ec864d6cd0e239fd1d4bb303439369e6ac0"} Oct 30 10:02:39.062848 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:39.062783 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-6llqt/must-gather-l5jrm" podStartSLOduration=2.29976525 podStartE2EDuration="3.06276627s" totalImagesPullingTime="763.00102ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 10:02:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:02:39.061517253 +0000 UTC m=+4119.323078242" watchObservedRunningTime="2026-10-30 10:02:39.06276627 +0000 UTC m=+4119.324327258" Oct 30 10:02:43.248229 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:43.248198 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-vb26w_5e591466-fcf6-469d-9c3d-cde0767f4dde/global-pull-secret-syncer/0.log" Oct 30 10:02:43.998324 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:43.998239 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-h5vr9_634749a3-dbc0-41f4-9e97-bf3d25f9b6a8/konnectivity-agent/0.log" Oct 30 10:02:44.512294 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:44.512261 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-244.ec2.internal_e62eebd98344a1611517e7c60f1efe61/haproxy/0.log" Oct 30 10:02:45.668443 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:45.647071 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-5bbbc667d4-sdbtx_fd98de9c-4d7f-4f22-b042-2867781cf88c/manager/0.log" Oct 30 10:02:45.703403 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:45.699918 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-5bbbc667d4-sdbtx_fd98de9c-4d7f-4f22-b042-2867781cf88c/manager/0.log" Oct 30 10:02:48.347400 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:48.346596 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-687fd9c8d8-gt7tl_625507b9-d06d-4071-b33a-b959a6e3129a/manager/0.log" Oct 30 10:02:48.361144 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:48.361119 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-687fd9c8d8-gt7tl_625507b9-d06d-4071-b33a-b959a6e3129a/manager/0.log" Oct 30 10:02:51.439808 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:51.439647 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-74d66bdc9c-wxffc_664fe1ef-e0ec-4bc3-afe1-9eac486c4ac4/authorino/0.log" Oct 30 10:02:51.813543 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:51.813399 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-gv64t_a9a30ca0-995e-4e77-bf92-b3923ecc14c0/proxy/0.log" Oct 30 10:02:51.840538 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:51.840501 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-gv64t_a9a30ca0-995e-4e77-bf92-b3923ecc14c0/proxy/0.log" Oct 30 10:02:52.981973 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:52.980808 4562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6llqt/perf-node-gather-daemonset-gq9c5"] Oct 30 10:02:52.988410 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:52.987760 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6llqt/perf-node-gather-daemonset-gq9c5" Oct 30 10:02:53.004427 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:53.004216 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6llqt/perf-node-gather-daemonset-gq9c5"] Oct 30 10:02:53.148511 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:53.148471 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0d2ef708-1771-41ce-9310-b0ea95693218-podres\") pod \"perf-node-gather-daemonset-gq9c5\" (UID: \"0d2ef708-1771-41ce-9310-b0ea95693218\") " pod="openshift-must-gather-6llqt/perf-node-gather-daemonset-gq9c5" Oct 30 10:02:53.148836 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:53.148809 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0d2ef708-1771-41ce-9310-b0ea95693218-proc\") pod \"perf-node-gather-daemonset-gq9c5\" (UID: \"0d2ef708-1771-41ce-9310-b0ea95693218\") " pod="openshift-must-gather-6llqt/perf-node-gather-daemonset-gq9c5" Oct 30 10:02:53.148969 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:53.148873 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0d2ef708-1771-41ce-9310-b0ea95693218-sys\") pod \"perf-node-gather-daemonset-gq9c5\" (UID: \"0d2ef708-1771-41ce-9310-b0ea95693218\") " pod="openshift-must-gather-6llqt/perf-node-gather-daemonset-gq9c5" Oct 30 10:02:53.149069 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:53.149042 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqzrp\" (UniqueName: \"kubernetes.io/projected/0d2ef708-1771-41ce-9310-b0ea95693218-kube-api-access-vqzrp\") pod \"perf-node-gather-daemonset-gq9c5\" (UID: \"0d2ef708-1771-41ce-9310-b0ea95693218\") " pod="openshift-must-gather-6llqt/perf-node-gather-daemonset-gq9c5" Oct 30 10:02:53.149201 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:53.149144 4562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0d2ef708-1771-41ce-9310-b0ea95693218-lib-modules\") pod \"perf-node-gather-daemonset-gq9c5\" (UID: \"0d2ef708-1771-41ce-9310-b0ea95693218\") " pod="openshift-must-gather-6llqt/perf-node-gather-daemonset-gq9c5" Oct 30 10:02:53.245762 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:53.245668 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-5bbbc667d4-sdbtx_fd98de9c-4d7f-4f22-b042-2867781cf88c/manager/0.log" Oct 30 10:02:53.249902 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:53.249873 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqzrp\" (UniqueName: \"kubernetes.io/projected/0d2ef708-1771-41ce-9310-b0ea95693218-kube-api-access-vqzrp\") pod \"perf-node-gather-daemonset-gq9c5\" (UID: \"0d2ef708-1771-41ce-9310-b0ea95693218\") " pod="openshift-must-gather-6llqt/perf-node-gather-daemonset-gq9c5" Oct 30 10:02:53.250062 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:53.249922 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0d2ef708-1771-41ce-9310-b0ea95693218-lib-modules\") pod \"perf-node-gather-daemonset-gq9c5\" (UID: \"0d2ef708-1771-41ce-9310-b0ea95693218\") " pod="openshift-must-gather-6llqt/perf-node-gather-daemonset-gq9c5" Oct 30 10:02:53.250062 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:53.249984 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0d2ef708-1771-41ce-9310-b0ea95693218-podres\") pod \"perf-node-gather-daemonset-gq9c5\" (UID: \"0d2ef708-1771-41ce-9310-b0ea95693218\") " pod="openshift-must-gather-6llqt/perf-node-gather-daemonset-gq9c5" Oct 30 10:02:53.250062 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:53.250045 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0d2ef708-1771-41ce-9310-b0ea95693218-proc\") pod \"perf-node-gather-daemonset-gq9c5\" (UID: \"0d2ef708-1771-41ce-9310-b0ea95693218\") " pod="openshift-must-gather-6llqt/perf-node-gather-daemonset-gq9c5" Oct 30 10:02:53.250262 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:53.250117 4562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0d2ef708-1771-41ce-9310-b0ea95693218-sys\") pod \"perf-node-gather-daemonset-gq9c5\" (UID: \"0d2ef708-1771-41ce-9310-b0ea95693218\") " pod="openshift-must-gather-6llqt/perf-node-gather-daemonset-gq9c5" Oct 30 10:02:53.250262 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:53.250175 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0d2ef708-1771-41ce-9310-b0ea95693218-lib-modules\") pod \"perf-node-gather-daemonset-gq9c5\" (UID: \"0d2ef708-1771-41ce-9310-b0ea95693218\") " pod="openshift-must-gather-6llqt/perf-node-gather-daemonset-gq9c5" Oct 30 10:02:53.250262 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:53.250200 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0d2ef708-1771-41ce-9310-b0ea95693218-proc\") pod \"perf-node-gather-daemonset-gq9c5\" (UID: \"0d2ef708-1771-41ce-9310-b0ea95693218\") " pod="openshift-must-gather-6llqt/perf-node-gather-daemonset-gq9c5" Oct 30 10:02:53.250262 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:53.250176 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0d2ef708-1771-41ce-9310-b0ea95693218-podres\") pod \"perf-node-gather-daemonset-gq9c5\" (UID: \"0d2ef708-1771-41ce-9310-b0ea95693218\") " pod="openshift-must-gather-6llqt/perf-node-gather-daemonset-gq9c5" Oct 30 10:02:53.250262 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:53.250240 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0d2ef708-1771-41ce-9310-b0ea95693218-sys\") pod \"perf-node-gather-daemonset-gq9c5\" (UID: \"0d2ef708-1771-41ce-9310-b0ea95693218\") " pod="openshift-must-gather-6llqt/perf-node-gather-daemonset-gq9c5" Oct 30 10:02:53.268869 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:53.268840 4562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqzrp\" (UniqueName: \"kubernetes.io/projected/0d2ef708-1771-41ce-9310-b0ea95693218-kube-api-access-vqzrp\") pod \"perf-node-gather-daemonset-gq9c5\" (UID: \"0d2ef708-1771-41ce-9310-b0ea95693218\") " pod="openshift-must-gather-6llqt/perf-node-gather-daemonset-gq9c5" Oct 30 10:02:53.309151 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:53.309056 4562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6llqt/perf-node-gather-daemonset-gq9c5" Oct 30 10:02:53.331844 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:53.331806 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-5bbbc667d4-sdbtx_fd98de9c-4d7f-4f22-b042-2867781cf88c/manager/0.log" Oct 30 10:02:53.620589 ip-10-0-143-244 kubenswrapper[4562]: W1030 10:02:53.620546 4562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0d2ef708_1771_41ce_9310_b0ea95693218.slice/crio-66e8448465fd2900f35bd2950d0dd505133b9780e207604c8a08edf65ea79a7a WatchSource:0}: Error finding container 66e8448465fd2900f35bd2950d0dd505133b9780e207604c8a08edf65ea79a7a: Status 404 returned error can't find the container with id 66e8448465fd2900f35bd2950d0dd505133b9780e207604c8a08edf65ea79a7a Oct 30 10:02:53.620849 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:53.620829 4562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6llqt/perf-node-gather-daemonset-gq9c5"] Oct 30 10:02:54.215537 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:54.215492 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6llqt/perf-node-gather-daemonset-gq9c5" event={"ID":"0d2ef708-1771-41ce-9310-b0ea95693218","Type":"ContainerStarted","Data":"346cd4cbfa1d6b08bc46f35367bf05cddbd534a8ccf04e4502e07d0a1d2852de"} Oct 30 10:02:54.216150 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:54.215563 4562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6llqt/perf-node-gather-daemonset-gq9c5" event={"ID":"0d2ef708-1771-41ce-9310-b0ea95693218","Type":"ContainerStarted","Data":"66e8448465fd2900f35bd2950d0dd505133b9780e207604c8a08edf65ea79a7a"} Oct 30 10:02:54.216150 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:54.215841 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-6llqt/perf-node-gather-daemonset-gq9c5" Oct 30 10:02:54.266429 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:54.265215 4562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-6llqt/perf-node-gather-daemonset-gq9c5" podStartSLOduration=2.265196346 podStartE2EDuration="2.265196346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 10:02:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:02:54.262796583 +0000 UTC m=+4134.524357573" watchObservedRunningTime="2026-10-30 10:02:54.265196346 +0000 UTC m=+4134.526757335" Oct 30 10:02:55.495977 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:55.495941 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-687fd9c8d8-gt7tl_625507b9-d06d-4071-b33a-b959a6e3129a/manager/0.log" Oct 30 10:02:55.562916 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:55.562880 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-687fd9c8d8-gt7tl_625507b9-d06d-4071-b33a-b959a6e3129a/manager/0.log" Oct 30 10:02:56.294812 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:56.294780 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-bbf8d8c5f-dxdml_e6056d7b-f80f-4579-9347-1a3029facddd/metrics-server/0.log" Oct 30 10:02:56.415096 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:56.415068 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-6vp95_07ca6f17-5369-4d89-94b7-1f510621475a/monitoring-plugin/0.log" Oct 30 10:02:57.269647 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:57.269612 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xtw9h_a9379d2d-3c28-4a90-9ee4-c4afed3b0306/node-exporter/0.log" Oct 30 10:02:57.385489 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:57.385461 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xtw9h_a9379d2d-3c28-4a90-9ee4-c4afed3b0306/kube-rbac-proxy/0.log" Oct 30 10:02:57.502697 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:57.502673 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xtw9h_a9379d2d-3c28-4a90-9ee4-c4afed3b0306/init-textfile/0.log" Oct 30 10:02:58.600774 ip-10-0-143-244 kubenswrapper[4562]: E1030 10:02:58.600735 4562 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lc8jz" containerName="extractor" Oct 30 10:02:59.101832 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:59.101798 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-dhnj7_1c3185f1-9fd0-4f9c-9ba5-6ba703f45cc6/prometheus-operator-admission-webhook/0.log" Oct 30 10:02:59.135815 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:59.135785 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-gv64t_a9a30ca0-995e-4e77-bf92-b3923ecc14c0/proxy/0.log" Oct 30 10:02:59.203804 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:02:59.203772 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-gv64t_a9a30ca0-995e-4e77-bf92-b3923ecc14c0/proxy/0.log" Oct 30 10:03:00.233346 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:03:00.233318 4562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-6llqt/perf-node-gather-daemonset-gq9c5" Oct 30 10:03:00.512204 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:03:00.512119 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-5bbbc667d4-sdbtx_fd98de9c-4d7f-4f22-b042-2867781cf88c/manager/0.log" Oct 30 10:03:00.607187 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:03:00.607150 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-5bbbc667d4-sdbtx_fd98de9c-4d7f-4f22-b042-2867781cf88c/manager/0.log" Oct 30 10:03:02.712075 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:03:02.712039 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-687fd9c8d8-gt7tl_625507b9-d06d-4071-b33a-b959a6e3129a/manager/0.log" Oct 30 10:03:02.825932 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:03:02.825898 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-687fd9c8d8-gt7tl_625507b9-d06d-4071-b33a-b959a6e3129a/manager/0.log" Oct 30 10:03:03.986068 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:03:03.986035 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-56bcdf8bfd-z2fmv_87ebdbba-18b3-43d3-a509-93382374d35f/console/0.log" Oct 30 10:03:04.116212 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:03:04.116183 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-nvfch_36743cf2-58e7-4e0f-a6d3-df4733e65937/download-server/0.log" Oct 30 10:03:06.403338 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:03:06.403292 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-gv64t_a9a30ca0-995e-4e77-bf92-b3923ecc14c0/proxy/0.log" Oct 30 10:03:06.508587 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:03:06.508555 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-gv64t_a9a30ca0-995e-4e77-bf92-b3923ecc14c0/proxy/0.log" Oct 30 10:03:07.599223 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:03:07.599181 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-95kld_6edea6bc-4807-48ff-b879-a77fc07d92de/dns/0.log" Oct 30 10:03:07.712058 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:03:07.712026 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-95kld_6edea6bc-4807-48ff-b879-a77fc07d92de/kube-rbac-proxy/0.log" Oct 30 10:03:08.448286 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:03:08.448250 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lq2p2_9c67a6b6-a3d8-49bc-a678-b203ef4201a9/dns-node-resolver/0.log" Oct 30 10:03:09.869046 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:03:09.869014 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-58cf89764b-plrvc_c209abe2-a298-4045-a241-5537b814167e/registry/0.log" Oct 30 10:03:10.220948 ip-10-0-143-244 kubenswrapper[4562]: I1030 10:03:10.220903 4562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xqjlj_1b64ad2b-20eb-4a29-93b9-ccc9a217135d/node-ca/0.log"